HttpOnly cookies and File Download Dialog with WebBrowser control in vb 2008

Question

Hi All,

Here's my problem:

I'm creating an automated browser for doing a bunch of repetitive tasks on an internal company website.  I have all of the basics working (login, navigate to the appropriate page, click the buttons etc). 

What I need to do is silently retrieve a file from the website, which gets fed into the program and worked on.

The url for the file to download is not directly accessible -- the web page uses *lots* of javascript.   When the appropriate button is clicked, it does a post to the server, a popup opens, it gets the url from the server, and a normal file download dialog happens. 

I can retrieve the url by using the filedownload event, and then stop the browser from navigating, which gives me the url without opening the file download dialog.

I tried to retrieve the file with HttpWebRequest, but it didn't work, and the website dumped me back out to the login screen.  I figured out that particular problem is that the website is using a couple of HttpOnly cookies.  I can see them with Fiddler2, but can't retrieve them for reuse with WebBrowser1.Document.Cookie, which of course won't handle HttpOnly cookies.

Ideally, if I can get ALL of the cookies from the WebBrowser control to the HttpWebRequest somehow, it would be pretty simple to dump the file into a textbox and go from there.  Alternatively, if I could handle the file download dialog silently and save the file to a program controlled directory, that would be a workable alternative.

I searched all over and haven't been able to find quite what I need.  I'm really stuck on this one, and up against a deadline, so I'd really appreciate any help at all.

Thanks for reading!

tt

---

tt

Answer 1

Since you are doing this for your company, why not ask them to make the info you need more easily accesable in the way you need it?

 

Here is an article describing the use of HttpOnly cookies in an HttpWebRequest.  Perhaps this will help.

--
Mike

Answer 2

Thanks for your reply Family Tree Mike.

We've asked the company repeatedly over the years to improve the website, but they're too cheap to spend their I.T. dollars for stuff that will help contractors like me, which is why I'm writing this software.  I just have to work with what I can get.

The article you cited was helpful, but I still need a way of extracting the HttpOnly cookies from the WebBrowser control.  The article cited another article to do that, but it was in C, which unfortunately I don't know.  VB is enough headache to deal with.

Any help extracting the HttpOnly cookies from the WebBrowser control would be really helpful.

Thanks for reading!

---

tt

Answer 3

I don't suppose they've left it so open that you can read the file through \\servername\c\inetpub\wwwroot\whatever\file.txt, have they?

Otherwise, perhaps you could say it will cost 10x to do it the complicated way they're forcing you to use versus x to add a web service (it can still be passworded) to the site.

--
Andrew

Answer 4

Thanks for your reply Andrew.

Unfortunately, no.  The Powers That Be have decided this is how it's going to be, and we just have to live with it.  So, to make my (and my compatriots) life easier, I've got to get this working -- and quickly.

At this point I'm figuratively grasping at straws trying to get it going.  The last piece of the puzzle that I need is as described above, so any help is greatly appreciated.

Thanks!

---

tt

Answer 5

This article: How to: Get and Set Cookies seems to imply that the CookieContainer will have the http-only cookies, even if you can't access them. At worst, you could host a WPF control on the form if it doesn't carry over to Windows Forms programs.

--
Andrew

Answer 6

Hi Andrew,

I did look at that already.  I need to use the cookies from the webbrowser control, and pass them to an appropriate httpwebrequest. I don't need to see them - just use them.  I haven't been able to get them from the webbrowser control.  Everything l've looked at so far seems to use the httpwebrequest to retrieve them initially, which isn't an option in this case (because of all of the javascript on the page that generates things). It looks like it would be a simple matter to pass them  if I could just capture them in the first place from the webbrowser control.

Thanks!

---

tt

Answer 7

If you need help converting C# to VB net I can help.  I did some reading on the web and ther appear to be only three solutions

1) Use HTTPwebrequest instead of a webbrowser.  I converted the C# code on this webpage (see below).   Can't guarentee it will work.

http://stackoverflow.com/questions/5027724/webbrowser-navigate-how-to-pass-ntlm-credentials

http://stackoverflow.com/questions/4583052/network-authentication-and-website-authentication-using-httpwebrequest

2) Add the credentials into you IE browser.  So I would add this site to the trusted site on the machine and then check the User Authentication -> Logon -> Automatic Logon with current username and password setting in IE for the trusted sites.

3) Make a WCF application and add the credentials into the web.config file.

Imports System.Net
Imports System.Web
Imports System.Text
Imports System.IO
Imports System.IO.Stream
Imports System.Threading
Public Class Form1
    Dim cred As ICredentials = New NetworkCredential(username, password, domain)
    Dim authorisationHeader As String = String.Empty
    Public Sub XYZ(ByVal url As String, ByVal width As Integer, ByVal height As Integer)
        Dim Active As Boolean = Ping(url)
        If Active Then
            ' Load the webpage into a WebBrowser control 
            Me.WebBrowser1.AllowNavigation = True
            Me.WebBrowser1.AllowWebBrowserDrop = True
            Dim postData As Byte() = {}
            Me.WebBrowser1.Navigate(url, "", postData, authorisationHeader)
            Do While (Me.WebBrowser1.ReadyState <> WebBrowserReadyState.Complete)
                Application.DoEvents()
            Loop
            Me.WebBrowser1.Dispose()
        End If
    End Sub
    Private Function Ping(ByVal url As String) As Boolean
        Try
            Dim RSSFeed As String
            Dim credentials As String = Convert.ToBase64String(ASCIIEncoding.ASCII.GetBytes("UserName" + ":" + "Password"))
            Dim outputData As StringBuilder = New StringBuilder()
            Dim myHttpWebRequest As HttpWebRequest = WebRequest.Create(RSSFeed)
            myHttpWebRequest.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials
            myHttpWebRequest.Headers.Add("Authorization", "Basic " + credentials)
            Dim myHttpWebResponse As HttpWebResponse = myHttpWebRequest.GetResponse()
            Dim streamResponse As Stream = myHttpWebResponse.GetResponseStream()
            Dim status As Boolean
            Dim req As HttpWebRequest = WebRequest.Create(url)
            req.Credentials = cred
            req.AllowAutoRedirect = True
            req.ContentType = "text/html"
            req.Accept = "*/*"
            req.KeepAlive = False
            Dim resp As HttpWebResponse
            resp = req.GetResponse()
            If resp.StatusCode.ToString() = "OK" Then
                status = True
            Else
                status = False
            End If
            Dim c As WebHeaderCollection = resp.Headers
            authorisationHeader = req.Headers.GetValues("Authorization")(0)
            Return status
        Catch
            Return False
        End Try
    End Function
End Class

---

jdweng

Answer 8

Hi Joel,

Thanks for your reply. 

Unfortunately, the website doesn't use ntlm (or Kerberos)  for authentication (at least as far as I can tell).  It's all handled with javascript and some httponly cookies (and regular cookies as well), which is one of the reasons this is giving me such a headache. 

I did find: http://zerosandtheone.com/media/p/277.aspx which will handle the popup window for the file download correctly. It then gives me the file download dialog, which I want to avoid having to do manually, because of the large number of times it has to be done.

As a possible alternative, I also looked at: http://www.codeproject.com/Articles/31163/Suppressing-Hosted-WebBrowser-Control-Dialogs with the idea of perhaps intercepting and automatically controlling the file download dialog, but not being a c# programmer, it didn't help me much, since it didn't specifically implement anything for the file download dialog. 

As far as controlling the file download dialog,  I just need to tell it to save to a filename, and overwrite an existing file if necessary. 

Thanks again - your help is appreciated.

---

tt

Answer 9

Ok - Here's the solution.  Thanks to everyone who replied.  Your replies got me thinking in different directions, and I (finally!) found the answer.

I'm posting this here, since this seems to be a common problem, and wasn't answered explicitly (at least nowhere else that I could find).

To handle the popup window correctly (opened by the website for downloading the file), I used kleinma's extended web browser control, which can be found here: http://zerosandtheone.com/media/p/277.aspx

This link gives code for using Urldownloadtofile:

http://www.vbforums.com/showthread.php?592926-URLdownloadtofile-problem

The sample code given for Urldownloadtofile works just fine -- it seems to correctly pass all of the cookies required by the server for authentication purposes.

What I did was handle the FileDownload event, use WebBrowserEx1.Stop() to halt the navigation, then extract the url for the file with WebBrowserEx1.Document.Url.ToString.  I then passed the url to the code given in UrlDownloadtofile, which then downloads the file to the specified filename (including path).  Ended up being pretty simple in the end.

Again, many thanks to those who took the time to reply, and I hope this helps someone else in the future.

---

tt