WebRequest and WebResponse has issues

Question

WebRequest and WebResponse has issues

I wrote a C# program that uses WebRequest and WebResponse to perform a simple web crawler.  I discovered something about web sites.  Web browsers such as IE and FireFox offer the capacity to view the HTML source code.  But it seems that html code that is sent to the browser is one thing and what the browser interprets and displays is something else.  For example, if you run a google search in IE and run the same google search in FireFox, the content that you can see when you view the source in IE will NOT have the hyperlinks and content from the search results, but you can see the html hyperlinks and content from the search results when you view the source in FireFox.  So my question is this.  How do you specialise the WebRequest and WebResponse to show the content after it is processed by the browser instead of before?

One possible solution might be to use HttpWebRequest instead of WebRequest and use the UserAgent property to somehow trick C# into thinking I am using the Firefox  browser.  But this does not seem to me to plausable.

I also tried the WebBroswer control without setting it to visible, but I don't know the syntax to use.

The examples I have found seem to suggest that before I use  the WebBrowser class it is required to assign it to a URL.  Following that< can set the visitiblity to hidden.  But wouldn't it be too late then?

The "Navagate" method does not seem to offer the capacity to load a URL hidden.

http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.navigate.aspx

Answer 1

I'm confused as to what the problem is.  I THINK you are saying that was is SENT to the browser is different depending on the user agent (IE vs. FF for instance).  That is normal.  The browsers are just making requests from the web site.  So is your WebRequest object.  So if you want the WebRequest to return to you what FF gets, then tell the web site that you ARE FF (specifying the UserAgent).

I don't know WHAT you mean by after the browser processes it.  That would be what you see on the screen.  Just a rendering of what the html was that got returned to the client.