none
HttpWebResponse. GetResponseStream() returns empty RRS feed

  • Question

  • I have developed a process to download data from a website, I downloaded the content of each url and write them into a text file.

    The process runs very well for nearly half a year, but recently some exception happens. the text files can be generated, but its content is empty, but I can see there some data if i browser the url in IE.

    attaching part of the code.

    void DownloadFile(string url, string username, string password, string path)
            {
                CookieContainer Cc = new CookieContainer();
                string postData = String.Format("username={0}&password={1}", username, password);
                StreamReader sr = null;
                HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
                System.Security.Cryptography.X509Certificates.X509Certificate yourCert;
                yourCert = new System.Security.Cryptography.X509Certificates.X509Certificate(certFile, certPwd);
                request.ClientCertificates.Add(yourCert);
                request.Method = "POST";
                request.Timeout = 10000000;
                request.CookieContainer = new CookieContainer();
                byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(postData);
                request.ContentType = "application/x-www-form-urlencoded";
                request.ContentLength = byteArray.Length;
                request.CookieContainer = Cc;
                string param = "hl=zh-CN&newwindow=1";
                byte[] bs = Encoding.ASCII.GetBytes(param);

                request.AddRange(1024);

                Stream dataStream = request.GetRequestStream();
                dataStream.Write(byteArray, 0, byteArray.Length);
                dataStream.Close();
                try
                {
                    using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                    {
                        if ((int)response.StatusCode < 400 || (int)response.StatusCode > 499)
                        {
                            sr = new StreamReader(response.GetResponseStream());
                            using (StreamWriter sw = new StreamWriter(path))
                            {
                                sw.Write(sr.ReadToEnd());
                            }
                        }
                        else
                        {
                            //
                        }
                    }
                }
                catch (Exception ex)
                {
                   
                }
                finally
                {
                    sr.Close();
                }
            }

    when debug, I can see the StatusCode  returns OK, but stream looks empty. I use the following code to check:

    sr = new StreamReader(response.GetResponseStream());

    string text = sr.ReadToEnd();

    and the text is empty.

    what could be the possible reason??

    Thursday, September 12, 2013 3:17 AM

Answers

  • Download for free a sniffer like fiddler or wireshark to confirm that no data was returned.  The data may be coming in multiple blocks and you just need to send another request to get more data or the query you performed didn't return any data.  Normally I go to an IE application an use a sniffer to see what occurs using the IE, then compare the sniffer results with what I get using my VS application.  Normally I find that the IE has additional headers that are missing in my IE application.  Adding the missing headers usually solves the problem.

    jdweng

    Thursday, September 12, 2013 7:12 AM
  • Hi Jason,

    Welcome to MSDN forum.

    Sometimes browser will receive HTTP CODE 3xx and redirect to a new URL. Browser knows to redirect to new URL, but your code could not do this. I think you encountered this scenarios. My solution is to find out the new URL and direct access the new URL. I agree with Joel to use sniffer to find the new URL. You could download free basic Edition on http://www.httpwatch.com/ and how to use it http://help.httpwatch.com/gettingstarted.html.

    Best Regards

    <THE CONTENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED>
    Thanks
    MSDN Community Support

    Please remember to "Mark as Answer" the responses that resolved your issue. It is a common way to recognize those who have helped you, and makes it easier for other visitors to find the resolution later.

    Friday, September 13, 2013 10:05 AM
    Moderator

All replies

  • Download for free a sniffer like fiddler or wireshark to confirm that no data was returned.  The data may be coming in multiple blocks and you just need to send another request to get more data or the query you performed didn't return any data.  Normally I go to an IE application an use a sniffer to see what occurs using the IE, then compare the sniffer results with what I get using my VS application.  Normally I find that the IE has additional headers that are missing in my IE application.  Adding the missing headers usually solves the problem.

    jdweng

    Thursday, September 12, 2013 7:12 AM
  • Hi Jason,

    Welcome to MSDN forum.

    Sometimes browser will receive HTTP CODE 3xx and redirect to a new URL. Browser knows to redirect to new URL, but your code could not do this. I think you encountered this scenarios. My solution is to find out the new URL and direct access the new URL. I agree with Joel to use sniffer to find the new URL. You could download free basic Edition on http://www.httpwatch.com/ and how to use it http://help.httpwatch.com/gettingstarted.html.

    Best Regards

    <THE CONTENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED>
    Thanks
    MSDN Community Support

    Please remember to "Mark as Answer" the responses that resolved your issue. It is a common way to recognize those who have helped you, and makes it easier for other visitors to find the resolution later.

    Friday, September 13, 2013 10:05 AM
    Moderator
  • thanks for your reply!

    sorry for the late response,

    our data supporter have corrected the server side issue, and now it works again. 

    thanks for your recommendations, will try that in future.

    Tuesday, October 1, 2013 2:34 AM
  • thanks for your reply!

    sorry for the late response,

    our data supporter have corrected the server side issue, and now it works again. 

    thanks for your recommendations, will try that in future.

    Tuesday, October 1, 2013 2:35 AM