none
[U8.1][C#] HttpWebRequest's GetResponseAsync fails with OutOfMemoryException

    Question

  • Hi everyone,

    I'm having a strange issue with fetching data from a web service. Here's my code, with a few boring bits trimmed:

    private async Task<IEnumerable<string[]>> GenericGrab(string query)
    {
        var req = (HttpWebRequest) WebRequest.Create(new Uri("http://example.org/" + query));
        HttpWebResponse res = null;
        try
        {
            res = (HttpWebResponse) await req.GetResponseAsync().ConfigureAwait(false); // BOOM!
        }
        // Boring exception handlers skipped
        finally
        {
            if (res != null)
            {
                res.Dispose();
            }
        }
        StreamReader read = new StreamReader(res.GetResponseStream());
        string l;
        while (true) {
            l = await read.ReadLineAsync().ConfigureAwait(false);
            if (l == null) break;
            // Process lines here
        }
        // Return result of processing
    }

    This works fine on requests that return 20 KB of data and dies, immediately after running the line containing GetResponseAsync, with an OutOfMemoryException (wrapped in the XAML-generated break on unhandled exception) with requests that return 600 KB of data. (The way this code is normally used is to first request a small amount of data, then a larger amount. If I remove the second call, I can do hundreds of request cycles without running into an exception.)

    I find this rather strange because as far as I understand, GetResponseAsync should not even read in all the data... after all, if it did, using a response stream would not make much sense. So, what could be the issue?

    Saturday, May 2, 2015 11:24 AM

Answers

  • I have accidentally resolved the issue by switching to the Windows.Web.Http components. Still not entirely sure why this problem happened with less than one MB of data, but oh well, it works and that's enough for me at this point.
    Thursday, May 28, 2015 8:17 PM

All replies

  • Question

    Does it work without the configureAwait(false)?

      try
        {
            res = (HttpWebResponse) await req.GetResponseAsync(); // BOOM!
        }

    • Proposed as answer by Mankdng Nef Tuesday, May 19, 2015 7:37 AM
    Sunday, May 3, 2015 12:43 PM
  • I only ever use HttpClient myself (please consider switching to it! It's a much nicer API than anything else!)

    The default behavior of HttpClient, surprisingly, is to read the entire response. Even when you read the content as a stream, the content has already been read onto your local machine, and you're streaming it from RAM.

    Instead, if you want HttpClient not to pre-fetch the entire content, you have to pass the "ResponseHeadersRead" flag in the call, e.g.

    Using response = Await client.GetAsync(uri,

                                  HttpCompletionOption.ResponseHeadersRead,

                                  cancel)

    I wonder if something similar is going on for you?

    • Proposed as answer by Mankdng Nef Tuesday, May 19, 2015 7:37 AM
    Sunday, May 3, 2015 10:27 PM
  • Does it work without the configureAwait(false)?

    That was actually what I had first, until I learned about configureAwait and decided to try if it made a difference. Thanks, though. :)
    Thursday, May 28, 2015 8:12 PM
  • I only ever use HttpClient myself (please consider switching to it! It's a much nicer API than anything else!)

    Definitely. HttpClient (the one from System.Net.Http) is actually what I used first, and when I had trouble with that I switched to HttpRequest based on allegations on the web that HttpClient had limitations the details of which I don't really remember.

    The default behavior of HttpClient, surprisingly, is to read the entire response.

    I actually figured as much but didn't think it would cause issues with a <1 MB response. I'll definitely keep this in mind, along with your snippet for changing the behaviour. Thanks!

    Thursday, May 28, 2015 8:15 PM
  • I have accidentally resolved the issue by switching to the Windows.Web.Http components. Still not entirely sure why this problem happened with less than one MB of data, but oh well, it works and that's enough for me at this point.
    Thursday, May 28, 2015 8:17 PM