none
Windows Phone 8: Portable HttpClient causes OutOfMemoryException downloading large file

    Question

  • I can reproduce this with a file as small as 200MB, so clearly the HttpClient is buffering the file in memory even though I'm telling it not to.

    I've tried setting MaxResponseContentBufferSize and such to no avail.  And WebClient has the same problem.

    Say it ain't so, HttpClient!

        async Task CrashAsync(string url)
        {
          using (HttpClient client = new HttpClient())
          {
            using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
            {
              Console.WriteLine("{0}", response.Content.Headers.ContentLength.Value);
            }
          }
        }

    Thursday, February 20, 2014 5:32 AM

All replies

  • Can you post a URL with a file big enough to create the issue?

    Jeff Sanders (MSFT)

    @jsandersrocks - Windows Store Developer Solutions @WSDevSol
    Getting Started With Windows Azure Mobile Services development? Click here
    Getting Started With Windows Phone or Store app development? Click here
    My Team Blog: Windows Store & Phone Developer Solutions
    My Blog: Http Client Protocol Issues (and other fun stuff I support)

    Thursday, February 20, 2014 2:07 PM
  • Hi, Jeff.  Thanks for the interest!

    I did some further experimentation today.  My observation is that the issue occurs when the file's download speed is very high.  A file with a low download speed will not reproduce the problem, regardless of size.

    My suspicion is that there's an unbounded download buffer somewhere underneath HttpClient, but I'm not entirely sure why speed matters.  Just adding a Task.Delay() doesn't cause the issue to occur with low speeds.  It's probably some kind of race condition.

    Anyway, here's a file you can test with: http://files.plexapp.com/mfeingol/public/file.avi

    My suggestion is to download it, place it on a local web server, and then download it from the local URL from inside a Windows Phone emulator with 512MB of RAM.  You should see the OOM exception fairly quickly if you keep the process alive after the GetAsync().

    In my testing, it took a few seconds over gigabit ethernet to see the OOM, whereas it was fairly instant over localhost loopback.  I couldn't reproduce it downloading from the internet.

    Please let me know if you have any further questions.

    I should add that this is a fairly important scenario for our application, and this limitation leaves us fairly dead in the water.  We may need to implement our own HttpClient...

    Thursday, February 20, 2014 6:23 PM
  • Jeff, isn't this the same as your other answer, you need to turn off streaming to buffer?

    http://pauliom.wordpress.com

    Friday, February 21, 2014 7:55 AM
  • Hi Paul,  I think so... But I like to test out stuff.  Feel free to give it a shot for me!

    Jeff Sanders (MSFT)

    @jsandersrocks - Windows Store Developer Solutions @WSDevSol
    Getting Started With Windows Azure Mobile Services development? Click here
    Getting Started With Windows Phone or Store app development? Click here
    My Team Blog: Windows Store & Phone Developer Solutions
    My Blog: Http Client Protocol Issues (and other fun stuff I support)

    Friday, February 21, 2014 2:34 PM
  • I have searched to no avail. Are we talking about the MaxResponseContentBufferSize property?  If so, do we mean setting it to zero?

    EDIT: for what it's worth, I tried using HttpWebRequest instead, with these properties set:

    HttpWebRequest webRequest = HttpWebRequest.CreateHttp(url);
    webRequest.AllowAutoRedirect = true;
    webRequest.AllowReadStreamBuffering = false;
    webRequest.AllowWriteStreamBuffering = false;
    webRequest.Method = "GET";
    And yet, I still see an OOM fairly quickly after calling BeginGetResponse().
    • Edited by mfeingol Friday, February 21, 2014 8:15 PM
    Friday, February 21, 2014 5:33 PM
  • What device are you using, you could easily be breaking the memory limits; http://msdn.microsoft.com/en-us/library/windowsphone/develop/jj681682(v=vs.105).aspx

    http://pauliom.wordpress.com

    Saturday, February 22, 2014 9:10 AM
  • 512MB emulator.

    Downloading a 200MB file to disk should be perfectly achievable on such a device.

    Saturday, February 22, 2014 4:47 PM
  • I believe the 512 MB emulator *is* the low memory device.

    http://pauliom.wordpress.com

    Saturday, February 22, 2014 7:08 PM
  • Right. This scenario should work.
    Saturday, February 22, 2014 8:44 PM
  • No it shouldn't, 150mb limit

    http://pauliom.wordpress.com

    Saturday, February 22, 2014 10:47 PM
  • On downloading a file to disk via HTTP?  I'm confused.
    Saturday, February 22, 2014 10:55 PM
  • If you stream each byte directly from the web to the file then yes it should only incur 1 byte memory, but I've not seen the code that is doing that. 

    http://pauliom.wordpress.com

    Saturday, February 22, 2014 11:32 PM
  • The repro code I posted above just opens a stream to the response. Just imagine that the next step is to read 64kB chunks from the stream and write them to disk.

    The problem is that with a fast connection (e.g. in-emulator), we never reach that point, because HttpClient and WebClient have already run the system out of memory by buffering data from the stream in an unbounded fashion.  They shouldn't do that.

    Saturday, February 22, 2014 11:39 PM
  • Jeff:

    Have you had a chance to take a look at this?

    Thanks!

    Wednesday, February 26, 2014 6:45 PM
  • Hi M,

    No not yet.  I won't likely have any free time until early next week.  If this is a pressing issue you can open a support case and get some 1:1 help immediately.

    Jeff


    Jeff Sanders (MSFT)

    @jsandersrocks - Windows Store Developer Solutions @WSDevSol
    Getting Started With Windows Azure Mobile Services development? Click here
    Getting Started With Windows Phone or Store app development? Click here
    My Team Blog: Windows Store & Phone Developer Solutions
    My Blog: Http Client Protocol Issues (and other fun stuff I support)

    Wednesday, February 26, 2014 6:48 PM
  • I think WP uses a pseudo-silverlight engine for web requests (which gives it caching capabilities), and a response.Read in your app isn't interlocked 1:1 with a socket.Read.  Silverlight pulls it in as fast as it can so it is ready for the app to consume. 

    I'd suspect the design decision was made also with some thought to cellular carriers, or for radio or power design reasons.  Maintaining a quick "get dat!" would be way more important for the carrier's network operations than a million devices keeping sockets open that extra couple seconds in a wait.  It's speculation of course - but if you can end a receive 2 seconds per device - across 1 million devices it's a lot of time (and resources) saved.

    I'd also wager that if you added the read/write - your app would never reach OOM.  I'd suspect memory-> storage would pre-empt the network receives anyway, so this problem isn't really a real world thing, until gigabit WiFi or gigabit cellular comes out.


    Darin R.

    Wednesday, February 26, 2014 9:54 PM
  • You make a very good point.  And I will say that I haven't been able to reproduce this on a Lumia 520 with 802.11N.

    That said, having added the read/write portion in the actual app, I do still see OOM over gigabit ethernet in-emulator.  So with faster CPUs and faster networks, this will be a problem in real life.

    Thursday, February 27, 2014 6:03 AM
  • I took a look at this, and I think HttpClient will always download the entire response before returning to your code in order to write data to a file.   Instead you should use HttpWebRequest.  I modified the sample code in the HttpWebRequest.BeginGetResponse that should download and write the data to a file without crashing the emulator.  Can you give it a try?

        public class RequestState
        {
            public const int BUFFER_SIZE = 1024;
            public Stream _file;
            public byte[] bufferRead;
            public HttpWebRequest request;
            public HttpWebResponse response;
            public Stream streamResponse;
    
            public RequestState()
            {
                bufferRead = new byte[BUFFER_SIZE];
                _file = null;
                request = null;
                streamResponse = null;
            }
        }
            private async Task Crash(string url)
            {
                //  Create the request state
                RequestState reqState = new RequestState();
    
                //  Create the file we want to write too
                StorageFolder local = Windows.Storage.ApplicationData.Current.LocalFolder;
                reqState._file = await local.OpenStreamForWriteAsync("myDownload.avi", 
    
    CreationCollisionOption.GenerateUniqueName);
                
                //  Start the http web request
                HttpWebRequest req = WebRequest.CreateHttp(url);
                reqState.request = req;
                req.BeginGetResponse(new AsyncCallback(RespCallback), reqState);
    
    
    
            }
    
            private static void RespCallback(IAsyncResult asynchronousResult)
            {
                try
                {
                    // State of request is asynchronous.
                    RequestState myRequestState = (RequestState)asynchronousResult.AsyncState;
                    HttpWebRequest myHttpWebRequest2 = myRequestState.request;
                    myRequestState.response = (HttpWebResponse)myHttpWebRequest2.EndGetResponse
    
    (asynchronousResult);
    
                    // Read the response into a Stream object.
                    Stream responseStream = myRequestState.response.GetResponseStream();
                    myRequestState.streamResponse = responseStream;
    
                    // Begin the Reading of the contents of the HTML page and print it to the 
    
    console.
                    IAsyncResult asynchronousInputRead = responseStream.BeginRead
    
    (myRequestState.bufferRead, 0, RequestState.BUFFER_SIZE, new AsyncCallback(ReadCallBack), 
    
    myRequestState);
                }
                catch (WebException e)
                {
                    // Need to handle the exception
                    // ...
    
                    Debug.WriteLine(e.Message);
                }
            }
    
            private static void ReadCallBack(IAsyncResult asyncResult)
            {
                try
                {
                    RequestState myRequestState = (RequestState)asyncResult.AsyncState;
                    Stream responseStream = myRequestState.streamResponse;
                    int read = responseStream.EndRead(asyncResult);
    
                    // Read the HTML page and then do something with it
                    if (read > 0)
                    {
                        myRequestState._file.Write(myRequestState.bufferRead, 0, read);
                        IAsyncResult asynchronousResult = responseStream.BeginRead
    
    (myRequestState.bufferRead, 0, RequestState.BUFFER_SIZE, new AsyncCallback(ReadCallBack), 
    
    myRequestState);
                    }
                    else
                    {
                        Debug.WriteLine("File is done downloading");
                        responseStream.Close();
                        myRequestState._file.Close();
                    }
                }
                catch (WebException e)
                {
                    // Need to handle the exception
                    // ...
    
                    Debug.WriteLine(e.Message);
                }
            }
    


    Bret Bentzinger (MSFT) @awehellyeah

    Thursday, February 27, 2014 11:37 PM
  • Bret:

    Thanks for the code.  I modified it a bit in order to integrate it into my project, and ... unfortunately... it still crashes.

    Try using this component, and observe the different between setting the PleaseCrash property to true and to false.  On my machine, even a 1ms delay will cause the crash...  So anything minorly significant we do in a progress notification callback is likely to run the process out of memory.

    using System;
    using System.Globalization;
    using System.IO;
    using System.Net;
    using System.Threading.Tasks;
    using Windows.Storage;
    
    namespace HttpCrashTest
    {
      public class BinaryTransferProgress
      {
        public ulong Progress { get; set; }
        public ulong Size { get; set; }
      }
    
      public class BinaryTransfer
      {
        const int BufferSize = 65536;
    
        string url;
        StorageFile file;
        ulong? offset;
    
        byte[] buffer;
        TaskCompletionSource<object> completion;
    
        Stream fileStream;
        ulong size;
        ulong progress;
    
        public BinaryTransfer(string url, StorageFile file, ulong? offset = null)
        {
          this.url = url;
          this.file = file;
          this.offset = offset;
    
          this.buffer = new byte[BufferSize];
          this.completion = new TaskCompletionSource<object>();
        }
    
        public bool PleaseCrash { get; set; }
    
        public async Task Transfer()
        {
          HttpWebRequest request = WebRequest.CreateHttp(url);
          if (offset.HasValue)
            request.Headers[HttpRequestHeader.Range] = String.Format(CultureInfo.InvariantCulture, "bytes={0}-", offset.Value);
    
          request.BeginGetResponse(this.BeginGetResponseCallback, request);
          await this.completion.Task;
        }
    
        public event EventHandler<BinaryTransferProgress> TransferProgress;
    
        async void BeginGetResponseCallback(IAsyncResult ar)
        {
          HttpWebRequest request = (HttpWebRequest)ar.AsyncState;
          try
          {
            HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(ar);
            if (response.StatusCode != HttpStatusCode.OK && response.StatusCode != HttpStatusCode.PartialContent)
              throw new ApplicationException(String.Format("Server returned {0}", (int)response.StatusCode));
    
            this.size = (ulong)response.ContentLength;
            this.fileStream = await file.OpenStreamForWriteAsync();
    
            Stream responseStream = response.GetResponseStream();
            responseStream.BeginRead(this.buffer, 0, this.buffer.Length, this.ReadCallBack, responseStream);
          }
          catch (Exception e)
          {
            this.completion.SetException(new ApplicationException(e.Message, e));
          }
        }
    
        async void ReadCallBack(IAsyncResult ar)
        {
          try
          {
            Stream responseStream = (Stream)ar.AsyncState;
            int read = responseStream.EndRead(ar);
            if (read > 0)
            {
              this.progress += (ulong)read;
    
              if (this.PleaseCrash)
                await Task.Delay(TimeSpan.FromMilliseconds(1));
    
              await fileStream.WriteAsync(buffer, 0, read);
    
              EventHandler<BinaryTransferProgress> handler = this.TransferProgress;
              if (handler != null)
                handler(this, new BinaryTransferProgress() { Progress = this.progress, Size = this.size });
    
              responseStream.BeginRead(buffer, 0, this.buffer.Length, this.ReadCallBack, responseStream);
            }
            else
            {
              responseStream.Close();
              fileStream.Close();
              this.completion.SetResult(null);
            }
          }
          catch (Exception e)
          {
            this.completion.SetException(new ApplicationException(e.Message, e));
          }
        }
      }
    }
    

    Thursday, March 06, 2014 2:52 AM
  • The problem is you are doing more async work in an async callback. Specifically Task.Delay when PleaseCrash = true.  I would not do any async work in this method.

    Using the following instead, will result in no crash:

    if (this.PleaseCrash)
    {
        Thread.Sleep(1000);
       //await Task.Delay(TimeSpan.FromMilliseconds(1));
    }
    



    Bret Bentzinger (MSFT) @awehellyeah

    Saturday, March 08, 2014 12:07 AM
  • I think what you're saying is that if I block the read callback, the responseStream won't continue to download and buffer.  Which makes sense.

    However, in real life I'm just about to write to the StorageFile's stream using an async call.  Should that be a blocking call as well?

    Further, how do I now that it's okay to block this thread?  Could HttpWebRequest on WP potentially use the UI thread?

    Saturday, March 08, 2014 12:42 AM
  • I've also been under the impression that the UI thread is involved.  Perhaps the web cache--the really annoying and not really avoidable without using sockets directly (some server return 304s in response to any request with an If-Modified-Since header, regardless of the timestamp) web cache--is managed on the UI thread?

    In my case, I'm trying to stream large media files and I really, really don't want to read them from the network faster than they play.

    Blocking the UI thread in order to avoid bugs in the web stack doesn't seem like an ideal solution.

    I've thought about trying to build an alternative to HttpClientHandler, perhaps as a Windows Phone Runtime Component with code from the C++ REST SDK (https://casablanca.codeplex.com/).  That would let me avoid these problem without ripping up too much of my existing code (which is based on using HttpClient).  One may run into problems with not being able get at the stuff needed to properly interact with the existing HttpClient code.  I've run into that trying to work around internationalization problems with outgoing User-Agent headers and incoming Date headers.

    Wednesday, March 12, 2014 11:34 PM
  • henric:

    For what it's worth, we're working around this bag of issues with a small custom http client of our own on top of StreamSocket.  We have the advantage for this particular scenario that we only need to talk to one known HTTP server.  :-)

    Wednesday, March 12, 2014 11:46 PM
  • henric:

    For what it's worth, we're working around this bag of issues with a small custom http client of our own on top of StreamSocket.  We have the advantage for this particular scenario that we only need to talk to one known HTTP server.  :-)

    Yeah, I may look at that now as well.  I had thought I should wait until after I heard more about WP8.1 in case they fixed this, but I've run into yet another problem that may be HttpClient's fault (it seems to hang sometimes when used to stream background audio; it could still be my code at fault, but that is looking less and less likely).

    Since they were asking for input about future directions, I put a comment on the .NET Framework Blog's "Upcoming .NET NuGet Releases" post about this.

    Thursday, March 13, 2014 1:19 AM
  • The code you provided, without the blocking caused by the PleaseCrash check should work fine.  Are you seeing problems with it?  HttpWebRequest doesn't use the UI thread to my knowledge.

    Bret Bentzinger (MSFT) @awehellyeah

    Thursday, March 13, 2014 11:49 PM
  • I put an example on GitHub where you can see how a download (using HttpClient in this case) eats up all the available memory.   One can easily see that native memory disappears much faster than the download progresses.  Note that there is almost no managed memory used.

    https://github.com/henricj/MemoryWoes

    100MB of native raam used after reading less than 2MB.

    The download is very simple:

            async Task Download()
            {
                var buffer = new byte[4096];
    
                using (var httpClient = new HttpClient())
                {
                    using (var stream = await httpClient.GetStreamAsync("http://dds.cr.usgs.gov/pub/data/nationalatlas/elev48i0100a.tif_nt00828.tar.gz").ConfigureAwait(false))
                    {
                        for (;;)
                        {
                            var length = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
    
                            if (length < 1)
                                return;
    
                            Interlocked.Add(ref _totalBytesRead, length);
    
                            await DoSomethingWithData(buffer, 0, length).ConfigureAwait(false);
                        }
                    }
                }
            }
    

    And the status update looks like this:

        download.Text = string.Format("{0:F2} MiB downloaded", Interlocked.Read(ref _totalBytesRead).BytesToMiB());
    
        var gcMemory = GC.GetTotalMemory(true).BytesToMiB();
    
        status.Text = string.Format("GC {0:F2} MiB App {1:F2}/{2:F2}/{3:F2} MiB", gcMemory,
            DeviceStatus.ApplicationCurrentMemoryUsage.BytesToMiB(),
            DeviceStatus.ApplicationPeakMemoryUsage.BytesToMiB(),
            DeviceStatus.ApplicationMemoryUsageLimit.BytesToMiB());
    

    Friday, March 14, 2014 2:37 AM
  • Thanks for reporting this issue.   The Portable HttpClient library is a Nuget package, and I suggest accessing that site and contacting the owners to report that issue.https://www.nuget.org/packages/Microsoft.Net.Http

    With that said, for large file transfers in your application you should probably be using the background file transfer API's to let the OS handle this:

    http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202955(v=vs.105).aspx?


    Bret Bentzinger (MSFT) @awehellyeah

    Friday, March 14, 2014 4:59 PM
  • In my case, the data is for streaming media, so background file transfers aren't really appropriate (apart from the horrid startup lag and excessive bandwidth usage if only the first few seconds are played, some content owners aren't all that enthusiastic about extra copies of their data).

    Any other application that needs a large dataset may not want to use background transfers if what they really need is some transformed version of the data.  For example, if they are trying to populate a local database with a large CSV file that also needs an occasional lookup to some web service:  background transfers would at least double their flash storage requirements and it would blow up in the foreground if their web service wasn't faster than the CSV download's ability to chew up all the app's memory.

    The "Range:" header may help for some, at the cost of a great deal of extra code complexity, but not all servers support this.

    As other posters have pointed out, the problem seems to be with the platform's HttpWebRequest (which HttpClient uses under the hood). Although, I did put a note on the NuGet package owner's blog as well (Microsoft's .NET Framework Blog).

    Friday, March 14, 2014 6:32 PM
  • What kind of media are you streaming?  Audio, or Video?

    Have you tried the Media Streamer class?  If so, what is it missing that doesn't fit your needs?


    Bret Bentzinger (MSFT) @awehellyeah


    Friday, March 14, 2014 6:43 PM
  • This is for the Windows Phone Streaming Media project.  I've see it used for audio/video, audio only and even some video only streams.

    Mostly, it is for playing HTTP Live Streaming (.m3u8/.m3u).  MPEG-2 Transport Streams (straight .TS streams, not related to a playlist) and some live internet radio stations also work (MP3 and AAC, optionally through a .pls playlists).

    The out of memory problem is generally for the non HLS streams, since the individual HLS segments are usually fairly small.  Most non-HLS streams are live, so they tend not to download any faster than they play, but point the background audio player to a 50MB .TS file and it blows up pretty quickly...

    Friday, March 14, 2014 8:21 PM
  • ... but I've run into yet another problem that may be HttpClient's fault (it seems to hang sometimes when used to stream background audio; it could still be my code at fault, but that is looking less and less likely).

    This "other problem" with the hang appears to have been fixed in Update 2.

    (The out of memory problem is alive and well in Update 3.)

    Sunday, March 16, 2014 6:35 PM
  • We had similar problem. When disposing HttpClient, its response and stream - the memory still grows. So if i start downloading 20 files 20 mb each and instantly dispose all the stuff - it will still consume all memory and crash. The bug is in HttpClient code, it doesnt make HttpRequest.Abort when streaming. It doesn't event hold the reference to HttpWebRequest IIRC.

    Solution was to switch to HttpWebRequests and call Abort manually.

    Wednesday, April 16, 2014 6:16 AM
  • I think what you're saying is that if I block the read callback, the responseStream won't continue to download and buffer.  Which makes sense.

    However, in real life I'm just about to write to the StorageFile's stream using an async call.  Should that be a blocking call as well?

    Further, how do I now that it's okay to block this thread?  Could HttpWebRequest on WP potentially use the UI thread?

    @mfeingol

    your code example works for me but have a problem with resume. If i passing OFFSET not it works. Why?


    Wednesday, April 16, 2014 3:22 PM
  • I don't know. You'd need to give me a few more hints. :-)
    Wednesday, April 16, 2014 3:25 PM
  • I don't know. You'd need to give me a few more hints. :-)

    you have problem with offset?
    Wednesday, April 16, 2014 3:34 PM
  • I finally have something of a work-around.  I grabbed the HttpWebRequest code from Mono, converted it (mostly) to TPL, swapped the guts to use StreamSocket, and then I could do this:

    async Task DownloadAsync()
    {
        var buffer = new byte[4096];
    
        var webRequest = SM.Mono.Net.WebRequest.Create("http://dds.cr.usgs.gov/pub/data/nationalatlas/elev48i0100a.tif_nt00828.tar.gz");
    
        using (var response = await webRequest.GetResponseAsync(_cancellationTokenSource.Token).ConfigureAwait(false))
        {
            using (var responseStream = response.GetResponseStream())
            {
                for (; ; )
                {
                    var length = await responseStream.ReadAsync(buffer, 0, buffer.Length, _cancellationTokenSource.Token).ConfigureAwait(false);
    
                    if (length < 1)
                        return;
    
                    Interlocked.Add(ref _totalBytesRead, length);
    
                    await DoSomethingWithData(buffer, 0, length).ConfigureAwait(false);
                }
            }
        }
    }
    
    It doesn't consume all of the app's memory:

    Not chewing up 100MBNote that the code is barely tested (I have tried "GET"s to a couple of URLs), was turned inside out for TPL, and is still a bit of a mess (in the near term, I plan on ripping out more of the remaining APM-related gunk and making sure that cancellation tokens propagate everywhere they should).

    The code lives here:  https://github.com/henricj/SM.Mono.Net

    Wednesday, April 23, 2014 12:20 AM