locked
Removing large uploaded files on heap - Out of Memory Exception (System.Drawing) RRS feed

  • Question

  • User439975351 posted

    Hi all,

    I am currently adding image uploading to our site, I am allowing upto 50Mb images to be transferred. This all works well for the first 3 attempts but then I get an out of memory exception in System.Drawing. I took a snapshot and viewed the heap and noticed that the "raw" uploads were still sitting there:

    Is there a way to remove / dispose / destroy these items and will this solve my out of memory error?

    Screenshot of heap snapshot

    Snapshot

    Monday, May 23, 2016 10:18 PM

Answers

  • User303363814 posted

    ImageResizer will need to uncompress the input jpg file.  This will use an amount of memory 7-10 times the size of the jpg file.  Say, half a gigabyte in your case.

    The resized image needs to be in memory (uncompressed) at the same time.  Maybe you will need a full gigabyte of memory for the resizing operation, original jpg plus uncompressed bitmap plus uncompressed result plus compressed result..  Note that this is virtual memory (eg not the number of memory sticks you have plugged into the machine).

    Are you running the process in 64-bit mode?  This is the first thing to check and set.  Get whoever does your web hosting to find out if this is the case and  set it up if not.  The change to 64-bit may be enough.  If not you may need to check that you have a sufficiently large working set and that there is sufficient backing file.  Your system administrator should be able to set this up for you.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Tuesday, May 24, 2016 1:29 PM

All replies

  • User-434868552 posted

    @1jus

    you've not shown us your code snippet.

    if you're not using FTP, i suggest you consider doing so:

    https://msdn.microsoft.com/en-us/library/ms229715(v=vs.110).aspx 

      "How to: Upload Files with FTP"

    i'm just guessing that FTP while possibly slower "may be" better for dealing with very large files.

    http://stackoverflow.com/questions/717200/comparing-http-and-ftp-for-transferring-files  

    Tuesday, May 24, 2016 12:03 AM
  • User303363814 posted

    The count is 1.  Does that just mean that only the last upload is present?  You are going to need to be able to handle having one image in memory.  The size is 25MB.  How much memory on your computer?  16,000 Megabytes maybe?  Why do you think that 25MB would be a problem?

    Hard to debug your code without seeing the code ....

    Tuesday, May 24, 2016 12:48 AM
  • User-1636183269 posted

    Best way declare any image or object with null. It will consider as temporary and will not store

    string foo = null;

    http://stackoverflow.com/questions/6066200/what-is-the-correct-way-to-free-memory-in-c-sharp

    Tuesday, May 24, 2016 1:16 AM
  • User-271186128 posted

    Hi 1jus,

    I suggest you could check the web.config file and make sure you have increase the maximum file size by modify the maxAllowedContentLength property.

    Besides, you can refer to the following links about this problem:

    http://ajaxuploader.com/large-file-upload-iis-asp-net.htm

    http://forums.asp.net/t/1528563.aspx?file+upload+and+system+OutOfMemoryException

    http://forums.asp.net/t/2092631.aspx?Out+of+Memory+Error+When+Uploading+Large+Files+Through+Handler

    Best regards,
    Dillion

    Tuesday, May 24, 2016 5:29 AM
  • User439975351 posted

    Sorry, there's a fair bit of code end to end but here's what I have:

    Form & JavaScript (not JQuery) to submit form & image:

    <div class="modal-header">
               <button type="button" class="close" data-dismiss="modal">×</button>
               <h3 class="modal-title">@title</h3>
           </div>  
     
           using (Html.BeginForm(formAction, "Admin"FormMethod.Post,                                                    
                                                       new { encType = "multipart/form-data", id = "imageUpload", name = "imageUpload" }
                                                       ))
           {
               @Html.AntiForgeryToken()
               <div class="modal-body">
     
                   <input id="guid" type="hidden" name="Guid" value="@ViewBag.Guid" />
                   <input id="imageTypeId" type="hidden" name="ImageTypeId" value="@ViewBag.ImageTypeId" />
     
                   @if (formAction == "EditImage")
                   {
                       @Html.HiddenFor(model => model.Id)
                   }
     
                   @if (formAction == "AddImage")
                   {
                       <div class="form-group">
                           @Html.Label("Select Image"new { @class = "control-label col-md-3" })
                           <div class="col-md-9">
                               <input id="imageUpload" type="file" name="upload" />
                           </div>
                           <div class="col-md-3"></div>
                           <div class="col-md-9">
                               <br>
                               <progress min="0" max="100" value="0">0% complete</progress>
                           </div>
                       </div>
                   }               
     
                   <div class="row">
                       <div class="col-sm-12">
                           <div class="form-group">
                               @Html.LabelFor(model => model.SortOrder, "Manual Sort Order (overrides sort by date)")
                               @Html.EditorFor(model => model.SortOrder, new { htmlAttributes = new { @class = "form-control" } })
                               @Html.ValidationMessageFor(model => model.SortOrder)
                           </div>
                       </div>
                   </div>
     
                   <div class="row">
                       <div class="col-sm-12">
                           <div class="form-group">
                               @Html.LabelFor(model => model.IsActive, "Is Active")
                               @Html.EditorFor(model => model.IsActive, new { htmlAttributes = new { @class = "form-control" } })
                               @Html.ValidationMessageFor(model => model.IsActive)
                           </div>
                       </div>
                   </div>
     
               </div>
               <div class="modal-footer">
                   <button type="button" class="btn btn-pink" data-dismiss="modal">Cancel</button>
                   <button type="submit" class="btn btn-pink">Save</button>
               </div>
           }
     
           <script>
               $(document).ready(function () {
     
                   $.validator.unobtrusive.parse("#contextForm");
     
               //XHR Uploader
               if ($("#imageUpload").length) {
                   var formAction = $("#imageUpload").attr("action");
                   var form = document.forms.namedItem("imageUpload");
                   form.addEventListener("submit", function (ev) {
     
                           var oOutput = document.getElementById("contextForm"),
                               oData = new FormData(form);
     
                           var oReq = new XMLHttpRequest();
                           oReq.open("POST", formAction, true);
                           oReq.onload = function (oEvent) {
                               if (oReq.status == 200) {
                                   oOutput.innerHTML = oReq.response;
                               } else {
                                   oOutput.innerHTML = oReq.response;
                               }
                           };
                           // Listen to the upload progress.
                           var progressBar = document.querySelector('progress');
                           oReq.upload.onprogress = function (e) {
                               if (e.lengthComputable) {
                                   progressBar.value = (e.loaded / e.total) * 100;
                                   progressBar.textContent = progressBar.value; // Fallback for unsupported browsers.
                               }
                           };
     
                           oReq.send(oData);
                           ev.preventDefault();
                       }, false);
                   };
     
               });
           </script>
       }

    Controller:

    [HttpPost]
           [ValidateAntiForgeryToken]
           public ActionResult AddImage([Bind(Include = "Guid,ImageSource,SortOrder,IsActive,ImageTypeId")]ImageModels viewModel, HttpPostedFileBase upload)
           {
               if (ModelState.IsValid)
               {
                   try
                   {
                       if (upload != null && upload.ContentLength > 0)
                       {
                           var maxSize = 52428800//50Mb
     
                           if (upload.ContentLength > maxSize)
                           {
                               upload.InputStream.Dispose();
                               ViewBag.Status = "failed";
                               ViewBag.Exception = "Image size must be less than 50Mb";
                               return PartialView("_ImageForm", viewModel);
                           }
     
                           var fileName = Guid.NewGuid().ToString() + ".jpg";
                           var path = Path.Combine(Server.MapPath("~/content/images/cms/"), fileName);
                           ImageResizer.ImageJob img = new ImageResizer.ImageJob(upload, path, new ImageResizer.Instructions("width=2000;height=2000;format=jpg;mode=max"), truefalse);
                           img.Build();
     
                           upload = null; //Tested, makes no difference

    //GC.Collect - Tested, no makes difference                        viewModel.ImageSource = fileName;                    }                    db.Images.Add(viewModel);                    db.SaveChanges();                    ViewBag.Status = "success";                }                catch (Exception e)                {                    ViewBag.Status = "failed";                    ViewBag.Exception = e.ToString();                    return PartialView("_ImageForm", viewModel);                }            }            return PartialView("_ImageForm", viewModel);        }

    My web.config has:

        <httpRuntime targetFramework="4.6" maxRequestLength="52428800" executionTimeout="1600" requestLengthDiskThreshold="52428800" />
      <security>
          <requestFiltering>
            <requestLimits maxAllowedContentLength="52428800" />
          </requestFiltering>
        </security>

    The screenshot of memory on heap was with 1 upload but I can get 3 before hitting probs, so this figure can hit 80Mb ish before hitting problems. Dev machine does have 16Gb yes, so I myself don't really understand why 80Mb would hinder a machine with adequate RAM. I have obviously Googled this to death but no fix, have tested setting to null & also manually running Garbage Collection but neither affect this raw upload figure nor fix the out of memory exception.

    Tuesday, May 24, 2016 8:13 AM
  • User753101303 posted

    Hi,

    So it happens in ImageResizer? If you have a Dispose method on img, what if you call it?

    Tuesday, May 24, 2016 8:29 AM
  • User439975351 posted

    Hi there,

    It doest *seem* to be ImageResizer as this also happened with the WebImage helper hence I used ImageResizer to see if I could resolve the issue that way. Dispose source is included in the .Instructions of the ImageJob (set to true).

    The exception is in as follows:

    System.OutOfMemoryException: Out of memory. at System.Drawing.Graphics.CheckErrorStatus(Int32 status) at System.Drawing.Graphics.DrawImage(Image image, PointF[] destPoints, RectangleF srcRect, GraphicsUnit srcUnit, ImageAttributes imageAttr, DrawImageAbort callback, Int32 callbackData) at System.Drawing.Graphics.DrawImage(Image image, PointF[] destPoints, RectangleF srcRect, GraphicsUnit srcUnit, ImageAttributes imageAttr) at ImageResizer.ImageBuilder.InternalGraphicsDrawImage(ImageState state, Bitmap dest, Bitmap source, PointF[] targetArea, RectangleF sourceArea, Single[][] colorMatrix) at ImageResizer.ImageBuilder.RenderImage(ImageState s) at ImageResizer.ImageBuilder.Render(ImageState s) at ImageResizer.ImageBuilder.Process(ImageState s) at ImageResizer.ImageBuilder.BuildJobBitmapToBitmap(ImageJob job, Bitmap source, Boolean transparencySupported) at ImageResizer.ImageBuilder.BuildJobBitmapToStream(ImageJob job, Bitmap source, Stream dest) at ImageResizer.ImageBuilder.BuildJob(ImageJob job) at ImageResizer.ImageBuilder.Build(ImageJob job) at ImageResizer.ImageJob.Build() at MVC.Controllers.AdminController.AddImage(ImageModels viewModel, HttpPostedFileBase upload) in \Controllers\AdminController.cs:line 658


    Tuesday, May 24, 2016 8:40 AM
  • User303363814 posted

    ImageResizer will need to uncompress the input jpg file.  This will use an amount of memory 7-10 times the size of the jpg file.  Say, half a gigabyte in your case.

    The resized image needs to be in memory (uncompressed) at the same time.  Maybe you will need a full gigabyte of memory for the resizing operation, original jpg plus uncompressed bitmap plus uncompressed result plus compressed result..  Note that this is virtual memory (eg not the number of memory sticks you have plugged into the machine).

    Are you running the process in 64-bit mode?  This is the first thing to check and set.  Get whoever does your web hosting to find out if this is the case and  set it up if not.  The change to 64-bit may be enough.  If not you may need to check that you have a sufficiently large working set and that there is sufficient backing file.  Your system administrator should be able to set this up for you.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Tuesday, May 24, 2016 1:29 PM
  • User753101303 posted

    So you target 2000x2000 which seems already huge and the source image is? As it seems to use System.Drawing under the hood I would likely just test with that. You could also try to process a series of file taken from a folder maybe in a console app and see what happens (or create test images programmatically).

    Are you sure it fails after 3 images or could it be that it fails on particular images having for example an excessive heigh or width or maybe using  a particular format. I would also dispose myself even if a flag is supposed to take care of this.

    Tuesday, May 24, 2016 1:46 PM
  • User439975351 posted

    Thanks Paul, changing the config from Any CPU to x64 seems to have resolved it on my dev machine. Will test again on our staging server. We have our own admins so any changes should be straight forward (famous last words!).

    EDIT:

    Further reading last night on this subject revealed that its not the change from "Any CPU" to "x64" rather the underlying host environment (IIS or IIS Express) as Paul eluded to. So the actual fix to this problem in my environment was simply this little check box in Visual Studio:

    IIS Express - 64bit

    Tuesday, May 24, 2016 5:14 PM