locked
Server Running out of Memory of Issue with Network RRS feed

  • Question

  • User-374939276 posted

    Hello. I am looking for some suggestions or help regarding a weird issue I have been having. Thank you in advance.

    I need to upload about 1000 images to AWS S3 within a few seconds which works fine on the virtual dedicated server. But What happens is later the server seems to run out of memory but I can't see what is using the memory. With further testing it seems that this method actually tries to connect with 1000 connections simultaneously which causes 
    Windows server to stop responding. 

    Is it perhaps an issue with my code? 

    Screenshot of Server Memory Usage

    private async Task UploadInParallelAsync()
            {
                var arrayOfImageFiles = GetListOfMainImageFiles();
    
                List<Task<PutObjectResponse>> tasks = new List<Task<PutObjectResponse>>();
    
                foreach (var item in arrayOfImageFiles)
                {
                    string key = $"{AuctionGroupId}/{UploadFolder}/{item.Name}";
                    tasks.Add(PutObjectS3.UploadObjectAsync(item.FullName, key));
                }
    
                var results = await Task.WhenAll(tasks);
            }
    public static async Task<PutObjectResponse> UploadObjectAsync(string filePath, string key)
            {
                string _bucketName;
                IAmazonS3 s3;
                const string CACHECONTROL = "max-age=2592000";
                _bucketName = System.Web.Configuration.WebConfigurationManager.AppSettings["AWSBucket"];
                s3 = new AmazonS3Client(); //Amazon.RegionEndpoint.USWest2
    
                PutObjectRequest putData = new PutObjectRequest();
                putData.FilePath = filePath;
                putData.BucketName = _bucketName;
                putData.CannedACL = S3CannedACL.PublicRead;
                putData.ContentType = "image/jpeg";
                putData.Headers.CacheControl = CACHECONTROL;
    
                putData.Key = key;
                PutObjectResponse response = await s3.PutObjectAsync(putData);
                return response;
            }

    Thursday, November 15, 2018 1:27 AM

Answers

  • User475983607 posted

    Is it perhaps an issue with my code? 

    As far as I can tell, the code shown opens as many connections are there in the arrayOfImageFiles collection.  A better idea is simply uploading Many images in a single request using multipart/form-data.

    I recommend contacting AWS support as it appears you are using AWS libraries.  There might be a better library or overload that you can use.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, November 15, 2018 6:06 PM

All replies

  • User475983607 posted

    Is it perhaps an issue with my code? 

    As far as I can tell, the code shown opens as many connections are there in the arrayOfImageFiles collection.  A better idea is simply uploading Many images in a single request using multipart/form-data.

    I recommend contacting AWS support as it appears you are using AWS libraries.  There might be a better library or overload that you can use.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, November 15, 2018 6:06 PM
  • User-374939276 posted

    Thank you so much for your help. I will take your suggestion.

    I agree with your answer. So now I am going to do batches of 8 connections at a time instead. This seems to work better to not overload the server.

    What I ended up doing is this:

    private async Task UploadInParallelAsyncVersion2()
            {
                var arrayOfImageFiles = GetListOfMainImageFiles();
    
                await Task.Run(() =>
                {
                    Parallel.ForEach(arrayOfImageFiles, new ParallelOptions { MaxDegreeOfParallelism = 8 }, (item) =>
                    {
                        string key = $"{AuctionGroupId}/{UploadFolder}/{item.Name}";
                        PutObjectS3.UploadObject(item.FullName, key);                    
                    });
                });
            }

    Friday, November 16, 2018 12:15 AM