locked
File download in Chunks RRS feed

  • Question

  • i am trying to download a file from 

    Error code 400 while downloading a file via REST API

    in the page there is pythin code that uses Range header to do so.

    i tried to implement it in C#, but i only get the beginning of the file over and over :

            static private void download_file(string filename, string token, int fileSize, int chunkNumber, long totalBytesRead)
            {
                string run_date = DateTime.Now.ToString("MMddyyyy").ToString();
                HttpManager httpManager = new HttpManager();
                string filePath = $"c://Temp//{filename}.xml";
                if (chunkNumber == 0 && File.Exists(filePath))
                    File.Delete(filePath);
                string url = "https://" + ip_address + "/mgmt/tm/asm/file-transfer/downloads/" + filename + ".xml";
                try
                {
                    HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
                    httpRequest.Method = "GET";
                    try
                    {
                        httpRequest = (HttpWebRequest)WebRequest.Create(url);
                        httpRequest.Method = "GET";
                        //httpRequest.AllowReadStreamBuffering = false;
                        httpRequest.Headers.Add("X-F5-Auth-Token", token);
                        if (chunkNumber > 0) {
                            int endRange = (chunkSize * (chunkNumber + 1)) - 1;
                            if (endRange > fileSize) endRange = fileSize;
                            httpRequest.AddRange(chunkSize * chunkNumber, endRange);
                        }
                        using (HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse())
                        {
                            using (Stream responseStream = httpResponse.GetResponseStream())
                            {
                                using (FileStream localFileStream =
                                    new FileStream(filePath, FileMode.Append))
                                {
                                    var buffer = new byte[4096];
                                    
                                    int bytesRead;
                                    int totalBytesRead_ = 0;
                                   while (totalBytesRead_ < httpResponse.ContentLength )
                                    {
                                        bytesRead = responseStream.Read(buffer, 0, buffer.Length);
                                        totalBytesRead += bytesRead;
                                        totalBytesRead_ += bytesRead;
                                        localFileStream.Write(buffer, 0, bytesRead);
                                    }
                                }
                            }
                        }
                    }
                    catch (Exception ex)
                    {
                        throw;
                    }
                }
                catch (Exception ex)
                {
                    throw;
                }
                if (fileSize > totalBytesRead * (chunkNumber+1))
                    download_file(filename, token, fileSize, ++chunkNumber, totalBytesRead);
    
            }//end of download_file
        }

    what can be wrong?


    • Edited by want 2 Learn Wednesday, September 2, 2020 6:47 PM spelling
    Wednesday, September 2, 2020 2:49 PM

All replies

  • That's an interesting approach to downloading a chunked file. For the most part it looks reasonable albeit a little non-standard. The fact that you're recursively calling yourself is going to cause potential problems for very large files though. It also isn't really necessary.

    Here's a reworked version. Unfortunately I don't have any chunking APIs so I cannot verify it is working correctly so you'll need to run it through its paces to make sure it is correct.

    private static void download_file ( string targetPath, string filename, string token )
    {
        string run_date = DateTime.Now.ToString("MMddyyyy");
    
        var targetFile = Path.Combine(targetPath, filename);
    
        //This will overwrite the existing file or create a new one
        using (var output = File.OpenWrite(targetFile))
        {
            //TODO: Make the API call - note you should be using HttpClient, not HttpWebRequest...
            //Ideally make this a standalone API client class to handle all the details for you
            var url = $"https://{ip_address}/mgmt/tm/asm/file-transfer/downloads/{filename}.xml";
            var httpRequest = WebRequest.Create(url);
            httpRequest.Method = "GET";
            httpRequest.Headers.Add(HttpRequestHeader.ContentType, "application/octet-stream");
            httpRequest.Headers.Add("X-F5-Auth-Token", token);
    
            //Start looping until we've read the entire file
            var totalRead = 0;
            var fileSize = 0;
    
            //TODO: Should store this separately so it is easy to change
            const int maxRead = 1_048_576;
    
            do
            {
                //If we are in the middle of a download then send the range of data we need
                if (totalRead > 0)
                {
                    //Calculate the range to request
                    var endRange = Math.Min(totalRead + maxRead, fileSize);
                    var range = $"{totalRead}-{endRange}/{fileSize}";
    
                    //Docs have this as a bug because content-range is server side, not client
                    //TODO: This may blow up at runtime because of use of response header in request, if so then
                    //use string version instead.
                    httpRequest.Headers.Remove(HttpRequestHeader.ContentRange);
                    httpRequest.Headers.Add(HttpRequestHeader.ContentRange, range);
                };
    
                using (var response = httpRequest.GetResponse() as HttpWebResponse)
                {
                    //TODO: Make sure the response was successful
                    if (response.StatusCode != HttpStatusCode.OK)
                        throw new Exception("Request failed");
    
                    //Copy all the data to the output
                    response.GetResponseStream().CopyTo(output);
    
                    //If the file size hasn't been set yet then get it from the Content-Range of the response
                    if (fileSize == 0)
                    {
                        //TODO: Not doing any error checking here
                        var rangeHeader = response.Headers[HttpResponseHeader.ContentRange];
                        if (!Int32.TryParse(rangeHeader.Split('/')[1], out fileSize) || fileSize < 0)
                            throw new Exception("Invalid file size in response");
                    };
                };
            } while (totalRead < fileSize);
        };
    }    


    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, September 2, 2020 3:33 PM
  • Hi,

    i fixed some code, it works for the first iteration and in the second iteration

    it fails and throw error on :

                    if (response.StatusCode != HttpStatusCode.OK)
    


    the response object is with errors

    Wednesday, September 2, 2020 7:36 PM
  • I'm guessing on their success code. Look at the response object when it fails. The reason phrase should have the error message (if any) and status code tells what went wrong. Also look at the header being sent and see if anything is wrong. The response body might have information as well.

    In my experience the easiest way to get a feel for an API is to call it from Postman or curl first so you can see the requests and responses and quickly play around with headers and bodies. Once you have that working then convert it to code.


    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, September 2, 2020 7:47 PM