locked
Multiple File Download from Azure Blob Storage RRS feed

  • Question

  • Hi,

    I have an issue for downloading the multiple files (dox, pdf, xls, etc.) from any given container. The list of the files are shown and user selects multiple files to be downloaded sequentially or parallelly.

    Please, can anyone advice me best way to handle it and tutorial to follwo if there are any.

    Thank you.

    Monday, September 12, 2011 10:01 AM

Answers

  • There is no mechanism to simply download an entire folder, instead you will need to perform many simultaneous operations for each file. If you are considering a large amount of smaller files you could consider using an intermediate role to compress them into a single file to download, but this would only be appropriate in a smaller subset of scenarios. 

    If you are going to be downloading multiple files concurrently, a key concern here is going to enforce some limit to parallelism at the application layer. If you simply try to download every file a user selects then you may encounter issues such as when a user selects 1000 files etc. 

    This can be done in many different ways, I would suggest you consider using the Async download methods ( Begin/End DownloadXXXX) on CloudBlob and using an interlocked increment / decrement to limit concurrency. This will allow your application to programatically limit the number of concurrent operations (a good number would be the number of logical cores on the machine to get best overall throughput).

    There are other alternatives such as using the threadpool, using waitany, synchronizing on a reset event etc, but the end result should be some upper limit to the number of simultaneous transactions. Also it sound as if this action is occuring on some Client GUI, in which case please take care not to block the GUI thread as this will lead to a undesirable client experience.

    For some good reference code see the code in the following blog http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/23/windows-azure-storage-client-library-parallel-single-blob-upload-race-condition-can-throw-an-unhandled-exception.aspx - (Note: the issue referenced here has since been resolved in the latest sdk, however the code is useful to illustrate the concepts I am referring to above.)

     

    joe

    Saturday, September 17, 2011 8:33 PM

All replies

  • Hi, what's the specific issue that you're facing?

    Typically, download action is performed or trigger by client via browser.

    From the backend we could either:

    - give a specific URL

    - throw a response to browser

    When dealing with multiple file, you can also consider to zip them up together before enable it for download.


    regards, wely
    Monday, September 12, 2011 2:55 PM
  • There is no mechanism to simply download an entire folder, instead you will need to perform many simultaneous operations for each file. If you are considering a large amount of smaller files you could consider using an intermediate role to compress them into a single file to download, but this would only be appropriate in a smaller subset of scenarios. 

    If you are going to be downloading multiple files concurrently, a key concern here is going to enforce some limit to parallelism at the application layer. If you simply try to download every file a user selects then you may encounter issues such as when a user selects 1000 files etc. 

    This can be done in many different ways, I would suggest you consider using the Async download methods ( Begin/End DownloadXXXX) on CloudBlob and using an interlocked increment / decrement to limit concurrency. This will allow your application to programatically limit the number of concurrent operations (a good number would be the number of logical cores on the machine to get best overall throughput).

    There are other alternatives such as using the threadpool, using waitany, synchronizing on a reset event etc, but the end result should be some upper limit to the number of simultaneous transactions. Also it sound as if this action is occuring on some Client GUI, in which case please take care not to block the GUI thread as this will lead to a undesirable client experience.

    For some good reference code see the code in the following blog http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/23/windows-azure-storage-client-library-parallel-single-blob-upload-race-condition-can-throw-an-unhandled-exception.aspx - (Note: the issue referenced here has since been resolved in the latest sdk, however the code is useful to illustrate the concepts I am referring to above.)

     

    joe

    Saturday, September 17, 2011 8:33 PM
  • Thank you for your suggestion.
    bipin
    Monday, October 31, 2011 3:54 PM