locked
Very Large File Transfers to Storage RRS feed

  • Question

  • I'm in a situation where we need to allow users to upload very large files (e.g. HD videos); but we can't require them to download a control (e.g. use Silverlight).

    I have no probems streaming an image to storage but these >GB files are not making it.  Any ideas?

    thanks.

     


    Fruber
    • Moved by DanielOdievichEditor Tuesday, September 28, 2010 10:29 PM forum migration (From:Windows Azure)
    Friday, September 17, 2010 8:06 PM

Answers

  • Fruber,

    I've been doing quite a bit of work in this space. There's a number of options, the basics involve chunking the file, uploading the blocks (sequentially or in parallel) and then committing the final (assembles the blocks into a single file). The MS-provided library is a good start, but you might hit some bumps. In my testing, I hit some errors wherein the default block size that MS uses (4MB) wouldn't upload prior to timing out so I had to reduce it down to a better working size (say, 1MB), etc.

    I also have seen some significant benefits in parallelizing my uploads (sometimes drastic improvement in overall transfer time). You can read more here:

    http://rob.gillenfamily.net/post/External-File-Upload-Optimizations-for-Windows-Azure.aspx

    Hope that helps some.


    Rob Gillen
    http://rob.gillenfamily.net
    http://www.csm.ornl.gov/
    http://www.go-planet.com
    • Marked as answer by Mog Liang Friday, September 24, 2010 9:59 AM
    Friday, September 17, 2010 10:15 PM

All replies

  • plupload.com might be a solution, Works without controls for modern browsers, and degrades nicely for not so modern ones. 
    Friday, September 17, 2010 8:18 PM
  • Fruber,

    I've been doing quite a bit of work in this space. There's a number of options, the basics involve chunking the file, uploading the blocks (sequentially or in parallel) and then committing the final (assembles the blocks into a single file). The MS-provided library is a good start, but you might hit some bumps. In my testing, I hit some errors wherein the default block size that MS uses (4MB) wouldn't upload prior to timing out so I had to reduce it down to a better working size (say, 1MB), etc.

    I also have seen some significant benefits in parallelizing my uploads (sometimes drastic improvement in overall transfer time). You can read more here:

    http://rob.gillenfamily.net/post/External-File-Upload-Optimizations-for-Windows-Azure.aspx

    Hope that helps some.


    Rob Gillen
    http://rob.gillenfamily.net
    http://www.csm.ornl.gov/
    http://www.go-planet.com
    • Marked as answer by Mog Liang Friday, September 24, 2010 9:59 AM
    Friday, September 17, 2010 10:15 PM
  • What's the difference between the StorageClientLibrary and the Managed Library - I assumed that the managed library would offer more flexibility - or did I misinterpret the difference between the two?

     

    thanks.

     


    Fruber
    Friday, September 17, 2010 10:21 PM
  • I'm not aware that there is a difference... I usually use the two terms as referring to the same thing.
    Rob Gillen
    http://rob.gillenfamily.net
    http://www.csm.ornl.gov/
    http://www.go-planet.com
    Friday, September 17, 2010 10:23 PM