locked
Slow upload to blob storage RRS feed

  • Question

  • I wonder if there's good ways to improve performance related to upload to blob storage, apart from running several parallell uploads. When deploying a new version of our Azure roles, we start by uploading the roles to blob storage. The package is currently 170MB. It takes us around 4 minutes to upload this, which means I'm uploading 0.7MB/sec (~5Mbit)

    I've tried uploading from several locations, each being 100mbit/s-upload internet connection, but still it takes a lot longer than I expect it to. When uploading to other web sites, I can get around 5MB/sec upload speed. Our package is 170MB, so I expect it to take at most a minute to upload.

    We upload blobs in parallell, but when uploading to the North EU data center from south Sweden this seems to give negliable effects. I've tried uploading with different block sizes from Cerebrata Storage Studio, but changing block size does not make any large changes to speed. (Splitting up uploads in parallell helped us improve performance when uploading to data centers in Asia though)

    Is there any way to make upload to blob storage faster, or is it just this slow?


    Nitramafve

    Friday, September 21, 2012 9:50 AM

Answers

  • Hi,

    >>It takes us around 4 minutes to upload this, which means I'm uploading 0.7MB/sec (~5Mbit)

    I ran a quick test and I get 12 Mbit, (1.5 MB/sec) with a single thread, 15 Mbit, (1,9  MB/sec) with 10 threads. I'm also on a 100 Mbit line and uploading to North Europe.

    Regards,

    Alan


    Free EBook: "Windows Azure Service Bus Developer Guide" http://www.cloudcasts.net/devguide/

    Sunday, September 23, 2012 5:08 PM
  • Hi,

    Is there a lot of static content in your web sites, such as images, CSS files etc. If so you might want to consider storing those files in Blob storage if the containr is public you can use HTTP uri's to access the content (make sure the Blob content type is set accordingly). This could help to keep the siz of your deployment packages down, and improve the speed of the deployment cycle. You will incur transaction charges on Azure Storage, but with the recent price reductions it should not affect your billing that much.

    Regards,

    Alan


    Free EBook: "Windows Azure Service Bus Developer Guide" http://www.cloudcasts.net/devguide/

    Friday, September 21, 2012 2:10 PM

All replies

  • When we deploy to Azure, we first upload our deployment (cspkg, cscfg) to blob storage. After that, we call the service management API in Azure to create a deployment in the staging area (we call CreateOrUpdateDeployment). Our package is 170 MB, consists of 2 medium-size worker roles and 2 medium-size web roles.

    After this, it takes between 20 and 30 minutes for Azure to create the deployment and put them in Stopped status.

    I find it strange that it would take this long time just to create the new deployments, or is it normal?


    Nitramafve

    • Merged by Jiang Yun Wednesday, September 26, 2012 2:11 AM
    Friday, September 21, 2012 10:15 AM
  • Hi,

    Is there a lot of static content in your web sites, such as images, CSS files etc. If so you might want to consider storing those files in Blob storage if the containr is public you can use HTTP uri's to access the content (make sure the Blob content type is set accordingly). This could help to keep the siz of your deployment packages down, and improve the speed of the deployment cycle. You will incur transaction charges on Azure Storage, but with the recent price reductions it should not affect your billing that much.

    Regards,

    Alan


    Free EBook: "Windows Azure Service Bus Developer Guide" http://www.cloudcasts.net/devguide/

    Friday, September 21, 2012 2:10 PM
  • Hi,

    I've foud parallel uploading to give good performance gains for uploading to blobs. I'm also based in Sweden (Stockholm) and use a 100 MB connection. I can run some tests and see what bandwidth I get over the weekend.

    (Off topic, if you are based in Sweden, will you be attending CloudBurst? http://azureug.se/cloudburst2012 )

    Regards,

    Alan


    Free EBook: "Windows Azure Service Bus Developer Guide" http://www.cloudcasts.net/devguide/

    Friday, September 21, 2012 2:18 PM
  • We are considering to place some of our content in blob storage. For instance, we are considering moving all the online documentation (html+css+flash videos, etc) to blob storage to speed up deployment.

    But our entire package is only 170MB. The documentation contains roughly 1400 files of a total of 40MB. It would be strange if reducing the package size by say 40MB would make any significant change to the deployment speed. I mean, during deployment the package is stored in blob storage. Blob storage performance internally in Azure is somewhere around 50MB/sec, so trying to optimize the package by reducing the size by a few MB should not have any real effect, right?

    We also have a couple of VM roles running in Azure. If we tell Azure to spin up a new deployment in staging area with 10 instances, this takes *less* time, around 15 minutes or so. Even though the image for them VM roles are ~10-15GB.

    I get the feeling that there is something going on which I don't understand at all. Do you really think reducing the package size from 170 to say 100 MB would make a significant improvement in the time it takes to deploy to Azure? And if so, how could that even be possible, unless Azure relies on some 2k4 baud modem for internal communication?

    • Edited by M. Knafve Friday, September 21, 2012 5:07 PM
    Friday, September 21, 2012 4:48 PM
  • I would appreciate if you could run some test from your side to see what performance you get. Maybe I'm just being unlucky with the machines I'm using. Of course, it's up to you. If you don't have time, or maybe either way, I'll register a case with Microsoft to get some kind of indication on what performance to expect when accessing Azure storage from outside.

    (I won't be participating in CloudBurst. While I would enjoy to listen to the crazy german guy (Christian Weyer) - and maybe you - I usually prefer to fast-forward through online sessions after events. I've been working full time with Azure for two years and to be honest I feel the last Azure-related events I've been on has been on quite a basic level. If it were located closer to my home town, I would drop by though.)


    Nitramafve

    Friday, September 21, 2012 4:58 PM
  • Hi,

    >>I would appreciate if you could run some test from your side to see what performance you get.

    My demo involves uploading about 3.5 GB to Blob storage, so I will be testing that quite a bit over the next few days, I'll let you know how it goes.

    >>I usually prefer to fast-forward through online sessions after events.

    We will hopefuly get the sessions on-line shortly after the event.

    Regards,

    Alan


    Free EBook: "Windows Azure Service Bus Developer Guide" http://www.cloudcasts.net/devguide/

    Friday, September 21, 2012 5:04 PM
  • I've spent some hours shrinking the size of the package we deploy to Azure. For instance, I've removed the online documentation (thinking about setting up a few Azure Websites in different data centers to host the online docs instead, seems like a good fit). I was able to reduce the size of the package from roughly 170MB down to 70MB. During this process, I've deployed 8 times to see what effect the MB-reduction has on the deployment time.

    Weirdly enough, the time it takes for me to deploy has gone down almost linearly with the size of the package. What before used to take 35 minutes now takes roughly ~18 minutes. 

    As the package size has decreased from 170MB to 70MB, the number of files in the package has decreased as well. The documentation was split up into ~1400 files which are now gone, which means that the number of files in our deployment package (=cspkg) has been cut with ~70% or so.

    Now, it could be that the deployment time has improved because it's weekend and less usage in the data centers. Will continue to run tests during the upcoming week to confirm that it's an actual reduction.

    Does anyone have any idea of why a decrease of only 100MB could cut the deployment time in with more than 15 minutes?


    Nitramafve

    Sunday, September 23, 2012 12:11 PM
  • Hi,

    >>It takes us around 4 minutes to upload this, which means I'm uploading 0.7MB/sec (~5Mbit)

    I ran a quick test and I get 12 Mbit, (1.5 MB/sec) with a single thread, 15 Mbit, (1,9  MB/sec) with 10 threads. I'm also on a 100 Mbit line and uploading to North Europe.

    Regards,

    Alan


    Free EBook: "Windows Azure Service Bus Developer Guide" http://www.cloudcasts.net/devguide/

    Sunday, September 23, 2012 5:08 PM
  • Thanks for the feedback. Will check that there is not an issue with my own Internet connetion.

    Nitramafve

    Sunday, September 23, 2012 5:16 PM