locked
Are there any performance limitations for containers with a lot of blobs? RRS feed

  • Question

  • I am about to start migrating 30 million images to Azure's blob storage.  Are there any performance issues I should be careful for?

    I plan to put everything in one container, and I plan to use the following naming convention.
    /MyContainer/{FileName}
    {FileName} = {GUID}/{Thumb}/{F or B}.jpg
    {Thumb} = Actual, Medium, Small, etc.

    For each GUID there will be 10 different JPGs.  We will start with about 3 million GUID's (30 million blobs), and plan to double that each year. 

    Any issues putting all that in the same container?

    Thanks,
    Tim

    Thursday, October 14, 2010 3:10 AM

Answers

  • Hello, I don't see any performance issues in your approach. The partition key of a blob is container name + blob name. That is, each blob has its own partition, so potentially they may be stored on different VMs (depending on the size), and thus client requests can be load balanced. Of course, if the blobs are stored in a public container, you can take advantage of CDN to further increase performance.

    One thing you need to take into account is a single storage account allows you to store up to 100TB data (combines blob, queue, and table). That is roughly 100,000MB. So depending on the size of your images, you may need multiple accounts.


    Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by timgetsch Thursday, October 14, 2010 7:00 AM
    Thursday, October 14, 2010 5:17 AM

All replies

  • You might want to look at the material on the Azurescope website.
    Thursday, October 14, 2010 4:07 AM
    Answerer
  • Hello, I don't see any performance issues in your approach. The partition key of a blob is container name + blob name. That is, each blob has its own partition, so potentially they may be stored on different VMs (depending on the size), and thus client requests can be load balanced. Of course, if the blobs are stored in a public container, you can take advantage of CDN to further increase performance.

    One thing you need to take into account is a single storage account allows you to store up to 100TB data (combines blob, queue, and table). That is roughly 100,000MB. So depending on the size of your images, you may need multiple accounts.


    Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by timgetsch Thursday, October 14, 2010 7:00 AM
    Thursday, October 14, 2010 5:17 AM
  • Thanks for the helpful info!  The Azurescope website is great too!  So nice to have active forums and good documentation.  
    Thursday, October 14, 2010 7:01 AM
  • Small math error that scared the ____ out of me:

    100 TB = 100,000 GB = 100,000,000 MB

    Best

    Martin

     


    -- My Azure Sites: www.eblizz.com , www.freshfugu.com Blog: martinatsunset.com Twitter: martin_sunset
    Thursday, October 14, 2010 7:50 AM