none
Azure Storage Blob v/s ADLS read operation network bandwidth

    Question

  • Hello Team,

    We have a situation for building up Hadoop cluster on Azure VMs and would like to keep storage option either WASB or ADLS but we have would to clarify the read operation network bandwidth within the same region for pulling the data from these storage layers (Storage Blob, General purpose Storage Blob and Azure Data Lake Storage). Also, if I have 100 Azure VMs to pull the data from same storage layer (same account, same container), does it have any impact on the network bandwidth? Does network bandwidth distributes or work independently for each of the VMs?

    Regards,

    Sakhuja

    Thursday, November 23, 2017 5:16 AM

All replies

  • Both ADLS and Storage can scale to meet the bandwidth needs of the workload on the storage side. Bandwidth/throughput is set at the account level. Every VM also has an individual network adapter limit. The distribution also depends on how the workload schedules work on every VM. 
    Monday, November 27, 2017 6:33 PM
  • Hi Amit,

    Thank you for the response. Could you please let me know what is the basic storage read bandwidth on any pay as you go account (Example: 1/2/10 Gbps etc)? And also what is an approx. cost to increase it?

    Regards,

    Sakhuja

    Tuesday, November 28, 2017 12:43 AM
  • Hi Amit,

    I am still waiting for a response on it!

    Regards,

    Sakhuja

    Friday, December 1, 2017 5:31 PM
  • Hi Sakhuja,

    The limits for Azure Storage are here.  ADLS can scale to provide the necessary bandwidth for all analytics scenarios.  By default, an ADLS account provides enough bandwidth to meet the needs of a broad category of use cases.  If you run into the default limit, your ADLS account can be configured to provide more bandwidth by contacting Microsoft support.  Thanks.

    Stephen

    Monday, December 4, 2017 11:13 PM
  • Thank you Stephen! This is really helpful.

    Just to clarify, I have stored an object on WASB of 120 MB then it means that, I have consumed 2 blocks of a single blob with one as 100 MB and another one as 20 MB whereas I can go up to 50,000 blocks in a single blob. However, I can store other unlimited block blobs until I reach a limit of (500 TB - 120 MB) for the storage account.

    Regards,

    Sakhuja

    Tuesday, December 5, 2017 11:15 AM