none
how can Much AUS reqired for how much data processing in data lake analytics

    Question

  • hi,

    i have requirement,

    my customer ask me how i justify for how much data volume how much AUS  i recommended for processing(best Practices).

    also how can i achieve dynamically assign Aus base on my file size.

    for example

    if filesize>1 MB

    then aus=1

    eles if filesize>100MB

    then aus=2

    elseif filesize>1000MB

    then aus=5

    end

    Saturday, May 20, 2017 8:29 AM

All replies

  • Hi

    Azure Data Lake VS tools has provided the capability for customers to decide what's the best number AU for this job. We also have a build-in model so that customers can play with it to understand the execution time vs number of AUs. You can suggest customers to run a job once and use this tool to analyze it.

    More details can be found here:

    https://blogs.msdn.microsoft.com/azuredatalake/2016/10/12/understanding-adl-analytics-unit/

    The number of vertices is a compiler decision by considering multiple factors, like number of files and file size. If you have multiple big files, we would suggest from 1 AU for 1 GB data.

    Thanks

    --Alicia

    Monday, May 22, 2017 2:28 AM