none
Auto Scaling - ADLA

    Question

  • Hi,

    I am using ADF v2 with Azure Data Lake Analytics.I am running USQL code from ADF v2. Is there anyway to scale ADLA compute automatically based on the data volume it is processing? Please advise.

    Thanks,

    Soumitra.

    Wednesday, August 1, 2018 11:09 PM

All replies

  • Hi Soumitra,

        ADLA dynamically scales and allocates resources according to its size to process the data. Please take a look at this documentation.


    Thursday, August 2, 2018 12:02 AM
    Moderator
  • You bet there is.  When you schedule the job, you can choose to assign more or less AUs to the job.   In the ADF code it's called degree of parallelism.  https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-data-lake-analytics 

    Guy

    Thursday, August 2, 2018 12:04 AM
  • Hi Guy,

    Is there anyway I can change degree of parallelism automatically based on data volume in ADF V2?

    Thanks,

    Soumitra.

    Thursday, August 2, 2018 5:16 PM
  • Hi Soumitra

    Currently there is no auto scaling. If you can predict approximate data volumes, you should write different pipelines. E.g., one ADF pipeline that handles the high-volume days and one that handles the low-volume days.


    Michael Rys

    Friday, August 3, 2018 10:49 PM
    Moderator