none
ADF in a Multi-Tenant Application RRS feed

  • Question

  • I have created a pipeline that processes files from ADL to Azure SQL.  The same pipeline I want to use for many tenants in platform.   Each customer of mine will get a dedicated folder in ADL.  Can ADF know to go through all client Folders and if file is found run the en tire piepline in parallel.

    Also the Azure SQL we have one DB and is controlled by Row Level Security.   so we want the least amount contention as we will not know when a Customer(s) will decide to upload files.  

    Best Regards

    Sunday, August 25, 2019 4:42 AM

All replies

  • Hello ipulsetv and thank you for you inquiry.  I have a suggestion for the 'work in parallel' portion, but the 'contention' portion is more difficult.

    For the 'work in parallel' portion, the general idea is to utilize two pipelines.  One pipeline will call the other to execute, so I will refer to them as 'parent' and 'child'.

    The parent pipeline will be given a list of client folders.  The parent pipeline will iterate over the list in a ForEach activity.  When the ForEach activity is run without the 'sequential' feature flag checked, it executes items in parallel.  Inside the ForEach activity, will be an Execute Pipeline activity.  This passes the client folder path to the child pipeline, which contains whatever buisness logic that finds and copies the file.

    As for the 'contention' portion, as long as the many pipelines are writing to the same single table, there will be contention issues.  Without knowing the details, there is nothing I can recommend.

    Tuesday, August 27, 2019 12:04 AM
    Moderator
  • Hi,

    Here as you are writing into Sql db on the same tables so i don't see much of the benefits unless you create staging tables for each client and then loading data into them in parallel before moving data into the target tables. 

    So create stage tables for each client, load them in parallel as Martin suggested

    then load data from stage tables into the target tables.


    Cheers,


    Tuesday, August 27, 2019 1:40 AM