locked
Resource on Load, extract data and send data via Azure Data Facory V2 to Azure Function App RRS feed

  • Question

  • I am seeking resources on how to implement this scenario below.

    I found some documentation, but I cannot find steps 2 and 3 below

    Using Azure Data Facory V2 to

    1 Load a *.csv file from SFP server

    https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal
    https://docs.microsoft.com/en-us/azure/data-factory/connector-sftp

    2 extract rows from the *.csv file based on condition in column

    There might be three separate extracted data, thus three sepearte Azure Aunction App.

    3 send each extracted data to each Azure Function App

    • Edited by Pingpong689 Sunday, August 4, 2019 11:49 PM
    Sunday, August 4, 2019 11:48 PM

Answers

  • Hi there,

    If you have a small no. of rows in the dataset, you can do the following :

    • Create a connection to the csv file on the SFTP server by following the docs you mentioned in step 1.
    • Have a getMetadata or Lookup activity to read the file using the connection you created in the above step.
    • Chain a forEach activity to loop through the rows by specifying child as something like @activity('Lookup1').output.value. Chaining ensures that the forEach activity will be run only once the Lookup or GetMetadata activity is complete. 
    • Within the ForEach activity, create an "azure function activity" as a child activity.
    • For each iteration of the forEach activity, run the Azure function activity and pass the item using dynamic expression - @item().

    Hope this helps.

    Monday, August 5, 2019 8:56 AM

All replies

  • Hi there,

    If you have a small no. of rows in the dataset, you can do the following :

    • Create a connection to the csv file on the SFTP server by following the docs you mentioned in step 1.
    • Have a getMetadata or Lookup activity to read the file using the connection you created in the above step.
    • Chain a forEach activity to loop through the rows by specifying child as something like @activity('Lookup1').output.value. Chaining ensures that the forEach activity will be run only once the Lookup or GetMetadata activity is complete. 
    • Within the ForEach activity, create an "azure function activity" as a child activity.
    • For each iteration of the forEach activity, run the Azure function activity and pass the item using dynamic expression - @item().

    Hope this helps.

    Monday, August 5, 2019 8:56 AM
  • @ChiragMishra-MSFT

    Your answer looks good, I will mark as answer, and hope it work. I will come back for any question. Thanks. :)

    Monday, August 5, 2019 9:24 PM
  • Sure do :) Let me know if you face any challenges implementing this.
    Tuesday, August 6, 2019 4:10 AM