none
Modifying data on the fly in Azure Data Factory Pipeline

    Question

  • I am doing a POC in Azure Data Factory and I am a newbie in it. This is for a pre-sales RFP. I have Azure Blob as source, Azure SQL as destination. Azure Blob gets a csv file uploaded from customer's place. I have used a Lookup Activity inside my pipeline first to get the collection of rows and then I am feeding it to ForEach activity so that I can process each record. Inside ForEach activity I have IfActivity which determines if customer is a senior citizen or not by evaluating AGE > 65 expression. In the SC field I need to put 'Yes' on the fly if expression evaluates to true or 'No' if false. I am not able to get how to modify this data on the fly. Initially from source I am getting the SC field as blank. Can we not modify data on the fly inside Azure Data Factory pipeline? Please help us.
    Wednesday, October 10, 2018 2:03 AM

Answers

  • In that case, you can use the Query option in the lookup activity and write the update statement in it to update your field post your copy activity, i.e. once the data is poured into the target SQL database table, you need to update the field. Hope it works for you


    Bhushan Gawale | RapidCircle

    • Marked as answer by Ray, Ambar Wednesday, October 10, 2018 8:13 AM
    Wednesday, October 10, 2018 6:27 AM

All replies

  • Hi,

    Is it possible to use Stored Procedure to modify the SC field? 

    You can append a stored procedure activity after the copy.

    Wednesday, October 10, 2018 2:35 AM
  • Hi Bo Xiao,

    Thanks for your prompt reply. In that case I need to create a stored procedure first in my Azure SQL database right? Is there a way I can avoid this? Is there any other way to do this. If I do not need to touch the Azure SQL database then I am fine with the solution.

    Wednesday, October 10, 2018 4:38 AM
  • In that case, you can use the Query option in the lookup activity and write the update statement in it to update your field post your copy activity, i.e. once the data is poured into the target SQL database table, you need to update the field. Hope it works for you


    Bhushan Gawale | RapidCircle

    • Marked as answer by Ray, Ambar Wednesday, October 10, 2018 8:13 AM
    Wednesday, October 10, 2018 6:27 AM