locked
Is it possible to append data for file in ADLS using ADF copy activity? RRS feed

  • Question

  • Hi

    I'm trying to create copy activity ADLS as sink, but data needs to be append to existing file rather than writing to new output file

    Thanks

    -Kishore-

    Saturday, July 8, 2017 2:58 PM

All replies

  • Hi Kishore,

    If you refer to https://docs.microsoft.com/en-us/azure/data-factory/data-factory-azure-datalake-connector and search for "MergeFiles", you can see how you can combine the files you are copying, rather than create new ones. Does this help?

    Thanks,
    Sachin Sheth
    Program Manager
    Azure Data Lake.

    Saturday, July 8, 2017 3:42 PM
  • hi Sachin,

    Thanks for reply

    i tried that option, it seems that works only for merging files in Source directory from in that iteration. but what i'm looking here is bit different. I do have pipeline which is running on every hour, for each run it should read file from Source directory and it has to merge in existing file in Azure Data Lake Store rather overwriting for each run. 

    is it possible to achieve?

    Thanks

    -Kishore-

    Monday, July 10, 2017 5:58 PM
  • Hi Kishore,

    I'm afraid this is not possible by default. I am involved in a project where we append files using a script that runs on an Azure Batch compute node (VM) that is executed from ADF using a custom .NET activity.


    Jorg Klein's Microsoft Business Intelligence Blog

    Wednesday, July 12, 2017 11:37 AM
  • Hi Jorg, could you provide the script for appending? Im with a similar issue for Blob Storage.
    Wednesday, October 17, 2018 6:57 PM
  • @Kishore , did you find solution for your scenario?
    Sunday, April 12, 2020 3:41 PM
  • Hi there,


    Unfortunately the Copy Activity doesn't support append behavior. Copy activity currently support merge files behavior when the source is files from a file-based data store (Merges all files from the source folder to one file). 

    I request you to provide this valuable suggestion at our feedback in ADF user voice forum. All the feedback shared in this forum are actively monitored and reviewed by ADF engineering team. 
    Here is the ADF feedback forum link: https://feedback.azure.com/forums/270578-data-factory

    Please do share your feature request suggestion  link here, once it is created. That way, other users with similar idea/suggestion can update/comment on it. 

    In addition to Jorg Klein sugggestion, one more possible workaround I can think of is using the merge file option in the copy activity. Move all the individual files to a folder in different runs and merge them to a single file and delete the individual files (as per requirement) using a Delete activity. This workaround will require two more activities to the your pipeline. 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Monday, April 13, 2020 8:22 PM