none
Is it possible to append data for file in ADLS using ADF copy activity? RRS feed

  • Question

  • Hi

    I'm trying to create copy activity ADLS as sink, but data needs to be append to existing file rather than writing to new output file

    Thanks

    -Kishore-

    Saturday, July 8, 2017 2:58 PM

All replies

  • Hi Kishore,

    If you refer to https://docs.microsoft.com/en-us/azure/data-factory/data-factory-azure-datalake-connector and search for "MergeFiles", you can see how you can combine the files you are copying, rather than create new ones. Does this help?

    Thanks,
    Sachin Sheth
    Program Manager
    Azure Data Lake.

    Saturday, July 8, 2017 3:42 PM
  • hi Sachin,

    Thanks for reply

    i tried that option, it seems that works only for merging files in Source directory from in that iteration. but what i'm looking here is bit different. I do have pipeline which is running on every hour, for each run it should read file from Source directory and it has to merge in existing file in Azure Data Lake Store rather overwriting for each run. 

    is it possible to achieve?

    Thanks

    -Kishore-

    Monday, July 10, 2017 5:58 PM
  • Hi Kishore,

    I'm afraid this is not possible by default. I am involved in a project where we append files using a script that runs on an Azure Batch compute node (VM) that is executed from ADF using a custom .NET activity.


    Jorg Klein's Microsoft Business Intelligence Blog

    Wednesday, July 12, 2017 11:37 AM
  • Hi Jorg, could you provide the script for appending? Im with a similar issue for Blob Storage.
    Wednesday, October 17, 2018 6:57 PM