locked
ADF copy data with condition RRS feed

  • Question

  • Hi,

    I copy data from AWS S3 to ADLS with a Copy Data activity and it works fine.
    Now I want to merge the files from AWS S3 to a file with a certain size.

    The idea is:

    AWS S3 send data to ADF, I merge thoses files until it size is 258 MB and then I copy the 258 MB file to the ADLS.

    Is it possible and how can I achieve that ?

    Thursday, July 4, 2019 8:01 AM

Answers

  • Hi Alex,

    One way to do so would be as follows.

    Have a pipeline with the following structure :

    • A copy activity to copy data from AWS S3 to Blob Storage / any other store.
    • Chain a get Metadata activity to the copy activity to get the size of the file after copy. 
    • Chain an if Condition activity to have your condition based on the size of the file.
    • In the true condition of the if condition activity, have a copy activity to copy data from the blob storage to the ADLS.

    Hope this helps.


    Thursday, July 4, 2019 9:28 AM