locked
Azure Data Factory - Create Dynamic Sink/Destination Folder Structure RRS feed

  • Question

  • Hi,

    I have successfully created a Data Factory Pipeline that copies files from on-premise to a Data-Lake Folder called "roottest"

    Is it possible to have a dynamic folder structure created (using Data Factory) in the form /yyyy/mm/dd/hh if the folders do not exist under "roottest" ?

    Example - the folder structure....

      /roottest/2019/08/13/16

    *hh is 24 hour format and the files are copied to this folder

    Regards

    Ian

    Tuesday, August 13, 2019 3:21 PM

Answers

  • Yes you can do that , please add the below dynamic expression in the folder name of ADLS 

    @concat('roottest','/',formatDateTime(utcnow(),'yyyy'),'/',formatDateTime(utcnow(),'MM'),'/',formatDateTime(utcnow(),'dd'),'/',formatDateTime(utcnow(),'mm')  )

    and it should work . 

    Thanks Himanshu

    • Marked as answer by Caz71 Thursday, August 15, 2019 7:46 AM
    Wednesday, August 14, 2019 10:36 PM

All replies

  • Yes you can do that , please add the below dynamic expression in the folder name of ADLS 

    @concat('roottest','/',formatDateTime(utcnow(),'yyyy'),'/',formatDateTime(utcnow(),'MM'),'/',formatDateTime(utcnow(),'dd'),'/',formatDateTime(utcnow(),'mm')  )

    and it should work . 

    Thanks Himanshu

    • Marked as answer by Caz71 Thursday, August 15, 2019 7:46 AM
    Wednesday, August 14, 2019 10:36 PM
  • Thank you Himanshu this worked perfectly, thank you.
    Thursday, August 15, 2019 7:46 AM
  • Is it possible to use a variable or something other than a date string to partition the folder structure? I'm looking to copy files to ADL and dynamically put them in a folder based on a value within the file name. So, "IX_ABCD_20200101_11111.gz" would be placed in the folder "/ABCD".

    I tried using Get Metadata, but @activity isn't available when entering a dynamic folder name for ADL. Next I'm trying to use variables, but doubt that method too.

    • Edited by Scott Faculak Wednesday, March 25, 2020 4:09 PM Additional context
    Wednesday, March 25, 2020 4:08 PM
  • Hi Scott Faculak,

    Thank you for your query. Your ask seems to be different than the OP query. For high visibility and immediate assistance, I would recommend you to open a new thread, we have our engineers and other community members who would be happy to help. 

    Please do consider to share the new MSDN thread link here.



    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.


    Wednesday, March 25, 2020 7:16 PM