locked
Create Folder-Structure in Azure Data Lake Gen2 based on FileName RRS feed

  • Question

  • Hi,
    German: wie kann man in Azure Data Lake Gen2 Folders auf Basis des Dateinamens anlegen?
    English: How to create Gen2 Folders based on file name in Azure Data Lake?
    e.g.:  "2019-02-26.csv"  => gives Folder 2019  SubFolder 02   SubFolder 26   ...and so on 27, 28..
    Regards
    Nicole


    Answer from MSDN


    Tuesday, February 26, 2019 4:00 PM

Answers

    Hello Nicole.  I figured out a work-around.  In this solution the pipeline takes the datetime as a parameter.  You can modify to meet your needs.
    Here is my solution:

    1. Give the Azure Data Lake Storage Gen2 Dataset a parameter
    2. have pipeline parameter of datestring
    3. have pipeline variable for newpath
    4. Create "Set Variable" activity and connect to "Copy Data" activity
    5. In Set Variable activity:  replaces "-" with "/" in datestring, and store in newpath variable
    6. In Copy Data Activity: assign the newpath variable to the Dataset's parameter
    7. In Azure Data Lake Storage Gen2 Dataset, use the parameter in the File Path field


    Friday, March 8, 2019 7:21 PM

All replies

  • Hi Nicole,

    This does not work with ADF. For more details, you may refer the MSDN thread which addressing similar issue.

    The following properties are supported for Azure Data Lake Storage dataset:

    For more details, refer “Copy data to or from Azure Data Lake Storage Gen2 using ADF”.

    Hope this helps.

    Wednesday, February 27, 2019 10:57 AM

    Hello Nicole.  I figured out a work-around.  In this solution the pipeline takes the datetime as a parameter.  You can modify to meet your needs.
    Here is my solution:

    1. Give the Azure Data Lake Storage Gen2 Dataset a parameter
    2. have pipeline parameter of datestring
    3. have pipeline variable for newpath
    4. Create "Set Variable" activity and connect to "Copy Data" activity
    5. In Set Variable activity:  replaces "-" with "/" in datestring, and store in newpath variable
    6. In Copy Data Activity: assign the newpath variable to the Dataset's parameter
    7. In Azure Data Lake Storage Gen2 Dataset, use the parameter in the File Path field


    Friday, March 8, 2019 7:21 PM