locked
Dataset - Passing name of the file Dynamically RRS feed

  • Question

  • Hi ,

        I Trying to Build a ADF pipeline that will push files from On-prem to Azure Blob. The Files that get generated on-prem is of the form xyz_2015_07_29.tsv. Note that file name keeps changing every day. How to dynamically mention the file names in the type property?

    The $$Text.Format does seem to work or dynamically resolve during run time and I keep getting the error

    local/Processed/data/xyz_$$Text.Format('{0:yyyy}',SliceStart)_$$Text.Format('{0:MM}',SliceStart)_$$Text.Format('{0:dd}',Date).tsv) with error: Stream not found  error?

    I have tried with curly braces too.             

    "typeProperties": {

                "folderPath": "/local/Processed/data/",
                "fileName": "xyz_$$Text.Format('{0:yyyy}',SliceStart)_$$Text.Format('{0:MM}',SliceStart)_$$Text.Format('{0:dd}',SliceStart).tsv"

             },

    Wednesday, July 29, 2015 11:31 AM

Answers

  • You can use the partitionBy property in dataset. In your case, you can dynamically set the file name like this:

    "fileName": "xyz_{Year}_{Month}_{Day}.tsv",
    "partitionedBy": 
    [
        { "name": "Year", "value": { "type": "DateTime", "date": "SliceStart", "format": "yyyy" } },
        { "name": "Month", "value": { "type": "DateTime", "date": "SliceStart", "format": "MM" } },
        { "name": "Day", "value": { "type": "DateTime", "date": "SliceStart", "format": "dd" } }
    ],

    For more details, refer to this doc about the partitionBy property.

    Friday, July 31, 2015 3:26 AM