locked
Waiting for files to arrive RRS feed

  • Question

  • Hi,

    We are setting up an environment for the first time.

    In our Azure stack we have

    1. Azure data Lake

    2. Azure data factory

    3. Azure DW

    A process (yet to be defined) will transfer files to the Azure data lake with a file format <prefix>ddmmyy.csv These files we need to process and load into our data warehouse.

    My question is that the files arrive at different times in the night, it could be 3am or it could be 6am. I beleive in Azure data Factory if you mark a dataset as external the timeslice will run regardless.

    Is there any way to wait for a file to exist before continuing a workflow either in ADF or U-SQL?

    Thanks

    Thursday, August 3, 2017 7:29 AM

All replies

  • yes, the external slices will always run at the defined time

    however, you could use the Offset-property to always start at e.g. 6AM 
    another idea would be to use retries but I dont know if you can configure your activity to throw an error if no file is found?

    -gerhard


    Gerhard Brueckl
    blogging @ http://blog.gbrueckl.at
    working @ http://www.pmOne.com

    Friday, August 4, 2017 6:54 AM
  • Yes.

    You will have to setup Retry attempts on a certain interval.

    Check the "External datasets" section @ https://docs.microsoft.com/en-us/azure/data-factory/data-factory-create-datasets

    Also, Provide Input Dataset name in your pipeline which will make sure this is PreReq

    Regards,

    Nimesh

    Tuesday, August 8, 2017 1:04 PM