locked
Azure Data Factory - Gett a List of Files and either raise or continue processing RRS feed

  • Question

  • Hi,

    Using Data Factory my pipeline gets a list of files and folders (using GetMetadata) and does processing on the list - copy then delete.

    I want to be able to set a pipeline parameter 'raise error on no files' to ether true or false - this I know how to do.

    How do I implement the below using Data Factory (latest version)...

    1) When the pipeline runs and there are no files to process and parameter 'raise error on no files' = true then an error is raise by the pipeline and it stops running.

    Regards

    Ian

    Thursday, August 15, 2019 9:00 AM

Answers

  • Ian did this help you?
    • Marked as answer by Caz71 Monday, August 19, 2019 8:21 PM
    Monday, August 19, 2019 4:38 PM

All replies

  • Hello Ian and thank you for your inquiry.  Let me break this down.

    You are using a GetMetadata activity which gets 'childItems'.  You want to raise an error on no files.  To get the number of files, we can use the dynamic content:

    @length(activity('Get Metadata1').output.childItems)

    When there are no files, the above returns a length of 0.

    Since we only care about when there are 0 files, it would make more sense to check whether the array is empty:

    @empty(activity('Get Metadata1').output.childItems)

    You want to force the pipeline to throw an error and fail.  This we can do by causing a datatype error.  To do this we can create a variable with datatype boolean, and assign it a value of datatype string.  This can be done like

    @if(greater(variables('numberOfFiles'),0), True, 'ICauseTypeError')

    Putting these two pieces together we get:

    @if(
      greater(
        length(activity('Get Metadata1').output.childItems),
        0),
      True,
      'ICauseTypeError')

    You want to only do this when your pipeline parameter 'raise_error_on_no_files' is set to True.  This is awkward to do with 'length', so lets use 'empty' instead.  If raise_error_on_no_files is true and childItems is empty, then cause type error, otherwise return True.

    @if(
      and(
        pipeline().parameters.raise_error_on_no_files,
        empty(activity('Get Metadata1').output.childItems)
      ),
      'ICauseTypeError',
      True
    )

    Friday, August 16, 2019 1:09 AM
  • Ian did this help you?
    • Marked as answer by Caz71 Monday, August 19, 2019 8:21 PM
    Monday, August 19, 2019 4:38 PM
  • Sorry for the delay. This was of great help. Thank you

    Regards

    Ian

    Monday, August 19, 2019 8:22 PM