locked
How to create condition on copy output property which may or may not exist RRS feed

  • Question

  • We are using some of the copy activity output properties to feed into a subsequent activity. One of the properties we're grabbing is the filesRead property. However, we're running into an error in the copy activity such that the filesRead property isn't being written in the output. (This error is a UserErrorAuthenticationFailure by the way - but this isn't the focus of this comment.)

    To handle this, we're trying to conditionally use this property in the subsequent activity, using the expression: 

    @if(contains(activity('CopyActivity').output, 'filesRead'), activity('CopyActivity').output.filesRead, 'NA')

    However, we are still getting the error message:

    The expression 'if(contains(activity('CopyActivity').output, 'filesRead'), activity('CopyActivity').output.filesRead, 'NA')' cannot be evaluated because property 'filesRead' doesn't exist, available properties are 'dataWritten, filesWritten, sourcePeakConnections, sinkPeakConnections, copyDuration, errors, effectiveIntegrationRuntime, usedDataIntegrationUnits, usedParallelCopies, executionDetails'

    even though we're using a conditional statement which is evaluating to false.

    It seems that the "if true" value (activity('CopyActivity').output.filesRead) is being evaluated even in the case where the contains expression evaluates to false. Maybe there is some validation being run on every expression in the activity's template before an activty is executed?

    Does anyone know how we can get around this?

    Thanks,

    Ed


    • Edited by ed.freeman Wednesday, August 28, 2019 11:02 AM
    Wednesday, August 28, 2019 11:01 AM

Answers

  • Ed, I have an even better solution for you.  Please try:

    @if(contains(activity('CopyActivity').output, 'filesRead'), activity('CopyActivity').output?.filesRead, 'NA')

    Wednesday, August 28, 2019 10:30 PM

All replies

  • Hello Ed and thank you for your inquiry.  I have reproduced your issue, and have two work-arounds.  It appears that Data Factory does not subscribe to 'lazy evaluation'.  I will make inquiries internally to confirm.

    One work-around is straightforward:  Use an IfCondition activity, with conditional 

    @if(contains(activity('CopyActivity').output,true,false)

    and inside the true and false activities do a SetVariable activity assigning the values @activity('CopyActivity').output.filesRead and 'NA' respectively.  The caveat is that IfConditions cannot be nested.

    The other work-around takes advantage of the 'on failure' dependency option.  Create two SetVariable activities.  The first is connected to the CopyActivity, and its value is: 

    @activity('CopyActivity').output.filesRead

    Now I know the origin of this problem, was not being able to use this statement successfully.  We can take advantage of this.  The second SetVariable activity is connected to the first SetVariable activity by 'on failure' dependency, and its value is 'NA'.  This way, when we are unable to assign the 'filesRead', we recover.

    Whatever activity follows the second SetVariable activity should be connected by both 'on success' and 'on skipped' dependencies.  This way, execution progresses in either case.

    Wednesday, August 28, 2019 7:36 PM
  • Ed, a much simpler answer just occurred to me.  This issue happens for you only when the copy activity fails, correct?  If this is the case, you can simply branch on whether the copy activity succeeds or fails, using the dependencies I mentioned in the previous posts.

    That is, have two SetVariable activities, one executes when copy fails, the other executes when copy succeeds.

    Wednesday, August 28, 2019 7:57 PM
  • Ed, I have an even better solution for you.  Please try:

    @if(contains(activity('CopyActivity').output, 'filesRead'), activity('CopyActivity').output?.filesRead, 'NA')

    Wednesday, August 28, 2019 10:30 PM
  • Hi Martin,

    Thank you for your responses! Sorry it took a couple of days to reply - I didn't seem to receive the message alert emails.

    I will try the final solution you mentioned. It looks very likely to work - I'm not sure whether it's in the docs anywhere, but if it's not - it should be :)

    I'll respond again once I've determined whether it's the solution, and mark it as the answer.

    Also , to address your second proposal - sometimes the filesRead property does actually appear when the copy activity fails. It totally depends when and why the activity fails. In my case here, the copying doesn't even begin because the activity has failed to connect to the data source, and therefore there is no filesRead property (although there are "filesWritten" and "dataWritten" output properties (which have blank values), which makes it strange that filesRead is omitted)... Anyway, when the activity fails during the data copying, the filesRead property is written in the output.

    Thanks again for your help,

    Ed

    Friday, August 30, 2019 11:16 AM
  • I agree, it should be in the documentation.  I asked around and was informed of the existence of a 'null-safe operator' (the ?).  I am actively working with the product team to add this to the documentation.
    Friday, August 30, 2019 5:27 PM
  • Thank you, Martin.

    I tested your proposed fix and it worked perfectly. I'll go ahead and mark the answer.

    Thanks for working on getting it in the docs.

    Regards,

    Ed


    Wednesday, September 4, 2019 4:18 PM