Azure Data Factory Copy Activity on Failure | Expression not evaluated RRS feed

  • Question

  • I'm trying to run a copy activity in ADF and purposely trying to fail this activity to test my failure logging.

    Here is what the pipeline looks like (please note that this copy activity sits inside a "for each" activity and (inside "for each") an "if conditional" activity.

    I'm expecting the copy to fail, however not for the "LOG FAILURE" stored procedure, since I want to log the copy activity details in a SQL DB table. Here is what the error says:

    In the LOG FAILURE activity:

    "errorCode": "InvalidTemplate", "message": "The expression 'activity('INC_COPY_TO_ADL').output.rowsCopied' cannot be evaluated because property 'rowsCopied' doesn't exist, available properties are 'dataWritten, filesWritten, sourcePeakConnections, sinkPeakConnections, copyDuration, errors, effectiveIntegrationRuntime, usedDataIntegrationUnits, billingReference, usedParallelCopies, executionDetails, dataConsistencyVerification, durationInQueue'.", "failureType": "UserError", "target": "LOG_FAILURE"

    In the Copy activity INC_COPY_TO_ADL (this is expected since the SQL query is wrong)

    "errorCode": "2200", "message": "Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Invalid object name 'dbo.CustCustomerV3Staging123'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Invalid object name 'dbo.CustCustomerV3Staging123'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=208,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=208,State=1,Message=Invalid object name 'dbo.CustCustomerV3Staging123'.,},],'", "failureType": "UserError", "target": "INC_COPY_TO_ADL"

    I wonder why the LOG Failure activity failed (i.e. the expression was not evaluated)? Please note that when the SQL query is correct, the LOG SUCCESS stored procedures works okay.

    Many thanks. RA

    Saturday, May 23, 2020 5:30 PM


  • This solved my problem, I basically set the following properties (RowsCopied, RowsRead, QueuingDurationInSecs, TransferDurationInSecs, TimeToFirstByteInSecs) to null since these are not available in case of failure.
    • Marked as answer by Rizal Ang Wednesday, May 27, 2020 11:48 AM
    Wednesday, May 27, 2020 11:48 AM