none
Lookup Activity error code 1003 "Failed to invoke web callback to workflow"

    Question

  • I have received the error "Failed to invoke web callback to workflow" (error code 1003) during a Lookup Activity that uses a SQL query on an Azure SQL DB. I have never seen this error before and I can't find any details about it online. This pipeline has worked thousands of times before without this problem. In fact, 50 other files were successfully processed in last night's batch load using this same pipeline, some before and some after the pipeline failed on this particular file.

    Does anyone know what this error means? How can I understand what to do about it?

    Here is the error JSON:

    {
        "errorCode": "1003",
        "message": "Failed to invoke web callback to workflow.",
        "failureType": "SystemError",
        "target": "Lookup_ProcessedFiles"
    }

    Here is the JSON for the Activity:

    {
                    "name": "Lookup_ProcessedFiles",
                    "description": "Lookup whether file was successfully processed",
                    "type": "Lookup",
                    "dependsOn": [
                        {
                            "activity": "If_Forecast",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false
                    },
                    "typeProperties": {
                        "source": {
                            "type": "SqlSource",
                            "sqlReaderQuery": {
                                "value": "@replace('select count(*) Count from ctlProcessedFiles where SourceFilename=''#'' and ProcessedStatus=''Processed''', '#', pipeline().parameters.filename)",
                                "type": "Expression"
                            }
                        },
                        "dataset": {
                            "referenceName": "SRC_SQL_ProcessedFiles",
                            "type": "DatasetReference"
                        },
                        "firstRowOnly": true
                    }
                }



    Thursday, August 2, 2018 10:46 PM

Answers

  • From the RunId, we see that it's caused by a random failure. Usually we set 3 is good enough for retry attempts like HttpClient retry. Also set retry is a good practice to improve reliability and to avoid transient network issue.
    Wednesday, August 8, 2018 12:20 PM

All replies

  • Do you have the failed RunId so that we can take a further look? Usually it is a transient issue and can be succeeded with a retry set in your policy.
    Friday, August 3, 2018 12:57 AM
  • Thanks for your response. Here is the RunId of the failed Lookup Activity:

    357082fd-2220-461e-a260-9ae47a5e0b15

    Can you recommend best practice for the retry settings? E.g. how many retry attempts should be made, and should every single Activity have retry attempts, or only some?

    Monday, August 6, 2018 1:31 AM
  • From the RunId, we see that it's caused by a random failure. Usually we set 3 is good enough for retry attempts like HttpClient retry. Also set retry is a good practice to improve reliability and to avoid transient network issue.
    Wednesday, August 8, 2018 12:20 PM
  • Thanks for the help. I'm still somewhat confused as to the source of the failure. Is it due to the way I am using ADF? Or should this simply be accepted as a possible random failure?
    Wednesday, August 15, 2018 11:27 AM