none
Until Activity with errorCode 400

    Question

  • Hi,

    In an ADF v2 pipeline, I use an until activity to check if the file is updated today. If it is no, it will wait a constant seconds and check it again. And If it is yes the pipeline will continue to run the following workflow.

    I got the error below from until activity, and I don't know how to fix this error.

    {
        "errorCode": "400",
        "message": "Activity failed because an inner activity failed",
        "failureType": "UserError",
        "target": "Until1"
    }

    The json of the pipeline is below, It contians one until activity(include a wait activity and a get metadata acitvity), One copy activity after until activity.

    {
        "name": "pipeline1",
        "properties": {
            "activities": [
                {
                    "name": "Until1",
                    "type": "Until",
                    "typeProperties": {
                        "expression": {
                            "value": "@equals(addhours(utcnow(),8,'yyyy-MM-dd'),formatDateTime(activity('last').output.LastModfied,'yyyy-MM-dd'))",
                            "type": "Expression"
                        },
                        "activities": [
                            {
                                "name": "last",
                                "type": "GetMetadata",
                                "dependsOn": [
                                    {
                                        "activity": "Wait1",
                                        "dependencyConditions": [
                                            "Succeeded"
                                        ]
                                    }
                                ],
                                "policy": {
                                    "timeout": "1.00:00:00",
                                    "retry": 0,
                                    "retryIntervalInSeconds": 30,
                                    "secureOutput": false
                                },
                                "typeProperties": {
                                    "dataset": {
                                        "referenceName": "ADLS_TopArticle",
                                        "type": "DatasetReference"
                                    },
                                    "fieldList": [
                                        "lastModified"
                                    ]
                                }
                            },
                            {
                                "name": "Wait1",
                                "type": "Wait",
                                "typeProperties": {
                                    "waitTimeInSeconds": 30
                                }
                            }
                        ],
                        "timeout": "1.00:00:00"
                    }
                },
                {
                    "name": "Copy Data1",
                    "type": "Copy",
                    "dependsOn": [
                        {
                            "activity": "Until1",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "1.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false
                    },
                    "typeProperties": {
                        "source": {
                            "type": "AzureDataLakeStoreSource",
                            "recursive": true
                        },
                        "sink": {
                            "type": "SqlSink",
                            "writeBatchSize": 10000
                        },
                        "enableStaging": false,
                        "dataIntegrationUnits": 0,
                        "translator": {
                            "type": "TabularTranslator",
                            "columnMappings": {
                                "col1": "QueryDate",
                                "col2": "Id",
                                "col3": "ANumber",
                                "col4": "DNumber",
                                "col5": "FNumber"
                            }
                        }
                    },
                    "inputs": [
                        {
                            "referenceName": "ADLS_TopArticle",
                            "type": "DatasetReference"
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "DB_TopArticle",
                            "type": "DatasetReference"
                        }
                    ]
                }
            ]
        }
    }


    • Edited by QingGINQ Wednesday, July 25, 2018 8:33 AM
    Wednesday, July 25, 2018 8:31 AM

Answers

  • Hi Wang,

    You are totally right.

    I definitely misspelled the 'lastModified'...

    After correcting it, the pipeline works fine.

    I am such an idiot. So sorry about that.

    Thanks,

    Qing

    Friday, July 27, 2018 6:54 AM

All replies

  • Hi Qing,

    Not sure if it's the cause, but you can't reference activity('last').output.LastModfied in Until activity expression with the reason that the activity 'last' is not an ancestor to the Until activity. 

    Hope this reminding point you to the right direction.

    Thursday, July 26, 2018 8:19 AM
  • Hi Wang,

    I also have another 7 pipelines with the almost identical logic of the error one mentioned above.

     The logic is constantly checking whether the specified file exists(not the last modified time) until the file exists, then copy the file to a SQL table and exit the pipeline.

    The expression of the Until Activity is @activity('last').output.exists which also has the 'ancestor' error, but the pipeline runs smoothly.

    Ancestor error detail:

    The output of activity 'last' can't be referenced since it is either not an ancestor to the current activity or does not exist

    the JSON of the pipeline(All the 7 pipelines work very well with the ancestor error ):

    {
        "name": "HotTopic_Pipeline",
        "properties": {
            "activities": [
                {
                    "name": "Copy_HotTopic",
                    "type": "Copy",
                    "dependsOn": [
                        {
                            "activity": "Until1",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "1.00:00:00",
                        "retry": 3,
                        "retryIntervalInSeconds": 300,
                        "secureOutput": false
                    },
                    "typeProperties": {
                        "source": {
                            "type": "AzureDataLakeStoreSource",
                            "recursive": true
                        },
                        "sink": {
                            "type": "SqlSink",
                            "writeBatchSize": 10000
                        },
                        "enableStaging": false,
                        "dataIntegrationUnits": 0,
                        "translator": {
                            "type": "TabularTranslator",
                            "columnMappings": {
                                "QueryDate": "QueryDate",
                                "TopQueryType": "TopQueryType",
                                "Body": "Body"
                            }
                        }
                    },
                    "inputs": [
                        {
                            "referenceName": "ADLS_HotTopic",
                            "type": "DatasetReference"
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "DB_HotTopic",
                            "type": "DatasetReference"
                        }
                    ]
                },
                {
                    "name": "Until1",
                    "type": "Until",
                    "typeProperties": {
                        "expression": {
                            "value": "@activity('last').output.exists",
                            "type": "Expression"
                        },
                        "activities": [
                            {
                                "name": "last",
                                "type": "GetMetadata",
                                "dependsOn": [
                                    {
                                        "activity": "Wait1",
                                        "dependencyConditions": [
                                            "Succeeded"
                                        ]
                                    }
                                ],
                                "policy": {
                                    "timeout": "1.00:00:00",
                                    "retry": 0,
                                    "retryIntervalInSeconds": 60,
                                    "secureOutput": false
                                },
                                "typeProperties": {
                                    "dataset": {
                                        "referenceName": "ADLS_HotTopic",
                                        "type": "DatasetReference"
                                    },
                                    "fieldList": [
                                        "exists"
                                    ]
                                }
                            },
                            {
                                "name": "Wait1",
                                "type": "Wait",
                                "typeProperties": {
                                    "waitTimeInSeconds": 1800
                                }
                            }
                        ],
                        "timeout": "1.00:00:00"
                    }
                }
            ],
            "folder": {
                "name": "ADLSToDB"
            }
        },
        "type": "Microsoft.DataFactory/factories/pipelines"
    }

    Thursday, July 26, 2018 11:03 AM
  • Well noted.

    If pipeline worked well with the validation error, it's necessary for us to improve UX experience.

    Now talk back to this issue, Until activity failed maybe because the inside copy activity running failed. Please make sure the copy activity setting is right, like column mapping setting.

    You could also provide us the RunID of your activity, based on which we could do more investigation.

    Thx.

    Friday, July 27, 2018 3:10 AM
  • Hi Qing,

    Finally find the reason: You misspelled 'lastModified'...

    Please use "@equals(addhours(utcnow(),8,'yyyy-MM-dd'),formatDateTime(activity('last').output.lastModified,'yyyy-MM-dd'))" and have another try.

    Thanks.

    • Proposed as answer by Wang Zhang Wednesday, August 1, 2018 2:01 AM
    • Unproposed as answer by Wang Zhang Wednesday, August 1, 2018 2:01 AM
    Friday, July 27, 2018 6:31 AM
  • Hi Wang,

    You are totally right.

    I definitely misspelled the 'lastModified'...

    After correcting it, the pipeline works fine.

    I am such an idiot. So sorry about that.

    Thanks,

    Qing

    Friday, July 27, 2018 6:54 AM
  • Never mind. :)

    Friday, July 27, 2018 7:41 AM