locked
V2 - UserErrorAdlsFileRenameFailed ERROR while copying to ADLS from on-prem filesystem RRS feed

  • Question

  • I am using data factory v2.

    While copying a file from my system to an adls resource, the copy activity fails with an error. The error is as follows:

    { "errorCode": "2200", "message": "ErrorCode=UserErrorAdlsFileRenameFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to rename a 'Azure Data Lake Store' file from '\\GLOBUS\\FOLDER1\\/_copy_tmp_bb238e33-18f4-475b-8b12-3404519baca3/b90e88ba-515c-4ff2-b476-1806b236e599/_tmp_file1.csv-0.tmp' to '\\GLOBUS\\FOLDER1\\/file1.csv'. The status code is '0x83090A1A', request id is 'a2be0e89-982a-4b01-8ded-e8033e1e7752'.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "Copy1"

    }

    This error was not generated for pipeline runs last week(4 days ago) and everything was running fine.

    Today while running the pipeline this error occured in the copy activity

    The details of the activity is as follows:

    source: on-prem FS(windows); file1.csv - 1kb; self hosted IR;

    sink: ADLS; default IR; folder- \\GLOBUS\\FOLDER1

    I am not able to identify the problem. Please help.

    EDIT:

    I WAS ABLE TO GET RID OF THE ERROR BY REMOVING THE SINK DATASET PARAMETERS I WAS USING FOR IDENTIFYING FOLDERNAME AND FILE NAME.

    HARDCODING IT FOR NOW.

    TESTS:

    1. CLONED THE PIPELINE AND RAN THE CLONED PL : FAILED WITH SAME ERROR.

    2. USED A SECOND ADLS DATASET (WITH PARAMETERS AND POINTED TO A SECOND ALDS SUBSCRIPTION) : FAILED WITH SAME ERROR.

    3. CLONED THE DATASET AND REMOVED THE FOLDERNAME AND FILENAME PARAMETERS : COPY SUCCESS.


    • Edited by AnarghyaV Wednesday, May 16, 2018 3:36 AM
    Tuesday, May 15, 2018 10:20 AM

Answers

  • Hi AnarghyaV,

    I think it's the folder path "\\GLOBUS\\FOLDER1\\" that's causing the rename issue. By specifying a folder path like this, you're actually creating a folder name "\\GLOBUS\\FOLDER1\\", is this by intension? If not, I think you should change the folder path to "/GLOBUS/FOLDER1".

    Thanks,

    Eva

    • Marked as answer by AnarghyaV Friday, May 25, 2018 4:02 AM
    Friday, May 25, 2018 3:34 AM

All replies

  • Could you share the pipeline and dataset json having problems?
    Monday, May 21, 2018 2:36 PM
  • pipeline:

    {
        "name": "LND_JOB",
        "properties": {
            "activities": [
                {
                    "name": "Copy1",
                    "type": "Copy",
                    "dependsOn": [
                        {
                            "activity": "ABC_JOB_START",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false
                    },
                    "typeProperties": {
                        "source": {
                            "type": "FileSystemSource",
                            "recursive": false
                        },
                        "sink": {
                            "type": "AzureDataLakeStoreSink"
                        },
                        "enableStaging": false,
                        "cloudDataMovementUnits": 0
                    },
                    "userProperties": [
                        {
                            "name": "Source",
                            "value": "@{pipeline().parameters.JOB_DTL.SRC_PATH_SCHEM}/@{pipeline().parameters.JOB_DTL.SRC_FILE_TBL}"
                        },
                        {
                            "name": "Destination",
                            "value": "@{pipeline().parameters.JOB_DTL.TGT_PATH_SCHEM}/@{pipeline().parameters.JOB_DTL.TGT_FILE_TBL}"
                        }
                    ],
                    "inputs": [
                        {
                            "referenceName": "SRC_FS_DS",
                            "type": "DatasetReference",
                            "parameters": {
                                "FOLDER_PATH": "@{pipeline().parameters.JOB_DTL.SRC_PATH_SCHEM}",
                                "FILE_NAME": "@{pipeline().parameters.JOB_DTL.SRC_FILE_TBL}"
                            }
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "SRC_ADLS_LND_GLOBUS",
                            "type": "DatasetReference",
                            "parameters": {
                                "FOLDER_PATH": "@{pipeline().parameters.JOB_DTL.TGT_PATH_SCHEM}",
                                "FILE_NAME": "@{pipeline().parameters.JOB_DTL.TGT_FILE_TBL}"
                            }
                        }
                    ]
                },
                {
                    "name": "ABC_JOB_START",
                    "type": "ExecutePipeline",
                    "typeProperties": {
                        "pipeline": {
                            "referenceName": "ABC_JOB_START",
                            "type": "PipelineReference"
                        },
                        "waitOnCompletion": true,
                        "parameters": {
                            "JOB_SK": {
                                "value": "@{pipeline().parameters.JOB_DTL.JOB_SK}",
                                "type": "Expression"
                            },
                            "PL_RUN_ID": {
                                "value": "@{pipeline().RunId}",
                                "type": "Expression"
                            },
                            "STEP_SK": {
                                "value": "@{pipeline().parameters.JOB_DTL.STEP_SK}",
                                "type": "Expression"
                            }
                        }
                    }
                },
                {
                    "name": "ABC_JOB_CLOSE_SUCCESS",
                    "type": "ExecutePipeline",
                    "dependsOn": [
                        {
                            "activity": "Copy1",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "typeProperties": {
                        "pipeline": {
                            "referenceName": "ABC_JOB_CLOSE",
                            "type": "PipelineReference"
                        },
                        "waitOnCompletion": true,
                        "parameters": {
                            "JOB_SK": {
                                "value": "@{pipeline().parameters.JOB_DTL.JOB_SK}",
                                "type": "Expression"
                            },
                            "STATUS": "SUCCEEDED",
                            "ROWS_READ": {
                                "value": "@activity('Copy1').output.rowsRead",
                                "type": "Expression"
                            },
                            "ROWS_LOADED": {
                                "value": "@activity('Copy1').output.rowsCopied",
                                "type": "Expression"
                            },
                            "DATA_READ": {
                                "value": "@activity('Copy1').output.dataRead",
                                "type": "Expression"
                            },
                            "DATA_WRITTEN": {
                                "value": "@activity('Copy1').output.dataWritten",
                                "type": "Expression"
                            }
                        }
                    }
                },
                {
                    "name": "ABC_JOB_CLOSE_FAILURE",
                    "type": "ExecutePipeline",
                    "dependsOn": [
                        {
                            "activity": "Copy1",
                            "dependencyConditions": [
                                "Failed"
                            ]
                        }
                    ],
                    "typeProperties": {
                        "pipeline": {
                            "referenceName": "ABC_JOB_CLOSE",
                            "type": "PipelineReference"
                        },
                        "waitOnCompletion": true,
                        "parameters": {
                            "JOB_SK": {
                                "value": "@{pipeline().parameters.JOB_DTL.JOB_SK}",
                                "type": "Expression"
                            },
                            "STATUS": "FAILED",
                            "ROWS_READ": "0",
                            "ROWS_LOADED": "0",
                            "DATA_READ": "0",
                            "DATA_WRITTEN": "0"
                        }
                    }
                },
                {
                    "name": "NOTIFICATION_FAILED",
                    "type": "WebActivity",
                    "dependsOn": [
                        {
                            "activity": "ABC_JOB_CLOSE_FAILURE",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false
                    },
                    "typeProperties": {
                        "url": "https://prod-46.westeurope.logic.azure.com:443/workflows/27e51d10e52842cc96afecc5e2299ed0/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=RxXryDY72yBVDhXgVDlZkyemrZY4rnLld_DEa7pC6Ag",
                        "method": "POST",
                        "headers": {
                            "Content-Type": "application/json"
                        },
                        "body": {
                            "value": "{\"message\":\"@{activity('Copy1').error.message}\",\"cycName\":\"CYC_GLOBUS\",\"stepName\":\"PL_DAT_GLOBUS_FILES_LND\",\"jobName\":\" \",\"fileName\":\"@{pipeline().parameters.JOB_DTL.SRC_FILE_TBL}\",\"runID\":\"@{pipeline().RunId}\",\"sink\":\"@{pipeline().parameters.JOB_DTL.JOB_TYP_CD}\",\"reciever\":\"anarghya.v@hpe.com\"}",
                            "type": "Expression"
                        }
                    }
                }
            ],
            "parameters": {
                "JOB_DTL": {
                    "type": "Object"
                }
            }
        }
    }

    dataset:

    {
        "name": "SRC_ADLS_LND_GLOBUS",
        "properties": {
            "linkedServiceName": {
                "referenceName": "LS_ADLS_LND",
                "type": "LinkedServiceReference"
            },
            "parameters": {
                "FOLDER_PATH": {
                    "type": "String"
                },
                "FILE_NAME": {
                    "type": "String"
                }
            },
            "type": "AzureDataLakeStoreFile",
            "typeProperties": {
                "format": {
                    "type": "TextFormat",
                    "columnDelimiter": ",",
                    "rowDelimiter": "",
                    "nullValue": "\\N",
                    "treatEmptyAsNull": true,
                    "firstRowAsHeader": false
                },
                "fileName": {
                    "value": "@dataset().FILE_NAME",
                    "type": "Expression"
                },
                "folderPath": {
                    "value": "@dataset().FOLDER_PATH",
                    "type": "Expression"
                }
            }
        }
    }

    Wednesday, May 23, 2018 10:46 AM
  • Hi AnarghyaV,

    I think it's the folder path "\\GLOBUS\\FOLDER1\\" that's causing the rename issue. By specifying a folder path like this, you're actually creating a folder name "\\GLOBUS\\FOLDER1\\", is this by intension? If not, I think you should change the folder path to "/GLOBUS/FOLDER1".

    Thanks,

    Eva

    • Marked as answer by AnarghyaV Friday, May 25, 2018 4:02 AM
    Friday, May 25, 2018 3:34 AM
  • SOLVED!

    Thanks Eva, as you pointed out the folder path had the wrong slash and was creating the problem.

    Working fine now.

    I believe the folder path was changed while testing and that is why it was working and then it stopped working.

    Friday, May 25, 2018 4:06 AM