none
Copy Data task with SFTP as Sink creates empty file RRS feed

  • Question

  • I have a Copy Data task in ADF that has a blob file (csv) as Source and an SFTP sink.

    The task runs successfully. The file is created on the SFTP destination but with no data - empty file

    The SFTP dataset looks good - test connection is successful.

    When i preview my source file in ADF I see the data as expected.

    The Mapping looks good - all fields present and mapped

    I've tested copying the same file to the SFTP location with an SFTP client and it works (file has data).

    I've tried several different files but they all get created on SFTP destination as empty files

    What could be wrong?

    Thanks

    Sunday, January 26, 2020 2:58 AM

All replies

  • Hello Phil , 

    Thanks for the questions , we are reaching out to the internal team and will update you on this .

    Thanks 

    Himanshu


    Thanks Himanshu

    Tuesday, January 28, 2020 6:09 PM
    Moderator
  • I was trying to repro the issue , can you please share the JSON the pipeline please ?

    Thanks Himanshu

    Wednesday, January 29, 2020 1:42 AM
    Moderator
  • To reproduce;
    Create a Pipeline, drag a Copy Data activity onto canvas
    On Source tab pick any dataset on your instance that points to a blob csv file
    On Sink tab specify a dataset that points to an SFTP destination
    On Mapping tab map source fields to sink fields
    Execute

    JSON below

    Thanks

    Phil

    {
        "name": "Upload file to SFTP",
        "description": "SFTP Upload",
        "type": "Copy",
        "dependsOn": [
            {
                "activity": "Generate csv file",
                "dependencyConditions": [
                    "Succeeded"
                ]
            }
        ],
        "policy": {
            "timeout": "7.00:00:00",
            "retry": 0,
            "retryIntervalInSeconds": 30,
            "secureOutput": false,
            "secureInput": false
        },
        "userProperties": [
            {
                "name": "Source",
                "value": "myPath/MyCsvFile.csv"
            },
            {
                "name": "Destination",
                "value": "/files/data/MyCsvFile"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "DelimitedTextSource",
                "storeSettings": {
                    "type": "AzureBlobStorageReadSettings",
                    "recursive": false,
                    "wildcardFileName": "*.csv",
                    "enablePartitionDiscovery": false
                },
                "formatSettings": {
                    "type": "DelimitedTextReadSettings"
                }
            },
            "sink": {
                "type": "DelimitedTextSink",
                "storeSettings": {
                    "type": "SftpWriteSettings",
                    "operationTimeout": "01:00:00"
                },
                "formatSettings": {
                    "type": "DelimitedTextWriteSettings",
                    "quoteAllText": true,
                    "fileExtension": ".csv"
                }
            },
            "enableStaging": false
        },
        "inputs": [
            {
                "referenceName": "MyCsvFile_blob",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "Totango_SFTP",
                "type": "DatasetReference"
            }
        ]
    }

    Wednesday, January 29, 2020 2:34 PM
  • As mentioned by you " On Source tab pick any dataset on your instance that points to a blob csv file" . 

    Can you please remove the "wildcardFileName": "*.csv" and see that makes a difference . 

    In my case when i put "wildcardFileName": "*.csv , there is no file created on the FTP side , but the pipeline never fails .

    Please let me know how it goes .


    Thanks Himanshu

    Wednesday, January 29, 2020 11:24 PM
    Moderator
  • Hi Himanshu,

    I did as you suggested but it made no difference

    As before it shows ~98MB data transferred but the file was still created as an empty file (no data) at the SFTP location

    Thanks

    Phil

    Thursday, January 30, 2020 4:56 PM