SFTP Connector Problem with large files RRS feed

  • Question

  • I'm trying to use a SFTP activity to copy some files from a SFTP location. I have 5 files, and everything is working just fine for 4 of the files. The fifth file is significantly larger (weighing in at about 3.2 GB). When I try the exact same operation with this file, the activity fails. 

    The error I get:

    Copy activity encountered a user error at Source side: ErrorCode=UserErrorFailedToReadStream,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from Sftp server 'host.domain.com', file path '/Client/PROD_DATA.CSV', offset '33554432', length: '16777216'.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=Renci.SshNet.Common.SshException,Message=The restart posion : 33554432,Source=Renci.SshNet,'.

    The operation fails after what looks like 1:30 or so. My guess is that the SFTP client is encountering some kind of disconnect or timeout. Fetching the file manually with filezilla works, so I don't think the server is the issue. Additionally, trying to get a smaller (250MB) version of the file seems to work as well. 

    Things I tried:

    • Increasing the timeout on the activity policy, 10 min, 15 min, 1 hr.
    • Set the cloudDataMovementUnits property to 4

    Tuesday, May 2, 2017 1:14 PM

All replies

  • Were you ever able to figure this out? We need to use the data factory v2 to transfer large files and are getting the same error.
    Friday, December 21, 2018 6:45 PM