locked
ADF V2 - Issue with delimiter when dealing with a connection-throttled SFTP server and big files RRS feed

  • Question

  • Thank you for your suggestions. I have sent a mail i the meantime, and am awaiting internal approval to see the Health Advisories (I currently lack permissions to see them). 

    When I set the parallellism option to 1 (which according to the documentation ought to guarantee only 1 file is processed at a time), it seemed to work. However, we ended up running into the same issue as before, where Data Factory starts interpreting a semi-colon as the column delimiter, despite us having configured that to be the "~" symbol (https://github.com/MicrosoftDocs/azure-docs/issues/36038).

    I'm waiting for our supplier to increase the amount of connections, as fiddling with that particular setting seemed to influence whether or not a file could successfully be processed. 

    As per your recommendation, I also changed the DIUs for the task (to 8). However, when checking the progress/status of the load, it still only shows 4 being used. The same applies when changing the value to 16.

    Tuesday, July 30, 2019 12:27 PM

All replies

  • We now have 5 times the connections at our disposal. The issue with the semicolon, however, remains. 

    Should we try and discuss this in a separate/isolated issue? 
    Tuesday, July 30, 2019 2:58 PM
  • Hi Peter,

    I have split the issue into two threads, since both are two different issues. 

    Here is the MSDN thread link for earlier conversation: https://social.msdn.microsoft.com/Forums/en-US/968b77d1-c005-4140-b390-badc740ed1ff/sftp-to-blob-storage-pipeline-issues?forum=AzureDataFactory (If it is a SFTP authentication issue, I have provided workaround steps to overcome the SFTP Max concurrent connection issue)

    And thank you for sharing the details through email. For a deeper analysis on your issue, we have enabled one time free technical support ticket for the subscription you have shared. A support engineer will provide a one-one assistance once you have opened a ticket by following the instructions sent in the email.

    Thank you.


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Tuesday, July 30, 2019 3:48 PM
  • Hi Peter,

    Could you try setting the column delimiter as Unicode character '\u0001' to treat each line as a single column? 


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Tuesday, July 30, 2019 10:04 PM