none
Copy file from SFTP server to Azure blob storage directly

    Question

  • Hi there,

    My client is uploading CSV files to a SFTP server(VM in Azure) . I have a service running in Azure which process the CSV and imports the data into Microsoft Dynamics Marketing.

    My plan was to use a Azure Logic App to get the file from the SFTP server and provide it to the processing service. But I encountered that Logic Apps aren't quite working yet. Not mature enough.

    My plan is now to get the file from the SFTP server with the WinSCP library and store it directly in Blob storage. The processing service will look at the blob storage then. I want to use the web role to transfer these files using SFTP with the WinSCP library from my SFTP server straight to my blob storage account (without copying to the local storage in the web role). Any advice to get this working?

    Code so far:

    session.Open(sessionOptions);

    string remotePath = "test.csv";
    string localPath = "c:\\backup\\" + fileName;

    if (session.FileExists(remotePath))
    {
             // Download the file and throw on any error
             session.GetFiles(remotePath, localPath).Check();
    }

    //I don't want to store in the localPath, but I need to store directly in blob storage, but how to get that file stream?
    blockBlob.UploadFromStream(fileStream);      

    Tuesday, August 11, 2015 8:33 PM

Answers

  • Solved it with the help of my colleague William de Bakker.

    First you should use https://sshnet.codeplex.com/ as library because that support streams.

    Here is the code which works:

                 Stream stream;

                using (var sftp = new SftpClient(Host, Port, Username, Password))
                {
                    sftp.Connect();

                    stream = new MemoryStream();
                    sftp.DownloadFile(RemoteFileName, stream);

                    sftp.Disconnect();
                }

                if (stream != null)
                {
                    if (!stream.CanSeek)
                    {
                        throw new Exception("Won't work :(");
                    }

                    stream.Seek(0, SeekOrigin.Begin);
                    blockBlob.UploadFromStream(stream);
                }

    • Marked as answer by Mark Willems Wednesday, August 12, 2015 9:59 AM
    Wednesday, August 12, 2015 9:59 AM

All replies

  • Solved it with the help of my colleague William de Bakker.

    First you should use https://sshnet.codeplex.com/ as library because that support streams.

    Here is the code which works:

                 Stream stream;

                using (var sftp = new SftpClient(Host, Port, Username, Password))
                {
                    sftp.Connect();

                    stream = new MemoryStream();
                    sftp.DownloadFile(RemoteFileName, stream);

                    sftp.Disconnect();
                }

                if (stream != null)
                {
                    if (!stream.CanSeek)
                    {
                        throw new Exception("Won't work :(");
                    }

                    stream.Seek(0, SeekOrigin.Begin);
                    blockBlob.UploadFromStream(stream);
                }

    • Marked as answer by Mark Willems Wednesday, August 12, 2015 9:59 AM
    Wednesday, August 12, 2015 9:59 AM
  • Hi Mark,

      I saw your solution to copy <g class="gr_ gr_77 gr-alert gr_gramm gr_disable_anim_appear Grammar only-ins doubleReplace replaceWithoutSep" data-gr-id="77" id="77">file</g> from SFTP Server to Azure Blob Storage directly. However, I would need some help from you:

    Where do you run this Code? Do you integrate it in Azure Logic App? Could you please provide me a step-by-step on how you accomplish this task? 

    I would really appreciate if you could help me?

    Thanks in advance,

      Neptune


    Thursday, March 2, 2017 4:38 PM
  • Hi Mark,

    Where do you run this Code? Do you integrate it in Azure Logic App?

    Could you please provide me a step-by-step on how you accomplish this task of moving Files from SFTP Server to Azure Blob Storage directly? 

    I would really appreciate if you could help me?

    Thanks in advance,

      Neptune

    Thursday, March 2, 2017 4:40 PM