none
SFTP: How to execute more than one SFTP process using multi-threading way? RRS feed

  • Question

  • Hi C# programmer all around the world,

    I am working on an application that uses SFTP to transfer data from a remote server to a local server over the network, and then the data are converted from .db to .csv format.

    In a real world scenario, we are using a hardware for data acquisition, each of these hardware is mounted on a mining vehicle. The data transfer happen only when the hardware enter the network zone, and in real life situation few vehicle will enter the network zone and then leave, in this case the data should be transferred during their presence within the network and once they leave the data transfer stream stop until they come again. Usually there is 20 vehicle. Each hardware has a specific IP address, the C# program read these IP from a JSON file and then ping each IP to see which hardware is within the network.

    My code is single threaded, that's mean if two vehicle was within the network, the data transfer perform only for one vehicle at a time, and sometime the other vehicle leave the network as soon as the first data transfer finish, which make the application unreliable and slow.

    Below is a snapshot of my single threaded code

    filePath = @"C:\temp\JSON\app-db.json";
    string text = File.ReadAllText(filePath);
    var currentList = JsonConvert.DeserializeObject<List<Datalogger>>(text);
    string host = item.IPaddress;
    string username = item.username;
    string password = item.password;
    string remoteDirectory = item.sourcefolder;
    string localDirectory = item.destfolder;
    string filextension = item.filextension;
    string removedownloaded = item.removedownloaded.ToString();
    using (SftpClient sftp = new SftpClient(host, username, password))
                    {
                        try
                        {
                            sftp.Connect();
    
                            var files = sftp.ListDirectory(remoteDirectory);
    
                            foreach (var file in files)
                            {
                                try
                                {
                                    string remoteFileName = file.Name;
    
                                    if ((file.Name.EndsWith(filextension)) || (file.Name.EndsWith(filextension.ToLower())) || (file.Name.EndsWith(filextension.ToUpper())))
                                    {
                                        using (Stream file1 = File.OpenWrite(Path.Combine(localDirectory, remoteFileName)))
                                        {
    
                                            string path = remoteDirectory + "/" + remoteFileName;
                                            sftp.DownloadFile(path, file1);
                                            
    
                                            if (removedownloaded == "1")
                                            {
                                                sftp.Delete(path);
                                            }
                                            else
                                            {
                                                sftp.DownloadFile(path, file1);
                                            }
    
                                        }
                                    }
    
                                }
                                catch (Exception er1)
                                {
                                    //MessageBox.Show("An exception has been caught " + er1.ToString());
                                }
    
                            }
                        }
                        catch (Exception entry)
                        {
                            MessageBox.Show(entry.Message);
                        }
                        //finally
                        //{
                        //    sftp.Disconnect();
                        //}
                    }
    
                }

    How to convert this code to multi-thread? Thanks in advance for your time and suggestion.



    Sami Arja

    Monday, October 7, 2019 10:19 PM

Answers

  • Hi samiarja,

    Thank you for posting here.

    You could use Parallel.Foreach to change signal thread to multi-thread. You can refer to the following code to do it.

          using (SftpClient sftp = new SftpClient(host, username, password))
                {
                    try
                    {
                        sftp.Connect();
    
                        var files = sftp.ListDirectory(remoteDirectory);
                        Parallel.ForEach(files, file =>
                         {
    
                             try
                             {
                                 string remoteFileName = file.Name;
    
                                 if ((file.Name.EndsWith(filextension)) || (file.Name.EndsWith(filextension.ToLower())) || (file.Name.EndsWith(filextension.ToUpper())))
                                 {
                                     using (Stream file1 = File.OpenWrite(Path.Combine(localDirectory, remoteFileName)))
                                     {
    
                                         string path = Path.Combine(remoteDirectory, remoteFileName);
                                         sftp.DownloadFile(path, file1);
                                         if (removedownloaded == "1")
                                         {
                                             sftp.Delete(path);
                                         }
                                         else
                                         {
                                             sftp.DownloadFile(path, file1);
                                         }
    
                                     }
                                 }
    
                             }
                             catch (Exception er1)
                             {
                                 //MessageBox.Show("An exception has been caught " + er1.ToString());
                             }
                         });
    
                       
                }

    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by samiarja Friday, October 18, 2019 2:57 AM
    Tuesday, October 8, 2019 1:54 AM
    Moderator

All replies

  • Hi samiarja,

    Thank you for posting here.

    You could use Parallel.Foreach to change signal thread to multi-thread. You can refer to the following code to do it.

          using (SftpClient sftp = new SftpClient(host, username, password))
                {
                    try
                    {
                        sftp.Connect();
    
                        var files = sftp.ListDirectory(remoteDirectory);
                        Parallel.ForEach(files, file =>
                         {
    
                             try
                             {
                                 string remoteFileName = file.Name;
    
                                 if ((file.Name.EndsWith(filextension)) || (file.Name.EndsWith(filextension.ToLower())) || (file.Name.EndsWith(filextension.ToUpper())))
                                 {
                                     using (Stream file1 = File.OpenWrite(Path.Combine(localDirectory, remoteFileName)))
                                     {
    
                                         string path = Path.Combine(remoteDirectory, remoteFileName);
                                         sftp.DownloadFile(path, file1);
                                         if (removedownloaded == "1")
                                         {
                                             sftp.Delete(path);
                                         }
                                         else
                                         {
                                             sftp.DownloadFile(path, file1);
                                         }
    
                                     }
                                 }
    
                             }
                             catch (Exception er1)
                             {
                                 //MessageBox.Show("An exception has been caught " + er1.ToString());
                             }
                         });
    
                       
                }

    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by samiarja Friday, October 18, 2019 2:57 AM
    Tuesday, October 8, 2019 1:54 AM
    Moderator
  • Since this is not CPU bound operation, the better way is to use Queue of Task (not Task<T> because your "work" saves/delete file instead of returning actual data) instead of using Parallel.ForEach() to avoid creating more thread than necessary. (Use the ContinueWhenAll() route when you want to make sure all tasks are done)

    https://devblogs.microsoft.com/pfxteam/parallel-extensions-and-io/

    Tuesday, October 8, 2019 2:49 AM
    Answerer