none
How restrict no of receive file from sftp

    Question

  • Hi All,

    i need to restrict no of files receive from sftp location , so once SFTP poll the location only one file will picked and other will pick on another poll.

    I want to do this because we don't want to process multiple file in a day. I can do this by singleton orchestration by using receive location name in correlation. But i just want to know is it possible from sftp adapter?

     

    Monday, October 24, 2016 5:22 AM

Answers

  • i need to restrict no of files receive from sftp location , so once SFTP poll the location only one file will picked and other will pick on another poll.

    So here are the two ways you can achieve this:

    1) Use a third party library to connect to SFTP server and to get the specific file. You can try to use the open source SharpSSH library to do it.

    2) You can accomplish this with the nsoftware SFTP adapter. Take a look at this tutorial for more information: http://www.nsoftware.com/kb/showentry.aspx?entry=09170901

    The SFTP adapter expose a way to do this through a solicit response port which you can use within an orchestration.

    This allows you to create an orchestration where you can choose the files you want to download, and when, on the fly. 

    Listing the Directory

    In your orchestration after referencing the SFTP or FTP adapter you can set the message context properties to configure the adapter. To specify that the adapter should list the directory set the Other property to ListDirectory=true. For instance:

    Message_2 = Message_1;
    Message_2(nsoftware.BizTalk.SFTP.SSHUser) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHPassword) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHHost) = "server";
    Message_2(nsoftware.BizTalk.SFTP.SSHAcceptServerHostKeyAcceptAny) = true;
    Message_2(nsoftware.BizTalk.SFTP.RemotePath) = "temp";
    Message_2(nsoftware.BizTalk.SFTP.RemoteFile) = "*.txt";
    Message_2(nsoftware.BizTalk.SFTP.Other) = "ListDirectory=true";

    The solicit response port will return a message with a body containing an XML formatted representation of the directory contents. For instance:

    <directory name="temp">
    <directory name="subDir1">
    <directory name="subDir2">
    <file name="file1.txt">
    <size>size</size>
    <time>time</time>
    </file>
    <file name="file2.txt">
    <size>size</size>
    <time>time</time>
    </file>
    <file name="file3.txt">
    <size>size</size>
    <time>time</time>
    </file>
    </directory>
    </directory></directory>

    Note: The SFTP adapter returns additional CTime and ATime elements representing the CreationTime and AccessTime. CreationTime is only supported on servers that support SFTP protocol version 4 or higher. To enable the adapter to use version 4 or higher you can set the Other property to "ProtocolVersion=4".

    Downloading a File

    Because a solicit response port can return only one message, you can only download one file at a time with the SFTP and FTP send ports. To do this you'll need to set the DownloadSingleFile configuration setting to true. For instance:

    Message_2 = Message_1;
    Message_2(nsoftware.BizTalk.SFTP.SSHUser) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHPassword) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHHost) = "server";
    Message_2(nsoftware.BizTalk.SFTP.SSHAcceptServerHostKeyAcceptAny) = true;
    Message_2(nsoftware.BizTalk.SFTP.RemotePath) = "temp";
    Message_2(nsoftware.BizTalk.SFTP.RemoteFile) = "test.txt";
    Message_2(nsoftware.BizTalk.SFTP.Other) = "DownloadSingleFile=true";


    Rachit Sikroria (Microsoft Azure MVP)

    Wednesday, October 26, 2016 8:18 PM
    Moderator
  • I would rather go with fixing the issue with SFTP if that is the requirement..to me personally, if my Biztalk app is supposed to process a single file in a day,it would not make sense if multiple files are uploaded to the SFTP site.

    In you situation, how are you going to decide what is the priority of the File to be processed??? What if there are 10 files uploaded then by your reqyuirement of processing one file per day, would just ensure that the files get piled up on the SFTP site.

    ---------------------

    One thing i am thinking that i can promote the receive location name from the receive pipeline and by using this property i can create a singleton orchestraton instance . then all file will go through one orchestration instance then there i can process only first file received and stop others. am i right here? do this will create any issue?

    >> This will work ...but it is just a lot of effort because the files will be polled upon each polling interval and this pollednumberof fileswould go on increasing...

    My suggestion would be to clear the requirement with the concerned teams ..beacuse that will save a lot of efforts.

    Regards,


    Mandar Dharmadhikari

    Monday, October 24, 2016 5:53 AM
    Moderator
  • Hi Giri,

    You have a few options here:

    • Change the requirement: Like John mentioned, which file we pick? Is it based on time-stamp? Those questions make this requirement seem like it needs some refining.


    • Write a customs adapter: You may choose to write an adapter yourself. I know you can extend WCF Adapters, but am not sure if you can extend the SFTP adapter. Maybe be worth a shot. Here is the scoop on MSDN - Developing Custom Adapters


    • Use a third party adapter: Like Pi_xel_xar mentioned, there are adapters out there that will fulfill that requirement. Here is nSoftware's website mentioning that capability- SFTP Adapter Comparison


    Please note that I do not recommend using products or approaches not supported by Microsoft, and am not in any way associated with products mentioned in this post.


    My Technet Articles

    If you like this or another reply, vote it up!
    If you think this or another reply answers the original question, mark it or propose it as an answer.


    Mauricio Feijo
    www.mauriciofeijo.com


    Wednesday, October 26, 2016 8:36 PM

All replies

  • are you picking up file using some file mask??

    Easiest way would be to keep the receieve location active only for sometime using service window and then uploading the new file out of the service window on the SFTP sevrer..There is no such stock property to limit number of files polled. If the polling criteria is satisfied, the data is polled.

    Regards,


    Mandar Dharmadhikari


    Monday, October 24, 2016 5:27 AM
    Moderator
  • by using service window will not solve my problem here. 

    Suppose three files came to sftp in same time then all files will be picked up. then how?

    Monday, October 24, 2016 5:35 AM
  • Agreed,

    Thats why I mentioned that the File be uploaded to the SFTP out of the service window...but that is not possible if  you are not the owner of the SFTP. But when you say that you dont need to process multiple files in a day, then how come that multiple files are uploaded to the SFTP server?? Just trying to understand the Business Requirement. ANd yes there is no stock proertyon the SFTP adapter (Out of the Box Shipped with Biztalk) which can limit the number of files that can be polled.  Then you can poll the files from the ORchestration.

    Regards,


    Mandar Dharmadhikari



    Monday, October 24, 2016 5:40 AM
    Moderator
  • Yes i am also agree that if only one file need to process then why multiple file will come.

    But it's not in my hand , it's uploaded by users so same time two user can upload the files then it will come. I need to handle this situation.

    One thing i am thinking that i can promote the receive location name from the receive pipeline and by using this property i can create a singleton orchestraton instance . then all file will go through one orchestration instance then there i can process only first file received and stop others. am i right here? do this will create any issue?

     
    Monday, October 24, 2016 5:45 AM
  • I would rather go with fixing the issue with SFTP if that is the requirement..to me personally, if my Biztalk app is supposed to process a single file in a day,it would not make sense if multiple files are uploaded to the SFTP site.

    In you situation, how are you going to decide what is the priority of the File to be processed??? What if there are 10 files uploaded then by your reqyuirement of processing one file per day, would just ensure that the files get piled up on the SFTP site.

    ---------------------

    One thing i am thinking that i can promote the receive location name from the receive pipeline and by using this property i can create a singleton orchestraton instance . then all file will go through one orchestration instance then there i can process only first file received and stop others. am i right here? do this will create any issue?

    >> This will work ...but it is just a lot of effort because the files will be polled upon each polling interval and this pollednumberof fileswould go on increasing...

    My suggestion would be to clear the requirement with the concerned teams ..beacuse that will save a lot of efforts.

    Regards,


    Mandar Dharmadhikari

    Monday, October 24, 2016 5:53 AM
    Moderator
  • With all that's already discussed and rightly pointed if u want to restrict the poll try looking at the Nsoftware adapter suit. The sftp adapter in there has a property MaxFileCount that you can set to fetch Certain count of file at one poll. U can use it in combination of poll interval and poll schedule to poll just 1 file within a span of time. There is a free set of adapter as well but you will have to investigate. Else based on the business need you Can ask client to buy it.

    Pi_xel_xar

    Blog: My Blog

    BizTalkApplicationDeploymentTool: BizTalk Application Deployment Tool/

    Monday, October 24, 2016 6:32 AM
    Answerer
  • Why don't you let the file be uploaded to a different folder and create a Powershell script attached to the Task Scheduler to copy the file to the target SFTP folder once a day.

    Regards, Sjoukje. Blog | LinkedIn | Twitter




    Monday, October 24, 2016 6:36 AM
    Answerer
  • How will you know which is the 'correct' file?  That seems to be the bigger issue.

    If three users upload at the same time, what do you do, just pick one at random?  That can't be right.

    Monday, October 24, 2016 12:10 PM
    Moderator
  • i need to restrict no of files receive from sftp location , so once SFTP poll the location only one file will picked and other will pick on another poll.

    So here are the two ways you can achieve this:

    1) Use a third party library to connect to SFTP server and to get the specific file. You can try to use the open source SharpSSH library to do it.

    2) You can accomplish this with the nsoftware SFTP adapter. Take a look at this tutorial for more information: http://www.nsoftware.com/kb/showentry.aspx?entry=09170901

    The SFTP adapter expose a way to do this through a solicit response port which you can use within an orchestration.

    This allows you to create an orchestration where you can choose the files you want to download, and when, on the fly. 

    Listing the Directory

    In your orchestration after referencing the SFTP or FTP adapter you can set the message context properties to configure the adapter. To specify that the adapter should list the directory set the Other property to ListDirectory=true. For instance:

    Message_2 = Message_1;
    Message_2(nsoftware.BizTalk.SFTP.SSHUser) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHPassword) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHHost) = "server";
    Message_2(nsoftware.BizTalk.SFTP.SSHAcceptServerHostKeyAcceptAny) = true;
    Message_2(nsoftware.BizTalk.SFTP.RemotePath) = "temp";
    Message_2(nsoftware.BizTalk.SFTP.RemoteFile) = "*.txt";
    Message_2(nsoftware.BizTalk.SFTP.Other) = "ListDirectory=true";

    The solicit response port will return a message with a body containing an XML formatted representation of the directory contents. For instance:

    <directory name="temp">
    <directory name="subDir1">
    <directory name="subDir2">
    <file name="file1.txt">
    <size>size</size>
    <time>time</time>
    </file>
    <file name="file2.txt">
    <size>size</size>
    <time>time</time>
    </file>
    <file name="file3.txt">
    <size>size</size>
    <time>time</time>
    </file>
    </directory>
    </directory></directory>

    Note: The SFTP adapter returns additional CTime and ATime elements representing the CreationTime and AccessTime. CreationTime is only supported on servers that support SFTP protocol version 4 or higher. To enable the adapter to use version 4 or higher you can set the Other property to "ProtocolVersion=4".

    Downloading a File

    Because a solicit response port can return only one message, you can only download one file at a time with the SFTP and FTP send ports. To do this you'll need to set the DownloadSingleFile configuration setting to true. For instance:

    Message_2 = Message_1;
    Message_2(nsoftware.BizTalk.SFTP.SSHUser) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHPassword) = "test";
    Message_2(nsoftware.BizTalk.SFTP.SSHHost) = "server";
    Message_2(nsoftware.BizTalk.SFTP.SSHAcceptServerHostKeyAcceptAny) = true;
    Message_2(nsoftware.BizTalk.SFTP.RemotePath) = "temp";
    Message_2(nsoftware.BizTalk.SFTP.RemoteFile) = "test.txt";
    Message_2(nsoftware.BizTalk.SFTP.Other) = "DownloadSingleFile=true";


    Rachit Sikroria (Microsoft Azure MVP)

    Wednesday, October 26, 2016 8:18 PM
    Moderator
  • Hi Giri,

    You have a few options here:

    • Change the requirement: Like John mentioned, which file we pick? Is it based on time-stamp? Those questions make this requirement seem like it needs some refining.


    • Write a customs adapter: You may choose to write an adapter yourself. I know you can extend WCF Adapters, but am not sure if you can extend the SFTP adapter. Maybe be worth a shot. Here is the scoop on MSDN - Developing Custom Adapters


    • Use a third party adapter: Like Pi_xel_xar mentioned, there are adapters out there that will fulfill that requirement. Here is nSoftware's website mentioning that capability- SFTP Adapter Comparison


    Please note that I do not recommend using products or approaches not supported by Microsoft, and am not in any way associated with products mentioned in this post.


    My Technet Articles

    If you like this or another reply, vote it up!
    If you think this or another reply answers the original question, mark it or propose it as an answer.


    Mauricio Feijo
    www.mauriciofeijo.com


    Wednesday, October 26, 2016 8:36 PM