none
How to make FTP adapter to pull all the files in FTP

    Question

  • Hello,

    I have implemented a small solution which takes files each second from ftp and then it transforms it and send it to another ftp folder using an orchestration with Scope Transaction Type = None.

    It works but I have seen that it takes between 14 and 15 files per second.

    Is it possible to increase the quantity of files being taken by FTP?

    Is there another FTP adapter that is not from MS that could do the job?

    Using FTP Adapter sure it has been difficult as I get throttled several times.

    Thanks in advance for your help.

    Regards,

    Juan


    Juan Pablo Torres

    Thursday, January 19, 2017 2:11 PM

All replies

  • Hi Juan

    By default the FTP adapter creates a batch of files each time it polls a FTP location - it's never a single file per poll. Make sure that the Maximum Files property is set to 0 (default) in the adapter config.

    If it is already set to 0, it could be that you have implemented a singleton orchestration? In that case, you are limited by how soon the singleton orchestration instance can finish processing a single file. In which case, you have to possibly optimize the orchestration processing. It is not an issue with the adapter.


    Thanks Arindam

    Thursday, January 19, 2017 2:49 PM
    Moderator
  • Hello,

    Thanks for your answer. How can I know that I've created a singleton orchestration?

    Regards,

    Juan


    Juan Pablo Torres

    Thursday, January 19, 2017 2:53 PM
  • Hi

    Have you implemented this pattern? In a singleton orchestration, only a single instance of the orchestration would be created for all messages from FTP, as opposed to a new orchestration instance per message polled from FTP(the normal case if you have not implemented this pattern).

    Can you share what is the flow in your orchestration?


    Thanks Arindam


    Thursday, January 19, 2017 3:05 PM
    Moderator
  • Hello,

    I haven't implemented that way.

    The flow is the following:

    - receive a message that comes from the port (without any constraint or filter for activation)

    - transform

    - message assignment to overwrite a promoted property

    - send to a port (ftp port)

    - send to another port (file port)



    Juan Pablo Torres

    Thursday, January 19, 2017 3:16 PM
  • So, in that case, you should get multiple orchestration instances activated at the same time - one instance per new message from FTP. Which means, you really have to optimize the processing in your orchestration - if possible.

    Also, check if some other process is interfering with this flow in terms of throttling. Try creating a separate Host Instance for receiving from FTP, one Host for the orchestration processing and another Host for sending via the SendPort.


    Thanks Arindam

    Thursday, January 19, 2017 3:20 PM
    Moderator
  • Which means, you really have to optimize the processing in your orchestration - if possible

    How can I achieve that? Can you give me some hints please?

    Also, check if some other process is interfering with this flow in terms of throttling. Try creating a separate Host Instance for receiving from FTP, one Host for the orchestration processing and another Host for sending via the SendPort.

    Already implemented. One host instance per process per project (one for receive, one for orch and one for send port).


    Juan Pablo Torres

    Thursday, January 19, 2017 3:23 PM
  • First step would be to check in Admin Console how long each orchestration instance is running for - total execution time.

    Then break up how much time is being spent at each stage/shape in the orchestration. You can use orchestration debugger to figure this out. 


    Thanks Arindam

    Thursday, January 19, 2017 3:30 PM
    Moderator
  • It works but I have seen that it takes between 14 and 15 files per second.

    Is it possible to increase the quantity of files being taken by FTP?

    Is there another FTP adapter that is not from MS that could do the job?

    Using FTP Adapter sure it has been difficult as I get throttled several times.

    Can you clarity?  Are you looking to increase or lower throughput?

    Meaning, what is the specific problem you are facing?

    Thursday, January 19, 2017 9:14 PM
    Moderator
  • Hello,

    Yes I need to take as many files as I can each second that I do the pulling.

    Regards,

    Juan


    Juan Pablo Torres

    Thursday, January 19, 2017 9:16 PM
  • So, what is the actual problem?

    If the Receive Location not keeping up with the files that are appearing in the FTP folder?

    Thursday, January 19, 2017 9:35 PM
    Moderator
  • That's correct.

    Juan Pablo Torres

    Thursday, January 19, 2017 10:07 PM
  • Remove the usage of the orchestration, get the messages in to the messagebox and then use filters on two ports.
    Sunday, January 29, 2017 7:39 PM
  • Hi ,

    Might be worth checking whether you are using default Application host instance for all the Biz Talk processing .

    It might be a thread issue and you can try creating a new host instance for receive ,send and process and divide the processing of your Application within these host instances .


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply

    Wednesday, February 1, 2017 4:35 AM