locked
Picking Large file from a FTP location RRS feed

  • Question

  • Hello all,

    I have a schenario in which BizTalk will receive a file (.Zip) in a FTP folder. But the source application will take time to write the entire file.
    BizTalk should wait until the file is written completely. Once the file is completely written, BizTalk should then pick the file for further processing.

    What is the best way to do this?

    Regards

    BizTalk Beginner
    Wednesday, October 28, 2009 6:08 AM

Answers

  • The FTP adapter has no built in functionality to get the information of the file that is being written to the FTP. If the adapter is polling the FTP location it will pick up the file. You can use the .NET code for polling the ftp location, you can start from here http://www.codeproject.com/KB/IP/FtpClient.aspx

    You can also customize the open source ftp adapter at codeplex and ensure that it does not try to delete the file after getting it. http://www.codeplex.com/SftpAdapter

    With this functionality in hand you can cache the version of the file at your file location at one polling interval and after polling and caching the ftp adapter will not delete the file. At the second interval e.g. 10 mins you can poll again to cache the file again. If the cache versions file size are the same you can pick the last version and start processing.

    I think its a bit lengthy approach and time consuming. I would try for the .NET component its easier and faster solution.
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this as answer if it helps
    Wednesday, October 28, 2009 2:54 PM
  • Write your file to temporary file name (*.zip.wip or wip-*.zip) and rename after the file has been written completely. Of course update the file mask of the ftp adapter to grap files if and only if the file names do not match these temporary names.
    Wednesday, October 28, 2009 3:13 PM

All replies

  • Hi,

    There may be multiple ways to acheive this.

    1. Create a .txt file on the completion of the .zip file.

    2. Enable the orchestration to pick the .zip only when it see the .txt file [ let the first recv shape look for .txt ]


    HTH
    Vijaykishan | http://v2attechhorizon.blogspot.com | Please mark this as answer if it helps
    Wednesday, October 28, 2009 6:28 AM

  • Hi,

    Please see my comments below:

    1. Create a .txt file on the completion of the .zip file. (How do BizTalk know whether the file is completly written?)

    Regards

    BizTalk Beginner
    Wednesday, October 28, 2009 6:40 AM
  • Hi,

    Your source application knows when ur .zip file gets completed.

    At the end of zip file creation code , create a .txt file.
    Vijaykishan | http://v2attechhorizon.blogspot.com | Please mark this as answer if it helps
    Wednesday, October 28, 2009 6:44 AM
  • Currently this is not in existence. Source application will write only the file (.Zip), nothing other than that. I should be able to handle in BizTalk.

    I have a thought, by checking the file size growth. But how to acheive that in BizTalk port level without using any custom c# method.
    BizTalk Beginner
    Wednesday, October 28, 2009 6:47 AM
  • Here are a couple of suggestions for you to consider but they do involve tweaking the system that produces the file:

    1. Write out the file to a temporary folder first, then move it in to the FTP folder.

    2. Some FTP servers do have the capability to know when a file is still being written to the file system; and in these cases, you can specify that the FTP server does not make the file available until it has finished being written.

    On a general point, if your file is flat, ensure it has a header and footer which BizTalk will check for. This way, if the file has been received prematurely by BizTalk you can detect it.

    Myles


    Catch me on twitter: http://twitter.com/mjthinkscape
    What I do: http://www.thinkscape.com/Succeed-With-BizTalk/
    Wednesday, October 28, 2009 7:48 AM
  • BizTalk dont have the control in the source part. Some application will write the file to a FTP location. That application will take some time say (10min) to complete writing the entire file. Now, BizTalk should not receive the file immediately, instead it should pick the file once the file is written completely.


    BizTalk Beginner
    Wednesday, October 28, 2009 8:52 AM
  • I think you misunderstood me: my suggestions have nothing to do with changing the BizTalk side. You have to change the side that produces the file, the "application". If you don't have control over the side that produces the file for BizTalk, then my suggestions will not help you.

    Wednesday, October 28, 2009 9:15 AM
  • Yes i know. That's what i mentioned in my previous comment that i dont have control over the source application. Thanks for your suggestion anyway...
    BizTalk Beginner
    Wednesday, October 28, 2009 9:20 AM
  • You are welcome. Sorry I could not help you with this scenario.
    Catch me on Twitter
    Wednesday, October 28, 2009 9:38 AM
  • You could write a simple console application or windows task that monitors all incoming files. You could check the size, if it is not changed in a time interval, you can consider that it is compleated, so you can create an empty txt file with the same name of the zip file, then a BizTalk Orchestration will be activated and depending on the txt file name, the orchestration can pick up the correct .zip file.
    HTH!!

    Massimo

    Massimo
    • Marked as answer by SKGuru Wednesday, October 28, 2009 11:18 AM
    • Unmarked as answer by SKGuru Wednesday, October 28, 2009 11:25 AM
    Wednesday, October 28, 2009 10:05 AM
  • Hey thank you Lisignoli... I have done the same thing. However am not sure whether, Is there any best practise for this schenario. Please suggest...
    BizTalk Beginner
    Wednesday, October 28, 2009 11:18 AM
  • The FTP adapter has no built in functionality to get the information of the file that is being written to the FTP. If the adapter is polling the FTP location it will pick up the file. You can use the .NET code for polling the ftp location, you can start from here http://www.codeproject.com/KB/IP/FtpClient.aspx

    You can also customize the open source ftp adapter at codeplex and ensure that it does not try to delete the file after getting it. http://www.codeplex.com/SftpAdapter

    With this functionality in hand you can cache the version of the file at your file location at one polling interval and after polling and caching the ftp adapter will not delete the file. At the second interval e.g. 10 mins you can poll again to cache the file again. If the cache versions file size are the same you can pick the last version and start processing.

    I think its a bit lengthy approach and time consuming. I would try for the .NET component its easier and faster solution.
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this as answer if it helps
    Wednesday, October 28, 2009 2:54 PM
  • Write your file to temporary file name (*.zip.wip or wip-*.zip) and rename after the file has been written completely. Of course update the file mask of the ftp adapter to grap files if and only if the file names do not match these temporary names.
    Wednesday, October 28, 2009 3:13 PM