locked
Architectural solution for a complex file processing system RRS feed

  • Question

  •  

    Hi,

    I am trying to develop the high level of architecture of a system to process flat files.

    Briefly the system is like this.

    Files are sent from an external system to a specified location in our file system.

     

    There can be may kinds of files with different extensions but all of them has to go through below mentioned process.

     

    The file must be picked up by a process/application then go through what is called a primary validation where data is checked for consistency, length header etc.

    If errors are reported they are to be recorded in a database. if primary validation passes, then a seconday validation runs which validates each of the data in these flat files against variuos master databses.As in the case of primary validation, erros are to be recored in a DB and if every thing else goes fine, then they are forwared to another system.

    Ehat would be the best technology to implement solution for this kind of  problem. Biztalk, Windows work folw or SSIS or any thig else.

    Thanks in advance

    Friday, January 16, 2009 12:14 PM

Answers

  • I agree I would use SSIS if the data was to end up in a database. 

     

    However, I wouldn't use workflow foundation or SSIS for a file processing system of any complexity.

     

    If you have lots of disparate sources, transformations and target systems I would definitely use biztalk. 

     

    I disagree that using biztalk comes down to a financial issue of whether you have the licenses already.  If the file processing system is of any complexity biztalk would be the cheaper option in the long term.

    • Marked as answer by Dinesh Upreti Wednesday, March 25, 2009 5:33 PM
    Tuesday, January 20, 2009 8:49 AM

All replies

  • Biztalk seems the best match for those requirements.  Biztalk can be used to integrate systems in the manner you describe - take input from one system, process it and forward it to another system.

     

    Biztalk may be over engineering if you only want to integrate a single system and only have a couple of formats that probably won't change or be added to.  However, if it is a complex file processing system as you indicate I would definitely investigate Biztalk or similiar products first.

    Friday, January 16, 2009 12:22 PM
  • If your target for the data stream is a database then I would recommend SSIS.  SSIS is Microsoft's ETL (Extract, Transform and Load).  Validating the data is just another step in the process.

     

    If this is not a typical ETL, and

    If you already have license for BizTalk or have enough budget to afford BizTalk then use BizTalk.  Otherwise Windows Workflow Foundation would certainly help your needs.  WorkflowFoundation is the engine running inside BizTalk.  Hence, you could certainly use BizTalk except lot of features are available out of the box on biztalk that you may have to handcode using Workflow foundation.

     

    { Gaja; }

    Tuesday, January 20, 2009 3:52 AM
  • I agree I would use SSIS if the data was to end up in a database. 

     

    However, I wouldn't use workflow foundation or SSIS for a file processing system of any complexity.

     

    If you have lots of disparate sources, transformations and target systems I would definitely use biztalk. 

     

    I disagree that using biztalk comes down to a financial issue of whether you have the licenses already.  If the file processing system is of any complexity biztalk would be the cheaper option in the long term.

    • Marked as answer by Dinesh Upreti Wednesday, March 25, 2009 5:33 PM
    Tuesday, January 20, 2009 8:49 AM