BTS - solution design RRS feed

  • Question

  • We have a requirement wehre we need to integrate some Oracle data with  AS 400 ..

    the data in question is text data + mages..Basically there are 10-20K  records with images..On the AS 400 side they want the text part in one single CSV file and all the images as separated .jpg file  over FTP.

    I was exploring various design options..I don't want the images to be flowing thru the message box, one thing the size being big and also  trying to avaoid orchestration for this and use a pure messaging solution..

    1. I was thinking of doing this ,

         1. Create  recieve location, execute a procedure to get  only the text part of the data

          2. Create 2 send ports   that uses FTP

                           1.One uses the  custom pipe line with flat file dissembler and  transforms the  incoming data into csv and uploads it to the FTP , which meets the  one requirement.

                            2. Create  another send port subscribed to the same message , but internally it will execute a  custom send pipe line component which invokes a  .net  component  , calls a oracle stored procedure to get just the  image data from the  Oracle database , loop thru the  data reader which has images and send the images one by one to the FTP send port .

     I am not sure its possible, because it being an hack, I am trying to generate multiple message from the send pipe line.. If its not possible I can do that part  in a custom recieve pipe line and then store the images in a local file store. Then  repeat the same step as above to send the data..

    What do u guys think abt the solution. I am just converned abt the impact of the send pipe line with a data reader loop and ftp port attached to it down stream..Is there a better way to handle  this ?

    1. The other solutions is  not much of BizTlak, develop a pure .net  component loop thru the data reader   and upload the image streams + text file  directly to the FTP using .NET FtpWebrequests. No adapters  or any other component





    • Edited by Kyrn Monday, October 31, 2011 9:59 PM
    Monday, October 31, 2011 8:31 PM

All replies

  • Hi,

    It very much depends on your exact requirements if you should BizTalk for this or just use a custom .Net solution. You can view BizTalk as "just" a toolbox that brings you a lot of funtionality that is hard and time consuming to build yourself. If you need scalability, transactions, reliability, monitoring, etc. you should use BizTalk because it brings this out of the box.

    If you decide to use BizTalk for this I think I would go fo the option where you stream the images to disk in a receive pipline. The receive pipeline should also create a small messages (containing the filenames of the files stored on disk) that flows through the message box. In the send pipeline the files are streamed back into the message and you can use FTP adapter to deliver the files to AS400. When you this you have the flow completely implemented in BizTalk which allows for better tracebility. (e.g. when was message A received and when was it sent to AS400). Besides that you can use polling for retrieving data from Oracle.





    Randal van Splunteren 2 x MVP BizTalk Server, MCTS BizTalk Server
    my blog

    Check out the PowerShell provider for BizTalk
    Please mark as answered if this answers your question.
    Tuesday, November 1, 2011 10:57 AM
  • Question is do you want it in real-time or batch process? If this is a batch process you can build it using SSIS so you don't need to worry about the host for your application. I would normally use BizTalk like what Randal said, if it requires scalability etc. etc. in real-time integration but SSIS for batch processing.

    Randy Aldrich Paulo

    MCTS(BizTalk 2010/2006,WCF NET4.0), MCPD | My Blog

    Tuesday, November 1, 2011 1:40 PM
  • its an ETL batch job...SSIS is the right tool but for the time being we are tryign to get it done with BTS because we have the BTS expretise but  no SSIS ..
    Tuesday, November 1, 2011 2:49 PM
  • 1. Image files being big If I use the WCF  oracle adapter the whole images , in one poll all the  10,000 images will come in as a single big xml file and then I will have to split it based on some XSLT etc...Instead of that I am planning to execute a oracle stored procedure inside the recieve pipe line , loop thru a data reader of images and save the images to the disk..

    Only thing is now I will have to write a scheduler to clean up the image files written locally..


    2. Can I do this in the send pipe line....Instead of doing creating local image files in the  in the recieve and then again grabbing it back in the send pipe line, can I do this in one shot in send pipe line..Loop thru a data reader and create the image  messages and send it out to the ftp adapter. Not sure Send pipeline can handle multiple files coming out of one instance ??


    Tuesday, November 1, 2011 2:58 PM
  • learning curve in SSIS is not too deep, You can easily call .NET components on it using Script task (C# or VB.NET) and data flow looks like orchestration in BTS.  Again it's up to you. For me I would rather invest probably 1-2days learning SSIS than spending a week trying to make it work with BTS (like you said 'hack').


    Randy Aldrich Paulo

    MCTS(BizTalk 2010/2006,WCF NET4.0), MCPD | My Blog

    • Edited by Randy Paulo Tuesday, November 1, 2011 3:03 PM
    Tuesday, November 1, 2011 3:03 PM
  • right now we don't have an SSRS environment and setting up for just this proj is not feasible at this point.

    I am inclined towards the  pure .net solutions,this being a once a day batch/etl kind of job , I am not worried  much abt the tracking etc..

    Wednesday, November 2, 2011 2:06 PM