locked
How to transfer files (any type) based on the details in the received xml message from a file location to a https restful service in BizTalk Server 2010 RRS feed

  • Question

  • I have the following requirements to transfer message and associated files from one location to another. As large number of messages need to be transferred, thus I am looking for an optimized and efficient approach\pattern for implementing this requirement.

    1. First BizTalk schema deployed  Web Service will receive an Order message from the source system. This message will contain order details and information related to the associated files (any type PDF, Excel, text, image, etc) along with this order such as name and path (shared windows network location) from were these files can be picked.
    2. BizTalk will first transfer the order information (excluding associated files) to the destination system Web Service (soap) and receive an Order ID in the confirmation response.
    3. After receiving a successful transfer confirmation response from Web Service (solicit request-response), the associated files will be picked from the source network location and send to the destination http Restful service (put) one by one.
    4. Finally, a Create Attachment request will be send to the destination system web service for all the attachment files that were send for this Order.

    Can you please explain with some sample\example the best way to implement this requirement in BizTalk 2010.


    • Edited by MSBhamra Sunday, August 14, 2016 5:27 AM
    Wednesday, August 10, 2016 6:31 PM

Answers

  • Based on your description, your situation really isn't that complicated.

    We have to ask, what exactly do you mean by "large number of messages"?

    1. Assuming this is a document based SOAP Service with a schema, this is just basic BizTalk.

    2. Again, this is just basic BizTalk.  Is there any special circumstance we should know about.

    3. This is also mostly basic BizTalk.  The only special scenario is the list of files and picking them up.  Since there is no Two-Way File Adapter, you will have to retrieve the files in code.  For this, it's best to write a helper module that you call from inside the Orchestration.  the File class will return the contents as a Stream so creating a BizTalk message from that is pretty easy.  Then, it's just basic BizTalk to send that file content to the REST service.

    4. Finally, just basic BizTalk.

    Thursday, August 11, 2016 4:43 PM
    Moderator
  • Hi MSBhamra,

    Refer below screen shots which capture the flow you need to implement in your Biztalk orchestration ti achieve your intended behaviour.

    TheLoop shape comes just below the receive shape in first screen shot

    But I will reiterate again that Biztalk 2010 does not have out of the box support for REST , you may have to refer to the links I posted in the first reply to get the REST working.

    As forparallel convoy the scenario can be implemeted but I will suggest to go with the flow that I have mentioned above, it is the simplest one. Wheteher the parallel convoy add any performance advantage or not that needs to be verififed through load test.

    Regards,


    Mandar Dharmadhikari

    Wednesday, August 17, 2016 9:58 AM
    Moderator

All replies

  • Hi ,

    Well i dont know the load/traffic which you are expecting here but 2 solutions you may think:

    A) Messaging level : 

    1)Create a input request schema and publish this as a service .

    2) Receive the request message and call the destination service through the send port (2 way).

    3) Create a custom pipeline component for the send port (receive pipeline for response)

    4) In the component check the orderID and then pick the pdf file from network location and call the rest service using  method written in C# in the pipeline component itself.

    5) Get the response from Rest service and create a response message in the pipeline component and publish it to the message box.

    6)Send this response to the corresponding system

    You can do the same in the orchestration  as well but the cost of performance.

    Note: you have to also implement the exception handling especially for Rest.


    Manish

    Thursday, August 11, 2016 9:57 AM
  • Hi,

    You can use an orchestration with Microsoft.XLANGs.BaseTypes.Any messege in the Messege Type of your Activate Rcv shape.

    Further create an property schema holding the property value of the BussinessRoutingId based on the msg you get,promote this property and set to a specific value on a custom pipeline level,Further  in the filter expression shape of the orchestration have the Promoted property look for the unique BussinessRoutingId and do your subsequent business processing.


    Regards Pushpendra K Singh

    Thursday, August 11, 2016 2:15 PM
  • Hi MSBramha,

    Let me take a shot at it. this is going to be along post

    first of all...you cannot consume the resp apis from Biztalk 2010 directly, that functionality is available in Biztalk 2013 version.

    If you still plan to consume the REST api, refer to the below posted link

    Invoke a REST api in BT 2010

    Now that being said, I would suggest the way

    1) Introduce a unique identifier like Request ID in your xmlorder request which will be sent to Biztalk

    2) Also make the REST api return the same unique ID along with the OrderID in the response

    3) That being said, what you now need to do is promote the Unique ID in the Message which is being sent to Biztalk and also promote the same ID in the response received from the service which provides the order iD. You will need to use a custom pipeline component to do so

    4) Now create an orchestration which exploits the concept of parallel convoy. You can find step by step guide to so 

    here and here

    5) This will make sure that your orchestration will start the further process only when both messages from the Biztalk service and the REST api are arrived..that is the beauty of parallel convoy.

    6)Once both the messages are received...now you have the folder location and file details in the message which was sent to Biztalk and order id from the REST api

    7)Introduce a looping shape now to send each of the files

    )8 Create a helper class and pick up the files from the folder location...you already have the details  :) in your orchestration now.

    8)and send the files to the api which will return the the file id in the response...orchestration will make sure that the file id is correlated back to you. Now save the ID

    9) For each of the call to the api which returns a file ID, you can create the message which you need to send to the destination webservice which is the final call in the process...you can use mapping for it.

    Thats it !!!

    now as for an example, yours is a specific process.....there is no such example available for it...but I will try to ceate a working example for the same and post it on technet wiki as soon as possible... 

    feel free to point out if I have misunderstood the requirement or if you fell that something is out of order.

    Regards,



    Mandar Dharmadhikari

    Thursday, August 11, 2016 4:00 PM
    Moderator
  • Based on your description, your situation really isn't that complicated.

    We have to ask, what exactly do you mean by "large number of messages"?

    1. Assuming this is a document based SOAP Service with a schema, this is just basic BizTalk.

    2. Again, this is just basic BizTalk.  Is there any special circumstance we should know about.

    3. This is also mostly basic BizTalk.  The only special scenario is the list of files and picking them up.  Since there is no Two-Way File Adapter, you will have to retrieve the files in code.  For this, it's best to write a helper module that you call from inside the Orchestration.  the File class will return the contents as a Stream so creating a BizTalk message from that is pretty easy.  Then, it's just basic BizTalk to send that file content to the REST service.

    4. Finally, just basic BizTalk.

    Thursday, August 11, 2016 4:43 PM
    Moderator
  • Hi MSBhamra,

    did you have look at the solution I provided...I am working on a sample..would upload it asap


    Mandar Dharmadhikari


    Friday, August 12, 2016 8:41 PM
    Moderator
  • Hello Mandar,

    The below diagram will make the requirements more clear. I would like to highlight the following points.

    1) I don't have control over the Destination Systems and their web service methods\api's. These systems are from other companies and will work as per their existing design.

    • The source system will send the Order Create Request (+ related attached files information such as name and location of the files) message to the BizTalk Schema exposed web service. Remember: Their can be more than one files attached against one Order.
    • On the Destination system web service their will be two Web methods, one for creating the Orders "Create Order" and the second for creating attachments "Create Attachment".
    • The Create Attachment web method can create attachment for multiple files in a single request. 

    2) The below diagram (process) will work as explained below:

    • The Source System will send the request to BizTalk Schema exposed Web Service. This request will contain Order + Attached (Files) information.
    • Their can be more than one files attached (associated) with any order.
    • After performing the transformation, the BizTalk Orchestration will first send the Order message (excluding attachment files info) to the Destination System "Create Order" Web method. And receive OrderID in the response.
    • Note: The transfer of Create Order message is of the highest priority, thus in my opinion this should be done first.
    • Now the BizTalk process will get all the attached files from the Source System for the above transferred Order and will put\post all these files one by one to the restful service (so if three files are attached to an order then the system will call restful service three times as only one file can be posted in a single call).
    • Finally, after creating order and sending attached files to the destination system, BizTalk process will send a "Create Attachments" request message to the Destination System Web method. In the single request information related with all the attachment file will be send along with the OrderID.

    I hope I am able to explain the process better now. Let me know if you have any other question. I will appreciate of you can help me to design this in BizTalk and send a POC example project.

    Also, in terms of performance BizTalk can receive 1500 to 2000 Order Create requests at any point of time.

    • Edited by MSBhamra Sunday, August 14, 2016 5:11 AM
    Friday, August 12, 2016 9:59 PM
  • Friday, August 12, 2016 10:02 PM
  • Please see my above response to Mandar and the diagram. We might receive 1500 to 2000 Create Order request at the same time.

    Thank you


    • Edited by MSBhamra Friday, August 12, 2016 10:07 PM
    Friday, August 12, 2016 10:06 PM
  • What exactly do you mean by "at the same time"?
    Saturday, August 13, 2016 1:17 PM
    Moderator
  • I mean at any given point of time the interface can receive any were from 1500 to 2000 order create messages and each order can have multiple attached files.
    Sunday, August 14, 2016 3:19 AM
  • Hello Mandar,

    Yes, I have gone through the solution that you had mentioned earlier. And I hope you have gone through the requirement points and diagram that I uploaded on 12th August. Also, I have slightly updated the original post.

    I have following questions

    1) How can parallel convoy pattern work in this scenario as their can be any number of attached files with an order. We will have to call restful service multiple time to transfer all these files (may be in a loop)?

    2) How are we going to send files (any type) to a restful service? The example that you are pointing to in your solution does not send the files, it only sends the text. Can you show me how we can tweak this sample to send (PUT) the files to the restful service?

    Thank you


    • Edited by MSBhamra Sunday, August 14, 2016 5:30 AM
    Sunday, August 14, 2016 5:11 AM
  • Hi MSBhamra,

    Refer below screen shots which capture the flow you need to implement in your Biztalk orchestration ti achieve your intended behaviour.

    TheLoop shape comes just below the receive shape in first screen shot

    But I will reiterate again that Biztalk 2010 does not have out of the box support for REST , you may have to refer to the links I posted in the first reply to get the REST working.

    As forparallel convoy the scenario can be implemeted but I will suggest to go with the flow that I have mentioned above, it is the simplest one. Wheteher the parallel convoy add any performance advantage or not that needs to be verififed through load test.

    Regards,


    Mandar Dharmadhikari

    Wednesday, August 17, 2016 9:58 AM
    Moderator