none
Ordered Delivery on BizTalk Dynamic port

    Question

  • Hi all.

    I know this is yet another question on ordered delivery of messages. But, this is an entirely different scenario and hence a separate question.

    • There are files 'One' and 'Two'
    • 'One' will be send first and 'Two' follows.
    • Both are send to the same send port and same folder location ('One' is send to operation1 and 'Two' is sent to operation2).
    • I am using a dynamic send port.
    • 'One' will be a large file (around 5-6 Mb) while 'Two' will always be around 1Kb
    • I need to make sure that 'Two' gets written only after 'One' is completely written to target.
    • I tried using delay before sending 'Two' (of 5 minutes) but this is not a reliable solution. whenever there is a network Lag, 'Two' reaches first.
    • I tried setting 'FILE.AllowCacheOnWrite' property to false. still doesnt help.
    • What I see on the target location is that 'One' is getting created first but all its contents are not downloaded at that moment itself. Now since 'One' is already created, BizTalk is writing 'Two' as well and all its contents are downloaded at that instance itself but 'One' completes downloading only after a while since it is larger and therefore the sequence is not followed.
    • Both 'One' and 'Two' are created inside the orchestration itself.

    Also, clients are not willing to send an acknowledgement when 'One' is written so that BizTalk can trigger 'Two' and therefore the acknowledgement solution is also not possible as of now.

    Can someone please help?


    Tuesday, August 2, 2016 1:39 PM

Answers

  • Hi

    You can use Delivery Notification on the logical SendPort to achieve this. You have to set "Delivery Notification" = Transmitted, and send the first message within a non-transactional, synchronized Scope shape (Transaction = None, Synchronized=True). The Send Shape for the 2nd message will be placed after the first Scope shape/Exception handling block for the first message.

    This pattern is described here-

    http://kentweare.blogspot.in/2007/11/biztalk-delivery-notification.html

    Note that if you have to use a Dynamic SendPort, you may have to setup your own DeliveryNotification by setting the promoted property of BTS.AckRequired to true, and initializing the BTS.CoorelationToken to new guid -

    http://www.agilior.pt/blogs/bruno.camara/archive/2008/04/17/4344.aspx

    Also make sure that the Retry Count is set to 0 on the first message that you are sending out via an Expression Shape-

    OrchMessage(BTS.RetryCount) = 0; (in case of static SendPort, this can be set in the physical SendPort)

    This is needed for Delivery Notification(ACK/NACK) to be sent back to the orchestration.


    Thanks Arindam








    Tuesday, August 2, 2016 2:06 PM
    Moderator
  • Hi J.Mathew,

    I still see that only acknowledgement solution will work in this case,your client need to send anacknowledgement only after that you should create B and then push it over the network.

    Regards,

    Mandar Dharmadhikari


    Mandar Dharmadhikari

    Thursday, August 4, 2016 8:53 AM
    Moderator
  • Hi Arindam,

    I know this is a bit too late but I think this reply might help people in future (It took us so many months to conclude the investigation with the help of Microsoft and to implement a workaround solution).

    I was getting inconsistent behaviour with ACK/NACK in dynamic port and finally we opened a case with Microsoft and they confirmed that ACK/NACK is not entirely reliable in case of 'dynamic ports'. Once BizTalk hands over the message to the file adapter, BizTalk no longer has any control over the message and the acknowledgement can most likely be from the file handler, and the message still can fail at some point in the transmission/the lock on the first message may not be released despite sending out an ACK signal.

    While the acknowledgement works fine in majority of situations, that is not the case always. I implemented a workaround solution to overcome this issue by implementing a manual retry using an approach very similar to what you suggested. Let me explain below:

    1. File one is send using a long running Transactional scope with 'Delivery Notification = Transmitted'.

    2. Exception handler is created to catch delivery notification failure error for the scope.

    3. The whole scope is put inside a loop and the condition for retrying is set based on whether delivery notification exception was caught or not.

    4. Now, we have set flags to identify whether the file was sent successfully or not based on the number of times the loop executed and the second file is sent only if this flag is true. Thus avoiding the risk of sending the second file before the first file is written.

    5. This solution is entirely dependent on Delivery Notification and we might face the ACK/NACK issue here as well rarely (Some times the orchestration doesn't get an ACK/NACK and goes to dehydrated state for an indefinite time). This issue is resolved by setting a time out on the scope

    This solution is being tested in production since two months now and seem to be working fine. Yes, this is definitely a work around but I think this is the best I could get for this particular scenario. Happy to help if you need more clarification :) .. And thanks for your inputs on the issue ..

    Monday, January 30, 2017 6:32 AM

All replies

  • Hi ,

    Ordered delivery will work in the order in which the messages are published to the message box.

    In your case you need to control the publishing of the Messages to the database...i.e......let the message 1 process first and get the acknowledgement and then only publish message two.

    Try looking into the sequential processing of messages but you will need some indicator that the File One has been completely processed.

    Regards,


    Mandar Dharmadhikari


    Tuesday, August 2, 2016 2:02 PM
    Moderator
  • Hi

    You can use Delivery Notification on the logical SendPort to achieve this. You have to set "Delivery Notification" = Transmitted, and send the first message within a non-transactional, synchronized Scope shape (Transaction = None, Synchronized=True). The Send Shape for the 2nd message will be placed after the first Scope shape/Exception handling block for the first message.

    This pattern is described here-

    http://kentweare.blogspot.in/2007/11/biztalk-delivery-notification.html

    Note that if you have to use a Dynamic SendPort, you may have to setup your own DeliveryNotification by setting the promoted property of BTS.AckRequired to true, and initializing the BTS.CoorelationToken to new guid -

    http://www.agilior.pt/blogs/bruno.camara/archive/2008/04/17/4344.aspx

    Also make sure that the Retry Count is set to 0 on the first message that you are sending out via an Expression Shape-

    OrchMessage(BTS.RetryCount) = 0; (in case of static SendPort, this can be set in the physical SendPort)

    This is needed for Delivery Notification(ACK/NACK) to be sent back to the orchestration.


    Thanks Arindam








    Tuesday, August 2, 2016 2:06 PM
    Moderator
  • Hi,

    using a sequential convoy should fit your need.

    https://psrathoud.wordpress.com/2014/12/25/sequential-convoys-singleton-orchestration-in-biztalk-demo/


    Regards Pushpendra K Singh

    Tuesday, August 2, 2016 5:09 PM
  • Why are you using a Dynamic Port?

    If you use an Orchestration to 'process' One, then Two, this shouldn't be an issue at all.  You do have to use the same Orchestration Send Port so they are published in order and are subscribed, however, by the same physical Send Port.

    There must be something else going in as this is a trivially easy scenario.

    Wednesday, August 3, 2016 1:50 PM
    Moderator
  • Hi Arindam,

    Thanks for the detailed explanation. I tried this but still there is problem.

    My observations:

    • File 'One' is getting created first (the way it is supposed to happen). But size is 0Kb. Gradually size increases bit by bit
    • While 'One' is still downloading its contents (Let's say it downloaded 100Kbs), File 'Two' also gets created in the folder (I was just monitoring the folder manually)
    • What I am confused about is when is the acknowledgement sent back to orchestration. Is it sent right after creation of the file on the folder location or after complete downloading of its contents? because if an acknowledgement is sent on creation itself and not after complete download, it won't help my situation.

    Do you have any idea about this?

    Thanks again

    Thursday, August 4, 2016 8:22 AM
  • Hi Mandar,

    You are right. But the sequence is maintained while processing the messages, Message one is processed and sent, and only then message two is processed. the Problem is once both the files are on the network, the file 'Two' gets downloaded before 'One' because of size difference.

    Thursday, August 4, 2016 8:31 AM
  • Hi J.Mathew,

    I still see that only acknowledgement solution will work in this case,your client need to send anacknowledgement only after that you should create B and then push it over the network.

    Regards,

    Mandar Dharmadhikari


    Mandar Dharmadhikari

    Thursday, August 4, 2016 8:53 AM
    Moderator
  • Hi

    As far as I know, as long as the SendPort instance does not encounter an exception during transmission, it will generate an ACK back to the orchestration in the happy flow. In your case, it looks like this does not translate to completion of writing the file - probably due to Windows Async I/O.

    Do you see the same behavior when you write to a local FILE path?

    Also, do you really need the dynamic SendPort? The pattern you want is easily implemented with a Static SendPort with Ordered delivery enabled.


    Thanks Arindam

    Thursday, August 4, 2016 9:26 AM
    Moderator
  • Hi J,

    You can achive this by using a static port in your orchestration.

    If u have a sequential convoy where u are sending these two files from, U can enable DeliveryNotification= Transmitted for the first send.

    ANd then u can have ur second send after that. In the exception handler of delivery failure for 1st message u can code what ever flow u want.

    Delivery Notification makes sure about completeness of file


    Pi_xel_xar

    Blog: My Blog

    BizTalkApplicationDeploymentTool: BizTalk Application Deployment Tool

    LinkedIn: LinkedIn

    Thursday, August 4, 2016 9:32 AM
    Answerer
  • Hi Arindam,

    I know this is a bit too late but I think this reply might help people in future (It took us so many months to conclude the investigation with the help of Microsoft and to implement a workaround solution).

    I was getting inconsistent behaviour with ACK/NACK in dynamic port and finally we opened a case with Microsoft and they confirmed that ACK/NACK is not entirely reliable in case of 'dynamic ports'. Once BizTalk hands over the message to the file adapter, BizTalk no longer has any control over the message and the acknowledgement can most likely be from the file handler, and the message still can fail at some point in the transmission/the lock on the first message may not be released despite sending out an ACK signal.

    While the acknowledgement works fine in majority of situations, that is not the case always. I implemented a workaround solution to overcome this issue by implementing a manual retry using an approach very similar to what you suggested. Let me explain below:

    1. File one is send using a long running Transactional scope with 'Delivery Notification = Transmitted'.

    2. Exception handler is created to catch delivery notification failure error for the scope.

    3. The whole scope is put inside a loop and the condition for retrying is set based on whether delivery notification exception was caught or not.

    4. Now, we have set flags to identify whether the file was sent successfully or not based on the number of times the loop executed and the second file is sent only if this flag is true. Thus avoiding the risk of sending the second file before the first file is written.

    5. This solution is entirely dependent on Delivery Notification and we might face the ACK/NACK issue here as well rarely (Some times the orchestration doesn't get an ACK/NACK and goes to dehydrated state for an indefinite time). This issue is resolved by setting a time out on the scope

    This solution is being tested in production since two months now and seem to be working fine. Yes, this is definitely a work around but I think this is the best I could get for this particular scenario. Happy to help if you need more clarification :) .. And thanks for your inputs on the issue ..

    Monday, January 30, 2017 6:32 AM