none
BAM tracking using OrchestrationEventStream RRS feed

  • Question

  • Hello

    We have 2 itineraries

    Itinerary1: Receives a request, transforms and calls an external service (async)  - this uses BRE Policy 1 as resolver

    Note: this request contains one or more repeating records

    Itinerary2: Receives response from the external service and updates the response to DB - this uses BRE Policy 2 as resolver

    Note: Response contains one more repeating records corresponding to the request message

    Everything is working fine and then we added BAM tracking by calling a .Net class (BAMTracker) from BRE. There are 4 static methods - TrackRequest, TrackResponse and TrackDBUpdateComplete - all these methods receive TypedXmlDocument as input

    Policy 1:

     - Call external service

     - Assert BAMTracker

     - Call TrackRequest(message) and Retract BAMTracker

         >>this loops through the records and creates activity records using OrchestrationEventStream (code given below)

    Policy 2:

     - Assert BAMTracker

     - Call TrackReponse(message)

         >>this loops through the records and updates activity using OrchestrationEventStream (code given below)

     - Call internal Service1 to update results to DB

    ---

    ---

     - Call internal ServiceN to update results to DB

     - Call TrackDBUpdateComplete(message) and Retract BAMTracker

         >>this loops through the records and marks records as Complete using OrchestrationEventStream (code given below)

    BAMTracker Class

    public static void TrackRequest(TypedXmlDocument message)

    {

    for each record

    {

        OrchestrationEventStream.BeginActivity(ActivityName, activityId);
        OrchestrationEventStream.UpdateActivity(ActivityName, activityId, "data1", value1);
         ---
        OrchestrationEventStream.UpdateActivity(ActivityName, activityId, "dataN", valueN);
        OrchestrationEventStream.EnableContinuation(ActivityName, activityId, token);
        OrchestrationEventStream.EndActivity(ActivityName, activityId);

    }

    }

    public static void TrackResponse(TypedXmlDocument message)

    {

    for each record

    {

        OrchestrationEventStream.UpdateActivity(ActivityName, token, "data1", value1);
        ---
        OrchestrationEventStream.UpdateActivity(ActivityName, token, "dataN", valueN);
        OrchestrationEventStream.EndActivity(ActivityName, activityId);

    }

    }

    public static void TrackDBUpdateComplete(TypedXmlDocument message)

    {

    for each record

    {

        OrchestrationEventStream.UpdateActivity(ActivityName, token, "status", "ok");
        OrchestrationEventStream.EndActivity(ActivityName, token);

    }

    }

    When the inbound message volume is low, we can see records updated in BAM table (_Completed) but when the volume increases the behavior is unpredictable - we see many records in _Active table where updated values are NULL and they never move to _Completed table (there are many completed records in _Completed table too)

    In addition, irrespective of volume ie even when everything works fine with low volume, we see error in [TDDS_FailedTrackingData] - TDDS failed to execute event. Violation of PRIMARY KEY constraint 'PK__bam_<activity>'. Cannot insert duplicate key in object 'dbo.bam_<Activity>_Continuations'. 

    Troubleshooting revealed that, when volume was high, there were multiple calls to TrackRequest and TrackResponse methods for the same record even though we call this method only once from Policy 1

    Any idea what is going wrong here?






    • Edited by btsdotnet Monday, April 25, 2016 5:42 AM
    Sunday, April 24, 2016 4:31 PM

Answers

  • It seems to be working now. I'll figure out what solved the issue(!) and update the thread soon
    • Proposed as answer by Angie Xu Friday, May 6, 2016 2:19 AM
    • Marked as answer by Angie Xu Monday, May 9, 2016 2:04 AM
    Tuesday, May 3, 2016 8:36 AM

All replies

  • Hi

    I had blogged about something similar to what you are seeing -

    https://blogs.msdn.microsoft.com/biztalknotes/2014/04/01/bam-activities-not-getting-completed/

    For the BAM records that are staying as Active, check if the corresponding Orchestration instances are ending up as Terminated/suspended.

    "If the Orchestration instance encounters a Suspend or Terminate shape, the corresponding BAM activity is still considered to be running"

    Ref- https://msdn.microsoft.com/en-us/library/ee270796%28v=bts.10%29.aspx

    ====================================

    For the 2nd error-

    [TDDS_FailedTrackingData] - TDDS failed to execute event. Violation of PRIMARY KEY constraint 'PK__bam_<activity>'. Cannot insert duplicate key in object 'dbo.bam_<Activity>_Continuations'.

    How are you setting the activityId param in your BAM API calls? Is it some field from the message payload? Are you sure there is no chance of a duplicate call being made with the same activityId ? Probably there is some edge case where you are passing a duplicate/already used value for activityId ?




    Thanks Arindam



    Sunday, April 24, 2016 5:20 PM
    Moderator
  • For the BAM records that are staying as Active, check if the corresponding Orchestration instances are ending up as Terminated/suspended 

    There are no orchestrations here, activity tables are updated from .Net class using OrchestrationEventStream.

    Are you sure there is no chance of a duplicate call being made with the same activityId 

    We set parameter value from payload and there are no duplicates - but we noticed that the static method used to create activity records (TrackRequest) is called multiple times which is causing duplicate inserts to continuations. But we call it only once per payload from itinerary (Policy 1)

    Monday, April 25, 2016 5:34 AM
  • For the BAM records that are staying as Active, check if the corresponding Orchestration instances are ending up as Terminated/suspended 

    There are no orchestrations here, activity tables are updated from .Net class using OrchestrationEventStream.

    Are you sure there is no chance of a duplicate call being made with the same activityId 

    We set parameter value from payload and there are no duplicates - but we noticed that the static method used to create activity records (TrackRequest) is called multiple times which is causing duplicate inserts to continuations. But we call it only once per payload from itinerary (Policy 1)

    Hi,

    Are the policies executing under the context of pipelines, if you have no orchestrations? You should be using OrchestrationEventStream only if you are calling BAM API's from the context of an orchestration instance. Or in other words, if your policy is being invoked from an orchestration.

    If your policies are executing in the context of pipelines/ports, you need to use the MessagingEventStream.

    The key thing here is context/container artifact - even if the API is being called from custom .NET code, you need to figure out where the policy is being invoked from, i.e., pipeline or, orchestration. Based on that, you need to use the correct EventStream type.


    Thanks Arindam




    Monday, April 25, 2016 5:49 AM
    Moderator
  • Yes, policies are executed under pipeline context. Using OrchestrationEventStream this way has been working for other cases so far... For testing, I changed it to BufferedEventStream (this reqd less changes) but now data is not even tracked to _Active table, not sure why. And I still see multiple calls to static methods for the same record

    I think BufferedEventStream is similar to MessagingEventStream except not participating in pipeline context transaction. I'll next try with MessagingEventStream but any idea why BufferedEventStream is not working?

    Monday, April 25, 2016 10:11 AM
  • Fire up SQL Server Profiler and check if the SQL BAM Stored Procs are getting triggered with BufferedEventStream - it should work.

    Thanks Arindam

    Monday, April 25, 2016 10:14 AM
    Moderator
  • Your issue is simple. Check the data that you're using to establish the continuation token. That like the Activity Id should be UNIQUE.

    Multiple inserts of the same record can be a result of

    1. Multiple Tracking Hosts (or Tracking enabled on Multiple Hosts) - Please review all the Host definitions available under the BizTalk Management console and ENSURE that ONLY ONE of these Hosts has "tracking Enabled". The recommendation is to have a dedicated Host ONLY for the purpose of "Tracking"
    2. The next reason is the simples of the lot - duplicate data or selection of a non-unique element as a continuation token.

    I recommend that you examine the data that is being used by you to enable "Continuation". The error indicates that those values are NOT UNIQUE.

    Regards.

    Monday, April 25, 2016 10:47 AM
  • When the volume is high, Please check if you are dropping the same ActivityID related messages.(Terminate all the suspended,Active and Running instances before testing)

    The Primary Key Violation may occur when the same ActivityID/CorrelationID messages been detected by BAM API.

    Also check with the SQL Profiler, to check the overlap of events.


    Regards, Vignesh S

    Monday, April 25, 2016 1:12 PM
  • I'm yet to figure out why this is not working with BufferedEventStream, I will update as soon as I have something to share

    As far as data is concerned, I see from admin console that we do not have duplicate requests and there are no duplicate records inside the request

    We have a single host enabled for tracking - I think tracking host will not move data to user created activities?

    One thing that I cannot understand is - why these methods are called more than once for some requests (I created a log file to view method calls - it shows that some methods are called more than once for the same request message which is causing primary key violation in continuation tables). I'm pretty sure that the Policies call these methods only once per request message


    • Edited by btsdotnet Monday, April 25, 2016 6:04 PM minor correction
    Monday, April 25, 2016 5:58 PM
  • What is your ContinuationToken for the activities (pointed out earlier in this thread by someone else) in the EnableContinuation call? Are you debatching the main payload and your BAM APIs are being called on each debatched message?

    Thanks Arindam

    Monday, April 25, 2016 6:38 PM
    Moderator
  • What is your ContinuationToken for the activities (pointed out earlier in this thread by someone else) in the EnableContinuation call? Are you debatching the main payload and your BAM APIs are being called on each debatched message?

    Thanks Arindam

    Request message is debatched and each record's [ID+Date+"tkn"] is the continuation token
    Tuesday, April 26, 2016 6:51 AM
  • And each debatched message has a unique ID field (in record's [ID+Date+"tkn"])?

    Thanks Arindam


    Tuesday, April 26, 2016 6:59 AM
    Moderator
  • Yes, we have a unique ID field and we append date and 'tkn' to that. For activity instance ID, we use ID+Date as the unique key. 
    Tuesday, April 26, 2016 7:18 AM
  • After changing to BufferedEventStream, I used various options such as 

    EventStream str = new BufferedEventStream(connectionString, 1) followed by flush at the end of the batch

    EventStream str= new BufferedEventStream(connectionString, 0) followed by flush at the end of the batch

    EventStream str= new BufferedEventStream(connectionString, 6) (including begin, update, enable continuation and end activity - there are 6 events in TrackRequest method)

    None of these seem to populate _Active tables


    • Edited by btsdotnet Tuesday, April 26, 2016 7:24 AM
    Tuesday, April 26, 2016 7:23 AM
  • The OrchestrationEventStream is derived from the BufferedEventStrem class so that is not the issue is you insist that there is ONLY Tracking Host enabled.

    The issue is your Continuation Token. Instead of Date use Date Time (yyyyMMddHHmmssfffffff) format. NOTE the fffffff which will be milliseconds and should help you get a very UNIQUE continuation ID.

    Regards.

    Tuesday, April 26, 2016 7:28 AM
  • @Shankycheil

    With BufferedEventStream, I don't see any errors in TDDS_FailedTrackingData (unique key violation error) - meaning it is not tracking data at all...

    Tuesday, April 26, 2016 7:32 AM
  • Did you get a chance to run a SQL Profiler trace to see what is going on? Are the BAM Stored Procs getting invoked?

    Thanks Arindam

    Tuesday, April 26, 2016 7:46 AM
    Moderator
  • I did run sql profiler and see many TDDS _upsertTDDS_streamstatus procs... do you know which SP is supposed to be called?
    Tuesday, April 26, 2016 11:02 AM
  • Guess it should call bam_<activity>_PrimaryImport and/or bam_<activity>_UpsertInstance  procedure? But I dont see them
    Tuesday, April 26, 2016 11:11 AM
  • So the bam_ procedures are actually called by your Tracking Host Instance and not the BAM APIs directly. BAM APIs will only call some internal SPs that move this data to the MsgBoxDb which is used as a staging DB for this. Can you check if your Tracking Host Instance is started. Just restart it and see if it makes a difference.

    Thanks Arindam

    Tuesday, April 26, 2016 11:36 AM
    Moderator
  • It seems to be working now. I'll figure out what solved the issue(!) and update the thread soon
    • Proposed as answer by Angie Xu Friday, May 6, 2016 2:19 AM
    • Marked as answer by Angie Xu Monday, May 9, 2016 2:04 AM
    Tuesday, May 3, 2016 8:36 AM