Message Tracking

Question

OK, I have been trying to locate an answer for this and have been unsuccessful thus far.  I am hoping someone here may know the answer to my question.

 

I need to create a complete copy of a message at three stages as it passes through the BizTalk processes.

1. Pre-Processing Copy - I need a clean unaltered copy of the message as it is submitted by the client.
2. Mid-Processing Copy - I need a copy of the XML which will be shredded and stored in the SQL database
3. Post-Processing Copy - I need a copy of the re-assembled response to the client before it is sent.

I have looked at several things, including the built-in port Tracking feature (can't locate where it stores the whole message...if it does that).  Can someone please help me understand where BizTalk stores these tracking copies and/or a better alternative to accomplish the message tracking as described above.

 

Thanks,

 

Bret

Answer 1

From your post, it sounds as though you want to be able to query for message bodies from a database directly - as opposed to having to use an API for retrieval.

If that is the case, I don't think tracking is your solution.  I would look at storing full message bodies using BAM, or your own custom database infrastructure. 

Scott Colestock
www.traceofthought.net

Answer 2

Scott,

 

Thank you for the quick reply.  Yes I am interested in preserving the original format with full contents of the message body.  I have looked at BAM, it seems to only capture promoted message body elements versus the entire message body.  I turned on the checkbox for Send/Receive ports to track message bodies before port processing and after port processing respectively.  I can not figure out where this information is stored so I can obtain the original message should I need to.

 

I do not have a need to store the message body in a database, a tracking database could simply store information about the .XML, .TXT, .EDI, .??? file which is physically stored on the hard drive.  I would presume you would want something along the lines of a Date/Time stamp, 1/2/3 to indicate Pre/Mid/Post processing, file location and file name type of fields in a physical storage tracking model.

 

Right now I am evaluating the path I will recommend for our team regarding message tracking.  At present I am leaning towards a custom solution.  I have been trying to research Message Tracking solutions for BizTalk and have pretty much come up empty handed as everything points me back to BAM and other internal BizTalk tracking methods.  Would you happen to know of any other methods to accomplish my tracking requirements?

 

Thanks,

 

Bret

Answer 3

I have had requirements in the past to track the original contents of a file.  i.e. Track Message body before port processing.  We have also had to build a GUI that will display the contents of the message.  By tracking the associated BTS.MessageID, you can then retrieve the contents out of the BizTalkDTADB using WMI.  We track the MessageIDs and associated Meta data inside of BAM.  We then have a ASP.net gui that queries the BAM view which contains this MessageID.  Note that once messages hit the ports and are potentially transformed that you can get a new MessageID for the new message.  So you would have to track all of the associated MessageIDs.

 

The other thing you need to be careful about is that you cannot have infinte growth on you BizTalkDTADb without a degration of performance.  What we have done with this is aggregate the Archive and Purge extracts into a Long Term archive which is also queryable in the same manor.

 

I have some of this documented here:  http://kentweare.blogspot.com/2007/05/biztalk-retrieving-tracked-messages.html If you are interested in hearing more about this, let me know.

 

 

Scott also brings up a valid point about tracking message bodies in BAM. In order to do this you need to use the BAM API inside of your orchestration.  The BAM API is pretty straight forward.  Darren Jefford, a well known Author and Microsoft Consultant, is a big supporter of this approach and touches abit about this in his blog:

 

"BAM can store up to 1MB of data through the AddReference method on the eventstream, a great well-pefoming solution for message-bodies which enables the storage of message bodies alongside the data that describes how it was processed."

 

http://blogs.msdn.com/darrenj/archive/2007/04/30/bam-best-practice-starter-for-10.aspx

Answer 4

 

Kent,

 

Thank you for your reply, I think that is exactly what I was looking for!

 

Bret

Answer 5

And there is a simple solution without BAM.

Just send all messages you'd like to audit to the Direct port shapes. Then create the Ports with filters for those message types.

Pluses of this solution: easy to implement; no addition technology in use (BAM); more control of the message flow (not so in BAM).

Minus: hit performance (but there was no real tests to compare it with BAM solution).

 

As I know Microsoft does not have recommendations of using BAM for auditing. Let me know if I'm wrong.

Answer 6

Hi Leonid,

 

On at least a couple of occaisions I have heard MS employees endorse BAM for auditing.  We brought a Microsoft consultant to review our architecture and design with respect to auditing.  He suggested we use BAM for this purpose.  Note he did not endorse message body tracking inside of BAM.  The other place that I have seen it endorsed is at TechEd at one of Darren Jefford's sessions.  

 

Without a doubt building up the required BAM infrastructure is not a trivial task(especially in an enterprise scenario).  Keeping things simple is often the best solution, so I do think that you have brought up some valid points.

 

Regards,

 

Kent

 

 

Darren's Session info:

 

SOA306 - Building an Enterprise-Wide Instrumentation Solution Using the Microsoft BizTalk BAM Infrastructure
Monday 4th June: 1645 to 1800
Track(s): SOA and Web Services
Level: 300
Speaker(s): Darren Jefford

Business Activity Monitoring (BAM) is a powerful feature of BizTalk Server and is often marketed as allowing "the business" to understand what is happening within your BizTalk solution. BAM does this really well, but it can in fact be used in a variety of other ways which can deliver huge value to customers and address a number of issues they have with BizTalk based solutions and non-BizTalk based solutions.

In this session, we cover some of the fundamentals of BAM and detail how you can utilize BAM to collect a variety of information and produce a "tracking portal" which you can use to support your application, perform manual repair of messages, and generally observe your solution.

We also show how BAM is not just for BizTalk solutions and how it can be used to produce an enterprise-wide instrumentation solution that is highly scalable and flexible; we touch on the new Windows Workflow Foundation and Windows Communication Foundation (WCF) BAM Interceptor technology that enables data to be collected from Workflows and WCF services enabling a true end-to-end instrumentation solution. 

 

http://blogs.msdn.com/darrenj/archive/2007/06/03/teched-2007-sessions-and-book-signing.aspx

Answer 7

Hi Kent,

 

Thank you for the interesting discussion!

 

My understanding is BAM is not *officially* marked as a auditing tool by Microsoft because it is easy to create the BAM solution with "disappearing" messages. There are several posts in the Microsoft forums about it.

 

In my experience it was one case when the customers requirement was to create a self made audit system, because Microsoft did not explicitly endorse BAM as an audit system. The *audit* term has a special meaning in the finances.

Answer 8

Hi Leonid,

 

I totally agree with what you say about industry specific auditing requirements.  The Health industry would probably also fall in line with the financial industry in terms of strict auditing requirements.

 

The type of information and the retention periods for my project were not as demanding as some of the other scenarios(Finance/Health) that we have discussed and therefore it worked for us. 

 

You also bring up a good point about disappearing or missing data.  As you know,  BAM will only track what you tell it to track.  If you do not track info deemed required, you will not have access to that information as well.  Unless you call upon BAM in the pipelines, you may also run into some pitfalls as well.  For instance if a receive pipeline fails and you are only tracking at the Orchestration level, you will be missing audit info.

 

Kent

Answer 9

Hi,

 

Interesting discussion about if BAM is the right place for message auditing ...

 

I'd just like to add some info about were to find the tracked message in the DTADb. I've also used another method (using the BizTalk operations library) than the WMI script that Kent described earlier. The WMI approach is rather slow and didn't work for what we were trying to do.

 

Message auditing is a very common requirement and it be nice if there was a easier way of achieving this using BAM ...

Richard Hallgren

http://www.richardhallgren.com