I've been playing around with the Service Bus EAI & EDI Labs Release CTP and executed the EAI and EDI tutorials that are part of the documentation. Until now I experienced that it is very hard to do troubleshooting on the EAI and EDI components in Azure.
I've been looking around in the forums, but I couldn't find any info on tracing, monitoring and logging in the cloud. More precisely I'm looking for some minimum functionality such as a trace or log file that I can use to troubleshoot my EAI or EDI project,
preferably through a graphical user interface such as a web page. This would not also be useful for troubleshooting during development, but also during the future exploitation/operation of a solution to follow up eventual production issues. Something like
Health and Activity tracking in BizTalk would already be a great help. Am I overlooking this functionality or will it be included in the final or futures releases?
As it was previously the case, I am confident that the actual CTP phase - which is focused on pure functionality - will be followed by richer features on tracing and monitoring level.
Until then, I troubleshooted my integration flows by breaking them into pieces and checking each of the steps individually, with the right message type and values for each intermediary step. In the same time, the integration input gate returns fault
messages in case of handling failure. I succeeded in identifying most of the issues this way.
I have worked with several other EAI/ESB/EDI solutions and this is typically how these things are addressed:
Logging: a log file or database (SQL Azure?) in which you can write log entries with configurable log levels (TRACE, DEBUG, INFO, WARNING, ERROR, FATAL,...). A log entry should contain a datetime stamp, a textmessage, an ID of the component that generated
the entry, stacktrace in case of error, ... Ideally these entries are cleaned up or archived automatically at a configurable time interval.
Auditing: would allow to group/correlate all log entries for a message instance, put an overall processing status (COMPLETED, FAILED, RUNNING) on the message instance, provide reprocessing capabilities,... This would also allow to
cover non-repudation in EDI scenarios.
Monitoring: some statistics data would be nice about number of messages processed, response times, fault count, ...
Tracing: the ability to temporarily enable tracing of data flowing through the system, such as SOAP or AS2 messages including HTTP headers, which would allow you to capture certain messages causing problems and replay those messages in a dev
or test environment.
I would expect these data sources to have a Web GUI through the Azure portal, together with a REST API that would allow you to integrate the information in another monitoring solution. Over time you could also include test functionality in the Web GUI,
As a fundamental part of each system is its instrumentation (logging) accros the layers, activities, boundaries and tiers. In addition, also a capability to inject application checkpoints and custom trackings.
All logging messages should be logical centralized in the database and enabled for quering data based on the logical context accross all boundaries, layers, etc. For instance, if the client located on-premises is sending messages to the Azure SB Topic and
the EAI/EDI bridge is a message mediator for invoking a 3r party service, we should see all activities from E2E in the logical context based on the configured severity, tracking, etc.
The EAI/EDI is a part of the Service Bus and therefore it should be followed the same logging model built-in the AppFabric (both on-premises and cloud). The good example is WF and WCF models.
Basically, the logging system is creating a large knowledge base of the behavior of the business processes which can be analyzed in the post-processes such as reporting, simulation, animation, learning, tuning, watchdog, etc.
This knowledge base can be used for monitoring, performance analyzing, usage of the resources for billing purposes, etc. and also for more sophisticate analyzers for tech supports, maintenance, tuning process on the fly, etc. It is a good candidate for MS
Basically, this knowledge base represents something similarly as the "aircraft black box".
What will be nice have in the EAI/EDI model is debugging feature for design time and runtime, where the message mediation can be under full control a remote debugger.
I think that this CTP version should be have already built-in and exposed same level of the logging mechanism which can be used it for development and usage of the EAI/EDI, specially when the bridges and mapper are driven by metadata and it is very hard
to troubleshoot this process.
We are looking forward for the upcomming next release.
In the April 2012 refresh of Service Bus EAI and EDI Labs we actually have the tracking functionality available. You can configure tracking while configuring the bridge and then view the tracking data using REST interface. That was for the EAI bridges.
You can go through the following links for more information for tracking for EAI bridges that you configure in Visual Studio and then deploy on the Service Bus.