Will there be any logging, monitoring or tracing functionality in the final release

Question

Hi there,

I've been playing around with the Service Bus EAI & EDI Labs Release CTP and executed the EAI and EDI tutorials that are part of the documentation. Until now I experienced that it is very hard to do troubleshooting on the EAI and EDI components in Azure. I've been looking around in the forums, but I couldn't find any info on tracing, monitoring and logging in the cloud. More precisely I'm looking for some minimum functionality such as a trace or log file that I can use to troubleshoot my EAI or EDI project, preferably through a graphical user interface such as a web page. This would not also be useful for troubleshooting during development, but also during the future exploitation/operation of a solution to follow up eventual production issues. Something like Health and Activity tracking in BizTalk would already be a great help. Am I overlooking this functionality or will it be included in the final or futures releases?

Thx

Kristof

Answer 1

Hi Kristof,

As it was previously the case, I am confident that the actual CTP phase - which is focused on pure functionality - will be followed by richer features on tracing and monitoring level.

Until then, I troubleshooted my integration flows by breaking them into pieces and checking each of the steps individually, with the right message type and values for each intermediary step. In the same time, the integration input gate returns fault messages in case of handling failure. I succeeded in identifying most of the issues this way.

Hope it helped.

---

Marius

Answer 2

Thanks for your answer Marius.

As far as I'm concerned, tracing and monitoring is an essential piece of functionality for a Service Bus.

I'll continu to follow up the progress of this project and hope some tracing and monitoring will be included soon.

Ciao

Kristof

Answer 3

Kristof,

  I think most of the people working with the ServiceBus EAI CTP would say that more diagnostics would be helpful. This is definitely true on the EDI functionality.

In future releases this should improve.

Thanks,

---

If this answers your question, please use the "Answer" button to say so | Ben Cline

Answer 4

Hi Kristof,

The current CTP does not have monitoring features exposed as part of both EAI and EDI scenarios and you are not overlooking them .

Additionally I think there are two aspects to this as you too have subtly pointed out :

1. Debugging during development
2. Monitoring the system after deployment

We are presently working on addressing parts of the above and you will be seeing them as part of upcoming updates and releases ( read this as quite soon ) .

It will good to hear your thoughts on how you think these should be addressed and it will help validate our thoughts too . Along with this mention the issues you ran into during development .

Until then the approach mentioned by Marius will go well .

-Harish Kumar Agarwal

Answer 5

Hi Harish,

Thanks for your answer.

I have worked with several other EAI/ESB/EDI solutions and this is typically how these things are addressed:

Logging: a log file or database (SQL Azure?) in which you can write log entries with configurable log levels (TRACE, DEBUG, INFO, WARNING, ERROR, FATAL,...). A log entry should contain a datetime stamp, a textmessage, an ID of the component that generated the entry, stacktrace in case of error, ... Ideally these entries are cleaned up or archived automatically at a configurable time interval.

Auditing: would allow to group/correlate all log entries for a message instance, put an overall processing status (COMPLETED, FAILED, RUNNING) on the message instance, provide reprocessing capabilities,... This would also allow to cover non-repudation in EDI scenarios.

Monitoring: some statistics data would be nice about number of messages processed, response times, fault count, ...

Tracing: the ability to temporarily enable tracing of data flowing through the system, such as SOAP or AS2 messages including HTTP headers, which would allow you to capture certain messages causing problems and replay those messages in a dev or test environment.

I would expect these data sources to have a Web GUI through the Azure portal, together with a REST API that would allow you to integrate the information in another monitoring solution. Over time you could also include test functionality in the Web GUI, etc...

Ciao

Kristof

Answer 6

Hi Harish,

As a fundamental part of each system is its instrumentation (logging) accros the layers, activities, boundaries and tiers. In addition, also a capability to inject application checkpoints and custom trackings.

All logging messages should be logical centralized in the database and enabled for quering data based on the logical context accross all boundaries, layers, etc. For instance, if the client located on-premises is sending messages to the Azure SB Topic and the EAI/EDI bridge is a message mediator for invoking a 3r party service, we should see all activities from E2E in the logical context based on the configured severity, tracking, etc.  

The EAI/EDI is a part of the Service Bus and therefore it should be followed the same logging model built-in the AppFabric (both on-premises and cloud). The good example is WF and WCF models.

Basically, the logging system is creating a large knowledge base of the behavior of the business processes which can be analyzed in the post-processes such as reporting, simulation, animation, learning, tuning, watchdog, etc.

This knowledge base can be used for monitoring, performance analyzing, usage of the resources for billing purposes, etc. and also for more sophisticate analyzers for tech supports, maintenance, tuning process on the fly, etc. It is a good candidate for MS StreamInsight technology.

Basically, this knowledge base represents something similarly as the "aircraft black box".

What will be nice have in the EAI/EDI model is debugging feature for design time and runtime, where the message mediation can be under full control a remote debugger.

I think that this CTP version should be have already built-in and exposed same level of the logging mechanism which can be used it for development and usage of the EAI/EDI, specially when the bridges and mapper are driven by metadata and it is very hard to troubleshoot this process.

We are looking forward for the upcomming next release.

Thanks

Roman

---

Roman Kiss, MVP Connected System Developer

Answer 7

Hi All,

In the April 2012 refresh of Service Bus EAI and EDI Labs we actually have the tracking functionality available. You can configure tracking while configuring the bridge and then view the tracking data using REST interface. That was for the EAI bridges.

You can go through the following links for more information for tracking for EAI bridges that you configure in Visual Studio and then deploy on the Service Bus.

• How is tracking implemented in the April 2012 release: http://msdn.microsoft.com/en-us/library/windowsazure/hh949804.aspx
• How to configure tracking: http://msdn.microsoft.com/en-us/library/windowsazure/hh949820.aspx
• How to view tracking data: http://msdn.microsoft.com/en-us/library/windowsazure/hh949828.aspx

There is tracking support for the EDI portal as well. You can now configure tracking as part of an agreement's general properties (http://msdn.microsoft.com/en-us/library/windowsazure/hh689800.aspx) and then view the tracking data on a new page introduced on the portal, called TRACKING (http://msdn.microsoft.com/en-us/library/windowsazure/hh949805.aspx).

---

______ Nitin Mehrotra, Service Bus EAI/EDI CCxG, http://blogs.msdn.com/nitinme/