Unofficial ESB Toolkit Survey

Question

Hello all,

I have some survey-style questions for anyone that feels like answering them. This is to satisfy my personal curiosity, and is not an official survey in anyway, nor will it likely have any effect on anything (though it might affect how I talk about and/or explain ESB). Hopefully we will be able to learn from each other though, in seeing how each of us is using (or not using) the Toolkit.

1.) What types of scenarios have you found that the ESB Toolkit (not just the exception handling stuff) is helpful? In which types of scenarios has using it increased productivity? Be as specific as you can.

2.) What types of scenarios have you found that just using regular BizTalk stuff (pipelines / orchestrations) without ESB has been more helpful? Be as specific as you can.

3.) What types of scenarios have you tried using ESB and then reverted to using regular BizTalk stuff? Be as specific as you can.

4.) What would you say is the biggest pain point in the Toolkit? How do you deal with it?

5.) What pain point (for you personally) does the Tookit address, if any?

---

BizTalk ESB Toolkit 2.0 Training: http://quicklearn.com/redir/?r=esbtraining

Answer 1

Nick,

It's sad that no one has replied. I wish I had some guidence for you but I'm in the same boat... I'm still struggling to find a real-life scenario where the ESB toolkit give me something I couldn't otherwise do in BizTalk.

Answer 2

Hi,

I would like to see ESB Dispatcher Disassemble pipeline component with the receive pipelines available with the Microsoft.Practices.ESB application. Why should I need to create a custom pipeline for multiple routing? If it is there readily available, it is better.

---

Please mark as answer if this helps you. Thanks and warm regards Ambar Ray EAI Architect - Microsoft Technologies

Answer 3

@Rob
It's not so much things that ESB can do that BizTalk can't (because ESB is an app built on top of BizTalk, and thus you could technically do everything in BizTalk on your own anyway), but more about which problems can be solved with greater ease by using ESB.

There are some problems that ESB can solve more easily than building something from scratch in BizTalk (e.g., things that require extensive runtime resolution), and there are some things which lend themselves better to plain old BizTalk orchestrations (e.g., things that rely on correlating multiple messages).

I'm really interested though to see how it's actually being used (rather than what it seems like it would be good for).

---

BizTalk ESB Toolkit 2.0 Training: http://quicklearn.com/redir/?r=esbtraining

Answer 4

We had a BizTalk User group meeting in the NL and we were lucky enough to have the inventor of the ESB explaining some stuff to us.
He was very clear and stated that the whole concept of the ESB evolved about Runtime Resolution....
Later on in the session we were able to ask questions....

My question was :

What is the advantage of having Runtime Resolution ?
And he was not able to give a clear answer to that.

In my opinion you should be able to explain and convince me within 5 minutes (if they are real advantages)
hoewever Brian Loesgen failed to do just that. He didn't have any real advantages and certeinly was not able to make me a believer.

I really cannot see ANY added value in the complete ESB toolkit (beside exception management) in real life integration scenario's
If there is anybody out there that can convince me please do so cause I would love to use the ESB Toolkit.

---

Well0549

Answer 5

Well0549, see my recent post about problems on the ESB exception management and you wont be saying the ESB exception management is worth having!
http://social.msdn.microsoft.com/Forums/en-US/biztalkesb/thread/6e1eda40-f1e0-4a66-9f2a-c28a2cfef0b8

That said, if all you need to do is simple message routing & transformation (a service bus!), then using the ESB toolkit is a very convenient tool - no orchestrations and no need to be fiddling with complex filters on send ports.

I have found the dynamic routing capabilities of the BRE resolver useful in handling differences between environments - just change the routing rule in each environment, so you can deploy the same MSI to each environment with the same bindings but messages get routed differently: could be a file drop in development but send to a WCF service in production.

Answer 6

Well0549, see my recent post about problems on the ESB exception management and you wont be saying the ESB exception management is worth having!

That said, if all you need to do is simple message routing & transformation (a service bus!), then using the ESB toolkit is a very convenient tool - no orchestrations and no need to be fiddling with complex filters on send ports.

I have found the dynamic routing capabilities of the BRE resolver useful in handling differences between environments - just change the routing rule in each environment, so you can deploy the same MSI to each environment with the same bindings but messages get routed differently: could be a file drop in development but send to a WCF service in production.

Where is this thread about Exception portal.....

---

Well0549

Answer 7

Found it here...

http://social.msdn.microsoft.com/Forums/en-US/biztalkesb/thread/6e1eda40-f1e0-4a66-9f2a-c28a2cfef0b8

and even in your own opinion this would be pretty easy to solve without using the ESB toolkit but since you are using it .....you have a problem

---

Well0549

Answer 8

Hi Nick,

I have been using the ESB Toolkit Itineraries for a project now, and I gained some nice insight in it. Have been creating a custom messaging component for message validation, another one for BAM (using brian birdoes' example as a basis). I also used the message broker component. I used the BRE and Static resolvers and decided not to use UDDI3.0 since it would add complexity for a small solution. The solution will go into production soon.

All in all, it became a sort of love / stress / hate relationship, similar to what I have with BizTalk in general but with different edges.

In general, I like to be able to replace much of the functionality that in BizTalk only can be done in orchestrations, and use configuration. Mostly because it's easier to deploy and more flexible then common BizTalk. For now, I love what it does (helps me use less orchestrations and be more flexible) but missing a more robust approach and a lot of functionality. In the end, I would like to have a BizTalk-like tool that is based on Workflow Foundation (XAML) and a good set of extensible core services. Ideally, this would be BizTalk next generation.

Prepaire for a long list...


I LOVE what the toolkit does / enables me to do:
- itinerary change and deployment is so much easier then building and deploying an orchestration
- Even with static bindings in an itinerary it's easy to do a text replace when moving between environment. I do the same with the business rules.
- Except for the maps and schema's the biztalk solution is now totally configuration driven
- The ability to chain a string of service calls instead of just request / response compared to just messaging
- Itinerary gives an overview of the message flow. message routing does not, making it hard to know what goes where.
- using the 'off-ramp' helps to get rid of solutions where you need multiple send ports or an orchestration (like multiple protocols / output locations / output message types)
- mapping inside the pipeline instead of on the port itself is great. Besides the choice of which map to use can be done using more then just the message
- I find it easier to create a messaging component compared to creating a pipeline components.
- Feels very service oriented. I can use UDDI (or some other service registry) to set service locations and policies instead of only putting them in BizTalk. This way, I can also expose the service endpoint directly.
- Support through this forum and some blogs is very helpful.

What I LIKE LESS in the ESB Toolkit
- deployment of the ESB Toolkit has improved, but is still non-trivial. Especially with 64bit you need to find out that some components (webservices and install scripts) need to run in 32bit mode in order to run.
- Itinerary (BAM) Tracking. If it would have a message ID in it, I could map it to a message for tracking. Another option would be the dump the message in the bam as well then itinerary tracking could replace BizTalk message tracking.
- Itinerary designer is an improvement but still rudimentary. Compared to orchestrations and Workflow Foundation it misses a lot of functionality. 
- I find it strange that in the itinerary designer I need to select the on- or off-ramp in a messaging service, but also need to draw a line. Just drawing a line should be enough.
- in the itinerary designer, the default should be 'strict'.
- not really clear what the functionality of the off-ramp extender really is. I think it could better be removed.
- The pipeline is streaming, but for the broker component to work (in my real life scenario) you need message context properties. I needed to create a message component that would pull the xmldocument through the XMLReceive pipeline component. In the end, it became the message validation component.
- Learning curve is (too) steep. You need BizTalk basics but also good knowledge of BRE, BAM, UDDI, pipelines and you need to learn the itinerary and resolver syntax.
- when an error in a component occurs, exception message is not very helpful
- ESB Portal is not really suitable for deployment at a customer site. It's too generic. But I think the same about the BAM portal.
- Itineraries run in the pipelines. I think this enables some functionality, but also makes it less flexible.

Miss (based on what I think the ESB Toolkit was meant to do):
- Maps and schema's are still a compiled resource in BizTalk. I guess you could just use the XSLT and XSD directly if you would store these in the DB. No more need for dll deployment
- if I use a message broker component I get a fork, but there is no 'join'. This makes lots of duplicate code if you want more complex processing
- I can't start a new itinerary for a message unless I am in a receive port. At least inside an orchestration this should be possible.
- debatching inside the itineraries (I do single request, get multiple responses, want to debatch and then process one by one).
- a way to map both the original request and the response of a service together to create the next service request.
- a scatter-gather without orchestrations.
- Itineraries are not really suited for big files. Just using dispatchers is probably enough for those things.
- 'Resubmit and resume' error handling (retry from last point in the itinerary)
- Low latency messaging (bypass message box persistance)
- the source code!

Answer 9

Spot on PIMW.

There is lots that could be added to the missing/should be done better. One I feel strongly about is the very weak implementation of Orchestration services:
- you have to go through a lot of hoops and code just to output a message to the next itinerary stage
- the orchestration ports should be exposed in the itinerary designer rather like the broker service. That way you can have multiple messages coming out from the orchestration, all partaking in the itinerary. Currently it is very difficult for the intermediate messages within an orchestration to make use of ESB itineraries.
- fault handling is poor: you can't put all your fault handling in a common sub-orchestration since the ESB.ExceptionHandling.ExceptionMgmt.CreateFaultMessage() has to be within the failing scope shape.
- you have to modify the esb.config file to specify the guid & name of the orchestration before you can use it in the itinerary designer
- etc

When you start using BRE for routing it seems to make sense. But when you try & configure a WCF service you'll soon be wishing for the standard BizTalk port designer wizard - its a nightmare trying to get the settings right within a business rule.

If you think about what the basic functionality of an ESB is, it is easy to envisage a simple GUI that would allow you select the message (via a variety of rules), assign an itinerary, perform the mapping, route to 1 or more destinations and handle errors. It should be a single, self-contained GUI, not a mash up of itineraries, business rules & port configurations like it currently is. Imagine you could just drag a connector from one service to another, specify the message type and the mappings at either end. Thats how simple it could be. BizTalk & the ESB toolkit turn the simple idea of an ESB into a complex development process.

Answer 10

Agreed. The CreateFaultMessage I ran into a few weeks ago. Also I'm right now adding an orchestration service in the itinerary and I will probably run into the others as well. At least now I know the solution :-)

I have to explain the point about itinearies run in the pipelines, since obviously you can do your transform and routing in the corresponding orchestration services and add a lot of traditional orchestration stuff. It's just that I prefer to not go through an orchestration since it adds another persistance point. Now that I have to intoduce orchestrations it adds another technology to the solution again. More complexity.

I think the ESB Toolkit 2.0 is is a good replacement for plain message routing, offering validation and transformation and smarter routing. An occasional orchestration or messagebroker component is ok, but if you need more functionality to solve your problem, I would stick to traditional orchestration based BizTalking and leave the ESB Toolkit alone.

Pim

Answer 11

I like all the dialog that we're having here, and I agree with most of what has been said. I'm still not seeing what I ws asking for and what I think Nick's original question was.

What does a real-world scenario that merits the use of the ESB Toolkit look like? With specifics. Pim said that it's a good replacement for plain message routing, but what does that scenario look like?

Answer 12

I can give an example of question 2 where a ESB stack would add nothing ....

2.) What types of scenarios have you found that just using regular BizTalk stuff (pipelines / orchestrations) without ESB has been more helpful? Be as specific as you can.

For a large energy company in the nedtherlands we had to interface with SAP. The people from SAP exposed a RFC and we could query that RFC to get customerinformation from the SAP system.

Now there was something strange with that RFC. We could get several responses back from SAP. ( We used the iWay adapter that is now incorperated into BizTalk).

We could get the following responses :

1. We could get a valid response with customer infromation...
2. We could get back some kind of error
3. We could get a valid response with no data at all. ( no error situation just no data due to some error in SAP )

We had an orchestration exposed as a callable service (Callable orchestration) with an input and an output message (customer according to CDM) so it was very easy to query the Customer service. All the errorhandeling could be done inside the callable orchestration. (even the very specific error where we got no information back. we needed some xpath queries to determine if a message was faulty)

We also deployed a property schema for the entire solution. One of the properties was 'HasError', so we could look at a message and see ( without looking at the content of the message) if an error had occurred downstream.

The callable orchestration could set this 'HasError' property on a message so you would call the orchestration ( doing all the mapping and chitchat with the outside world) and wait for a response. After the response was given back, you could easily look at the context of the message and see if the message had errors. ( In a decide shape do stuff like : messageCustomerResponse(HasError))  if there was an error you could go to some error handeling. Depending on the caller of the business process the flow could change.

If the caller was a webservice, you would want to give the fault response back to the webservice. If the caller was some lengthy business process you could save some state info to a message and send the message to a fault portal (where it could be resubmitted.)

Important in this scenario is that the Behaviour of the callable orchestration was always the same and the caller would implement it's own error handeling or give the response back to the caller ( the error of the called orchestration would be encapsulated in the response).

I really don't know if such a scenario is easy to implement with the ESB toolkit.

I know it is very easy to implement this in a standard BizTalk Environment.

Also developement is very easy if you want to call the CustomerService, you simply add a reference in the BT Project, add a call orchestration shape and you get the request and response message format for free.

---

Well0549