Config Services and ESB

Question

Hi Greg!

 

It's been a long time since we discussed on the road-ahead for Config-Svcs. I'm sure you have some plans already on this. Anyway, I was just thinking - now that you have given us a platform to create and host SERVICES that integrate within or across n/w's over standard protocols and be able to participate in LOAD-BALANCE and FAULT-TOLERANCE, How about the possiblitiy to extend this CORE-PLATFORM in order to allow registering, discovering, publishing & scubscribing, mapping and management of services in a loosely-coupled manner.

 

I guess you got what i'm saying - basically How about extending Config-svcs to support light-weight Enterprise Service Bus (ESB) features. I know it's outside the scope of your project probably, but what is your take ? or do you see any such open-source products that can be aligned with Config-svcs to achieve this goal?

 

Kindly share your views.

 

Regards

SKBG

 

Answer 1

Hi SKBG,

 

I do plan on doing a rev of Config Services to take advantage of "Dublin" and CLR4, and plan to start this (both are in pre-release and not yet generally available to public) soon.  I do not plan on building-in ESB features directly, but would look for Config-service hosted services to be able to integrate with existing technologies, such as MSE, BizTalk (which includes ESB guidance), and in future "Dublin" that incorprates routing/loosely coupled capabilities.  But I do not want to try to re-invent a whole ESB platform here; rather leveredge existing and future MS platform technologies...

 

 

-Greg

Answer 2

Thanks Greg for putting on right track, with somany things happening around, it's good have the right focus.

Again, goodluck for your initiatives.

 

Regards

SKBG

 

Answer 3

re: registering, discovering, publishing & scubscribing, mapping and management of services in a loosely-coupled manner.

 

Some of these features are available /or are available soon.

 

registering , mapping and management are partially handled in MSE.

 

 

Auto discovery will be in 4.0 via WS-Discovery. Come to think of it the config-svc layer for MSE which we discussed in another thread could use WS-discovery to find new services.

 

Publish/subscribe implies push which is a much bigger and complex issue ( such as firewalls on clients , identifying clients etc) . However Roman Kiss has a full implementaton on codeproject for WS-Eventing ( which wont be in 4.0) , WS-Eventing allows you to move to an EDA type architecture as well which can have significant business advantages ( over polling) .

 

Regards,

 

Ben

Answer 4

cool.  I have to dig into the discovery stuff.  I have concerns that discovery might be great for discovering (within intranet) what service nodes are currently *online*; but it may be terrible for immediately determining which nodes that 2 seconds ago were online are now offline and should not be used anymore.  Its a broadcast thing; and having thought about failover quite a bit; there needs to be some marriage of routing to a front end service gateway; with gateway having more immediate/intimate knowledge of what back-end load balanced nodes are online 'right now'.  The mechanism Config Service uses is an intelligent client (that can work both against Java nodes and .NET nodes) that performs this check before dolling out channel to business logic code.  Of course this does not eliminate *all* errors on a node hard crash (currently in-process requests where message has not been received by client yet); but does make failover work within a couple of seconds with no user/admin intervention and no 'polling' until a node that is expected to be online is found to be offline (and then polling just every 60 seconds just of these offline nodes).  How would one 'poll' such a discovery mechanism without real perf costs and potential brittleness?  You cannot keep making requests to an offline node for more than a few seconds, given the network response times involved (anything more than .1 seconds is an eternity in my mind, if over and over again).

 

Also, wrt to a central 'router' based approach; I certainly believe but am always open to being shown I am wrong; that its the client and only the client that can ultimately best determine what it can talk to and what it cannot, at that precise moment.  Load balancing strictly on the server implies a central point of entry; and if that goes down; everything goes down.  So I think a good thing is that Config Service can be used, especially in intranet scenarios, with point to point and near-immediate failover (including when someone gracefully closes a host, clients immediately know not to direct to that host anymore).  While this is just one scenario, I think describes a vast majority of load balancing scenarios between a Web App client (such as ASP.NET web site) to backend remoted business logic (and one that works with incredible perf advantage of WCF TCP/Binary encoding vs. full text-XML encoding over HTTP transport many load balancers expect--such as Windows NLB).  Ala Java RMI and what Java app servers attempt to accomplish (some better than others).

 

But, clearly I see that an option Config Service needs (marriage between Config Service and MSE) is the functionality of MSE; that can combine loosely coupled message routing with fast, failover ready/point to point between the front-end router and the backend service hosts with the business logic.  So this is what I want to plow into starting as soon as I can.

 

-Greg

 

Answer 5

One other thing that is wrapped up in this, and to some extent covered by MSE, is a strange kind of decoupling of client from service contract.  I guess that is the holy grail---kinda like auto negotiation that TCP NICS accmplish, for lack of a better analogy.  I do not like the idea of potential clients having to explicitly program to a specific contract/generate a specific proxy.  But I am smart enough to know the advantages (perf, compile-time safety, to name just two) and not smart enough to know how to get around easily (with advantages of auto-contract versioning, implied dynamic naure of a system, and corresponding dev and business benefits).  But I do know there are plenty of *really* smart people here at MS that think about this stuff with brains much larger than mine, even as I work to do some stuff I think we (MS and industry at large) have not done yet that are practical, achieveable by mortals (such as myself), and have easy UI/dev/and configuration capabilities.

 

-Greg

 

Answer 6

You're right - for a successful SOA, it's critical that clients can expect the brokers to know more and do more for them, than what the actual services otherwise would. So brokers being alble to monitor active service-nodes and load-balancing, is just as important. And, there will be perf trade-off's, should be within acceptable range.

Two key things for success:

- Clients being able to subscribe to events at broker level

- and, Broker taking care of load-balancing the virtual-service

 

Any similar systems that we can do a comparitive analysis with (combination of Virtualization engine and Config-Svcs)?

 

And these might be a few more important issues to cinsider?  http://www.codeplex.com/servicesengine/Thread/View.aspx?ThreadId=43484

 

Regards

- SKBG

Answer 7

You always keep me thinking!

 

I did goto cross-post;  I hope to dig into MSE in next couple of weeks....but as I pointed out on another thread am busy doing a ton of benchmarks that are targeted for MSDN in near-term; which I think will be interesting to people.  As you know performance is a passion of mine; and hence some of the reasons I do things the way I do....

 

BUT.....back to your question...I would be remiss if I did not point you to:

 

http://www.amberpoint.com/

 

-Greg

 

Answer 8

I'm, glad you say that. I'm in a position to choose the right platforms for a up-coming SOA initiative for a large enterprise, and so i've not jumped into any one big product as such, while open-source initiative from Microsoft is keeping me optimistic. But these initiatives are far from being complete (with most of them being in CTP) except Config-Svcs

There's no doubt in my mind on where to position config-svcs in such SOA architecture and i have faith in your effort in that direction.

 

But, a little more, and we stand to benifit more from these products in a Real-World SOA. Cos as you rightly pointed (amberpoint), we as architects have a huge respoinsibility to make two ends (drastically different) meet - technology & business. Amberpoint is a very nice reference to such "governance" needs that need to be met for an enterprise to truely benifit from adaptig SOA. But i've often felt such enterprises have had little commitment from their own end (often rushing into deadlines and targets), that's a different story. I've always chosen to be out of such failure-driven projects.

 

Bu things are hot at MS at the moment, it has two edges though - early adoption could be a risk, but we cannot wait too long as well. So i'm putting my money on few good MS initiatives like Config-Svcs, MSE, WCF-LOB Adapters, ESB Guidance and Geneva, and leading towards BizTalk. And if you agree, that's the best approach for a SOA, start with basic service-exposing, and that's where Config-svcs really comes in. But then soon as we start thniking about meeting some of the governance standards, then have to start looking outside config-svcs, like MSE for a virtualization platform (with decent support for loose-coupling, policy & SLA implimentation) but lacking some key BROKER features such as load-balancing virtual-svcs, and subscription to business-events for clients etc. If some of these issues can be covered by config-svcs and MSE, it will be a great combination towards the BizTalk roadmap. Amberpoint interestingly have worked with Vital which is one of our own customers - it's interesting they talk about a top-down view when it comes to SOA, they probably mean from enterprise view point, which is different from how we architects see i.e. middle-out-approach. So you see the GAP, and this gap never gets spoken out during SOA implementations, that eventually are cause of failed initiatives.

 

Regards

-SKBG

 

Answer 9

..and I can assure you, a marriage between MSE and Config-Svcs will give Config-vcs a more solid foot-hold in this SOA solutions framework of Ms.

-SKBG

 

Answer 10

...justchecked out amberpoint express, and it's just what MSE is lacking at the moment - monitoring health of services.

Thanks for that pointer.

- SKBG

 

Answer 11

No problem...I think a cool enterprise product that integrates very well with WCF and .NET stack.

 

-Greg

 

Answer 12

Hi Greg

Back to this topic after a long while. I'd like to know from your side if any interesting progress been made in Config-Svcs, or planned for next release. I saw your thred on the next planned release and features you plan, but still a long way from release you say.

I'm a bit stuck when it comes to rolling out low-cost ESB, meaning, if we're dealing with a handful of services, I still prefer to do point-to-point using config-svcs, and not worry too much about abstracting nodes, contracts, governance etc which MSE does well.
Config-svcs with config-web is simply great. But when i'm faced with scenarios, that are neither large-scale nor small, then it's hard to decide the right tools for a medium-scale ESB. Of course, the services will still be planned with config-svcs ground-up at a min, i still need the load-balancing etc it brings. MSE - may well be considered for what it does best, virtualization and some governance. But when it come to ESB, looks like wedon't have a light-weight way of achieving it.

I'm looking at following ESB features - content-publishing between producers and consumers using content-based routing (probably), or publish-subscribe mechanism, Async transport protocols, loose-coupling, message-transformation, security across enterprise, and a monitoring mechanism which the ESB can use to determine endpoint state. The last one is probably solvable in different ways depending upon how we plan the routing itself - request/response, publish/subscribe, broadcast etc.

BizTalk + ESB Toolkit seems too big for this. Proably not in later stages, but to start with atleast. What is your suggestion?

Regards
- Satish

Answer 13

Hi Satish, good to hear from you!

I am working on Config Services vnext, on the .NET Framework 4.0/Visual Studio 10 platform.  I hope to have this out shortly after VS 2010 and CLR 4 release, sometime Q1 next year (in about 3-5 months).

However, from a lightweight ESB standpoint, I am not trying to implement this, since there is work within Microsoft itself to bring many of the features you want to light within the platform itself.  I can't go into any real detail; and until then your best bet is still either BizTalk + ESB toolkit; or MSE.  But, the key thing is there is much core work being done within the platform team itself to light up these features, so makes little sense for me to try anything within config services; although I am doing a 4.0 version as mentioned that will add some cool new features and be implemented on .NET Framework 4.0.

-Greg

---

Greg Leake, Microsoft

Answer 14

Thanks greg.

You're right, like you, now i'm waiting for .Net 4.0 :) my guess is it will be worth the wait, as it surely
closes the gap we're talking here. Of all the features .Net 4.0 is expected to bring into WCF, the most
attractive to me is the Content-based routing along with Correlation in WF, ... atleast for now.
I think this is the light-weightedness I was talking. WCF WorkFlow Services
that support long-running conversations + content-based routing, should allow some kind of (ofcourse not all ESB features) service-bus to be implemented. I think the real question hidden in the above discussions was actually
related to scenarios where you don't control the services or un-known services scenario and how config-svcs would address this case. Cos it's not always that you control all services in real-life scenarios. How do you address this issue in the next release? or will it mean we still live with point-to-point integration with tight-coupling to the service contracts. You already kind of answered this :) but still curious in case you have something else in store ;)

Regards
Satish