ApplicationEndpoint terminates with TooManyActiveEndpoints

Question

Can we get more information regarding this arbitrary decision to terminate application endpoints?

Here's the source post to get some context

http://social.msdn.microsoft.com/Forums/en-US/ucmanagedsdk/thread/dfda1828-d86f-495b-ad2d-2537a3131132/

So, Ahmed, you mention this:

"The server has a liimt on the number of consecutive endpoints that it allows to be registered for one uri. I believe that the limit is eight.

When the ninth endpoint is registered, one of the other endpoints will be terminated."

Originally, the way I interpreted this is that one Application can only have 8 ApplicationEndpoints at any given time -- but this is clearly not the case.  Our application currently has 46 endpoints running, and it has been around that number for a year now.  We have numerous customers running similar numbers of application endpoints.  However, yesterday, we experienced this error for about 10 of our endpoints.  One of our customers also experienced this problem yesterday for a number of endpoints. 

Every time we have seen this issue, about 10 endpoints get terminated simultaneously with the same error.

Prior to yesterday's outages, the last time we saw this was in our performance lab about 6 months ago, where it happened once.  And clearly it has happened before I joined our Lync team, since my former co-worker was involved in the linked post. :)

So can you explain what "registered for one uri" means?  Do you mean that you can create 8 ApplicationEndpoint objects that point toward the registed uri via New-CSTrustedApplicationEndpoint in Powershell?  Our application only creates one ApplicationEndpoint object per registered CSTrustedApplicationEndpoint, so that doesn't make sense either.

I have a couple of other concerns.  If we simply restart an ApplicationEndpoint, will it then force another endpoint to terminate?  How do I recover existing calls to that endpoint when it gets terminated?  Can we get a notification?

Are there any powershell commands where we could raise this limit?

Thanks,

Jeramy

Answer 1

I think that Ahmed was confusing this with the limit on UserEndpoints.  By default, OCS/Lync limits the maximum number of registered user endpoints to 8.  You can change this if you like in the admin tools.

See this blog post for details: Configuring the Maximum Devices Per User

Is it possible that your endpoints are terminated for inactivity?

---

Oscar Newkerk

Oscarnew Consulting

Answer 2

I believe you're right that the limit of 8 doesn't apply to ApplicationEndpoints, otherwise we would have seen it much more frequently than we have.

However that doesn't explain why my ApplicationEndpoints are terminating with the TooManyActiveEndpoints reason code.  One would think it would involve too many active endpoints :)

That's a good guess about inactivity, but it's not the case :)  On our production server it so happens that some of the more active endpoints were terminated.  When it happened in our performance lab we were hitting them are hard as we could.

Answer 3

We investigated this extensively and found a few extra things out. Firstly are you cleanly shutting down your endpoint? If you use UseRegistration=true on your ApplicationEndpointSettings when establishing the endpoint and you don't shutdown cleany without removing the registration then a ghost is created that lives on the FE. This happens alot during devel as you are running then shutting down the app with an abort. While developing turn set the UseRegistration false and try and clean up your objects. Otherwise to clear this error you have to restart the FE services to kill the ghosts. We found this to be especially true when setting DefaultRoutingEndpoints. This problem apears to be related to presence registration not endpoint utilization.

Shannon

Answer 4

Thanks for the great answer... I'll definitely try to reproduce based on this post.

Answer 5

This just happened again at a customer site -- after two weeks of uptime.  Last time the app was shut down (two weeks ago) it was done cleanly.

This is the 4th time we've ever seen this issue, both in-house and on all customer sites.

Shannon, we tried killing our service a bunch of times, to no avail.

Once we register the endpoint (as an automaton), we just leave it.  This error just comes out of the blue.

Answer 6

Are you setting the UseRegistration=false? The registration in presence is why this happens (as far as we can tell ). Reseting the service does nothing( assuming your app is the service ) because the FrontEnd server is the one with the ghosts. You have to rest the FrontEnd service and it will reset( I know, I know, suicide on a production server ) We haven't had the issue since we did stopped the presence registration.

Answer 7

I don't believe we are setting it.  We're not doing any unique presence publishing, so I suppose it can't hurt to turn it off.

I'll let you know in six months if it reoccurs :)

Answer 8

Hey Jeramy-have you managed to catch this again?  I'd just like to add that I saw the same thing happen last night on an endpoint that was under active load.  The app only had 7 active endpoints at the time, and not all of them were killed.  It had been running for several hours uninterrupted, when the OnPortalEndpointStateChanged event fired with the TooManyActiveEndoints reason. 

Is there, perhaps, a global limit on the server that would need to be configured?  I'm going to dig into this a little more and see what I can find out. 

Answer 9

No, we haven't seen it since the last post.    I have to double-check if we ended up setting UseRegistration to false or not... I believe we tried it, and it caused some endpoint presence publish issues.