"The server was unable to process the request due to an internal error"

Question

I have four receivers set up doing long-polling on a Service Bus queue.  Four receivers because the queue may receive a lot of messages that need to be processed quickly and so far, four is enough.  The queue has been empty all weekend because the sending process is currently disabled, but the receivers have been polling anyway.  I receive intermittent "The server was unable to process the request due to an internal error" exceptions, and I'd like to fix this, if possible.  Today I received over 200 of these errors.  Here's data and a stack trace from a typical one:

Microsoft.ServiceBus.Messaging.MessagingException, Microsoft.ServiceBus, Version=1.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
The server was unable to process the request due to an internal error.
TrackingId:44b2ffdd-77b9-428c-b580-edfbc0dc74d7_18, Timestamp:4/29/2012 9:07:26 PM

Server stack trace:    at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.EndReceiveCommand(IAsyncResult result, IEnumerable`1& messages)   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result) Exception rethrown at [0]:    at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.OnEndTryReceive(IAsyncResult result, IEnumerable`1& messages)   at Microsoft.ServiceBus.Messaging.MessageReceiver.TryReceive(TrackingContext trackingContext, Int32 messageCount, TimeSpan timeout, IEnumerable`1& messages)   at Microsoft.ServiceBus.Messaging.MessageReceiver.Receive(TimeSpan serverWaitTime)   at
(my code) at
Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()   at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) at (my code)

Inner exception
System.ServiceModel.FaultException`1[[System.ServiceModel.ExceptionDetail, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The server was unable to process the request due to an internal error.
TrackingId:44b2ffdd-77b9-428c-b580-edfbc0dc74d7_18, Timestamp:4/29/2012 9:07:26 PM

Server stack trace:    at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result) Exception rethrown at [0]:    at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result)   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__13(RequestAsyncResult thisPtr, IAsyncResult r)   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result) Exception rethrown at [1]:    at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.EndReceiveCommand(IAsyncResult result, IEnumerable`1& messages)

My code:

Uri serviceUri = ServiceBusEnvironment.CreateServiceUri("sb", serviceNamespace, servicePath);
TokenProvider tokenProvider = TokenProvider.CreateSharedSecretTokenProvider(username, secret);
MessagingFactory messagingFactory = MessagingFactory.Create(serviceUri, tokenProvider);
queueClient = messagingFactory.CreateQueueClient(queueName, ReceiveMode.ReceiveAndDelete);
RetryManager retryManager = EnterpriseLibraryContainer.Current.GetInstance<RetryManager>();
Microsoft.Practices.TransientFaultHandling.RetryPolicy serviceBusRetryPolicy = retryManager.GetDefaultAzureServiceBusRetryPolicy();
serviceBusRetryPolicy.ExecuteAction(() => {
	using (BrokeredMessage message = queueClient.Receive(TimeSpan.FromHours(1))) {
		if (message != null) {
			//do stuff with the message
		}
	}
});

Is there something obvious that I'm missing, or does the service really encounter internal errors on a regular basis?

Answer 1

Another, with a similar stack trace and different error message:

50002: Provider Internal Error.TrackingId:3e4166e0-9e05-4f07-9581-bf14057122fd_3,TimeStamp:4/30/2012 1:36:29 PM

Answer 2

hello sdallamura,

you can follow this tutorial http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-queues/ to check if you missed something in the service bus queue.

just from the stack information here, it is not obvious to locate the root cause.

so we will analyze it more for some more suggestion.

how is it working now? i assumed fabric event may also caused this type of "internal error"

regards.

Jian

Answer 3

My code is similar to that in the tutorial.  I created my queues using the management portal, instead of code.  I've also integrated the Transient Fault Handling Application Block.  For the most part, everything works.  These errors are intermittent and seemingly random.  I catch and handle the exceptions and future messages are received normally.

I also suspect a "fabric event", but I'd like to confirm this to make sure there's no problem with my code.  I'm hoping that's what the TrackingId value is for.

Answer 4

Is there a specific location where I can submit my TrackingId for analysis or review?  If this happens during a period of high load, I'm going to get a lot of phone calls, and I'd like to be able to say with confidence that it's a temporary system error.

Answer 5

Hi,

AFAIK, You can get in touch with Microsoft Support for more details at: https://www.windowsazure.com/en-us/support/contact/ 

I hope it helps!!

---

If you found this post useful, Please "Mark as Answer" or "Vote as Helpful". Thanks! Vaibhav Gujral.

Answer 6

Scott -- if you do encounter this issue again during a period of high load, please let me know in this forum and I'll see if I can escalate the issue to the Service Bus team.

Thanks,
Seth Manheim
Microsoft

Answer 7

I'm actually encountering this issue right now.... for the past 90 minutes or so.