Azure Service Bus Messaging Factory Close-method times out

Question

Getting some stacktrace:

12:01:06.4752 +01:00|14|Info|Receiver|closing all message factories and receivers 
---- DEBUG ASSERTION FAILED ----
---- Assert Short Message ----
12:02:06.4927 +01:00|14|Error|Receiver|could not close messaging factory System.TimeoutException: The operation has timed out.

Server stack trace: 


Exception rethrown at [0]: 
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.OnEndClose(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.OnClose(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
   at Microsoft.ServiceBus.Messaging.Channels.SharedChannel`1.CloseOrAbortAsyncResult.<GetAsyncSteps>b__b(CloseOrAbortAsyncResult thisPtr, TimeSpan t)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.RunSynchronously()

Exception rethrown at [1]: 
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.RunSynchronously()
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.OnClose(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessagingFactory.CloseAsyncResult.<GetAsyncSteps>b__4c(CloseAsyncResult thisPtr, TimeSpan t)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)

Exception rethrown at [2]: 
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.RunSynchronously()
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessagingFactory.OnClose(TimeSpan timeout)
   at Microsoft.ServiceBus.Messaging.MessageClientEntity.Close(TimeSpan timeout)
   at Microsoft.ServiceBus.Messaging.MessageClientEntity.Close()
   at Receiver.closeColl() in Receiver.fs:line 92
---- Assert Long Message ----


    at TraceTarget.Write(LogEventInfo logEvent)  
    at Target.Write(AsyncLogEventInfo logEvent)  
    at Target.WriteAsyncLogEvent(AsyncLogEventInfo logEvent)  
    at LoggerImpl.WriteToTargetWithFilterChain(TargetWithFilterChain targetListHead, LogEventInfo logEvent, AsyncContinuation onException)  
    at LoggerImpl.Write(Type loggerType, TargetWithFilterChain targets, LogEventInfo logEvent, LogFactory factory)  
    at Logger.WriteToTargets(Type wrapperType, LogEventInfo logEvent)  
    at Logger.Log(Type wrapperType, LogEventInfo logEvent)  
    at NLogLog.Error(Object obj, Exception exception) NLogLog.cs(121)
    at Receiver.closeColl()  Receiver.fs(94)
    at Receiver.System-IDisposable-Dispose()  Receiver.fs(136)

Code:

/// Closes the receivers and message factories created
let closeColl () =
  logger.Info("closing all message factories and receivers")
  for (mf, recv) in mfAndRecvsColl do
    if not(mf.IsClosed) then
      try
        mf.Close()
      with
        | x -> logger.Error("could not close messaging factory", x)
    if not(recv.IsClosed) then
      try
        recv.Close()
      with
        | x -> logger.Error("could not close receiver", x)

I'm currently using a single message receiver per message factory and each message receiver/factory pair has maximum of 30 open async BeginReceive calls (in this particular case).

The obvious question is; why does this happen?

The second most obvious question is, why can't I either pass a cancellation token or a timeout?

Thirdly, I'm curious; what is the semantics of these messaging entities on the server, really? I can't close it - then what do I do? What is this state that the Close call is manipulating and why don't they implement IDisposable to handle the termination-parts of that state?

Answer 1

Hi,

Please make sure your network is available at the time you try to close the channel. For now, I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay.

Appreciate your patience.

Best Regards,

Ming Xu.

---

Please mark the replies as answers if they help or unmark if not.
If you have any feedback about my replies, please contact [e-mail].
Microsoft One Code Framework

Answer 2

I have network.

I think the problem is that the functionality is faulty when calling Close() twice on the same receiver; despite IsClosed equalling false the second time around. This is the same problem that exists with creating and deleting queues, forcing the consumer to recursively try create until either QueueExists returns true or MessagingEntityAlreadyExistsException is thrown.

Answer 3

Further, it seems to be happening even for different instances of MessagingFactory if one of them experienced an exception.

Answer 4

Bump

Answer 5

If the Close method does not throw an exception, it should mean the message factory is closed successfully. If IsClosed is still false at that time, it could be a bug. You can submit bugs on https://connect.microsoft.com/VisualStudio/feedback/. You can also try to use the async methods Begin/EndClose.

Answer 6

I hope that by writing in the official forum, bugs will be added to the relevant issue trackers...

Answer 7

Do you have any trace ID for your error?

Can you provider sample projec to reproduce this issue?

Thanks

Leo Lin

Answer 8

How can I produce a trace ID if it's a local error?

Yes, I can reproduce it in a test project. Here you go.

Answer 9

Bump

Answer 10

Do you check the MSDN document for Best Practices for
Leveraging Windows Azure Service Bus Brokered Messaging API

http://msdn.microsoft.com/en-us/library/windowsazure/hh545245(v=vs.103).aspx

 

Handling Transient Communication Errors
TimeoutException
 This exception indicates that the Service Bus messaging service infrastructure did not respond to the requested operation in the specified time which is controlled by the OperationTimeout setting. The requested operation may have still been completed; however, because of network or other infrastructure delays, the response may not have reached the client in a timely manner. Compensating this type of exceptions must be done with caution. If a message has been delivered to a queue but a response has timed out, resending the original message will cause duplication
 

Do you check any issues from firewall and does it connect with service bus correctly from your local environment?

To close a given messaging object, you must invoke its
Close() method using one of the following techniques: When a queue client is no
longer required, let's close it so that it doesn't consume a connection. Option
#1: Closing a specific messaging object instance. queueClient.Close(); Option
#2: Closing all messaging objects tracked by the messaging factory.
messagingFactory.Close(); It is also worth noting that in some rare cases, the
messaging objects may end up in a state that prevents them from being closed
gracefully. Should such a situation occur, the brokered messaging API will
ensure that appropriate actions will be taken, including aborting a connection
if it cannot be closed successfully. You do not need to perform a status check
to decide whether to call the Abort() or Close() methods. This is performed
internally by the API. Please note that the Close() method is not guaranteed to
complete without throwing an exception. Therefore, if you want to ensure that
closing a messaging object is always safe, an additional layer of protection in
the form of a try/catch construct is recommended.

Answer 11

Why do you guys keep asking me if I have internet or firewall problems? I keep telling you that this isn't the issue. Otherwise, how would I be able to create the connection in the first place???

I realize that _sometimes_ it's proper and well to time out; because the route to Azure might truly be unreachable, but obviously this isn't the case here! If that was the case; again, I wouldn't be able to send messages. As such, it's a bug and a problem of the API, and not my fault for not following 'best practices' - I've given you sample code, so you are free to verify this claim or otherwise tell me so I can fix it!

This is the problem: '1 passed, 0 failed, 0 skipped, took 139,14 seconds (NUnit 2.5.10).'

What's the point of having Close throw exceptions? I can't react in any reasonable way to those exceptions; right? So why throw exceptions? Aren't exceptions supposed to tell me that something is wrong and I need to take compensatory action?

Answer 12

Any help appreciated. Bump.

Answer 13

You will unfortunately seldom get a real answer on this forum.

I had the same issue regarding TimeoutException on Close()-method. Since no one provided the real answer then I choose to catch and ignore the exception. When the timeout occurs I really don't know what you can do about it.

Answer 14

The system is resilient against sockets going away, so you could theoretically just abandon the factory with Abort(). I've raised this with the dev team to see where we spend so much time for the graceful server-side session cleanup on Close(). I agree you shouldn't get a timeout here.

Clemens

Answer 15

This is a client-side bug that has been fixed and will ship with the Windows Azure SDK 1.7.