Azure Service Bus Queue: Messages are not being sent

Question

I'm sending messages to service bus queue from a worker role. I noticed that randomly some messages are lost.

When I debug, I set a breakpoint after the Send method and log into my Azure Panel to check if the message queue increased. What I found out is that strangely, messages are not added to the queue some times. But it is not randomly. The pattern is: One message is added correctly and the next is lost, then again next is ok, and the next is lost.

I've implemented a retry pattern like the following but apparently not all the messages are sent correcty.

Sometines works fine for hours. When I redeploy something that has nothing to do with the sevice bus, I fall into this situation.

My code is:

var baseAddress = RoleEnvironment.GetConfigurationSettingValue("namespaceAddress");
        var issuerName = RoleEnvironment.GetConfigurationSettingValue("issuerName");
        var issuerKey = RoleEnvironment.GetConfigurationSettingValue("issuerKey");
        var retryStrategy = new FixedInterval(5, TimeSpan.FromSeconds(2));
        var retryPolicy = new RetryPolicy<ServiceBusTransientErrorDetectionStrategy>(retryStrategy);
        Uri namespaceAddress = ServiceBusEnvironment.CreateServiceUri("sb", baseAddress, string.Empty);

        this.namespaceManager = new NamespaceManager(namespaceAddress, TokenProvider.CreateSharedSecretTokenProvider(issuerName, issuerKey));
        this.messagingFactory = MessagingFactory.Create(namespaceAddress, TokenProvider.CreateSharedSecretTokenProvider(issuerName, issuerKey));
        //namespaceManager.GetQueue("chatmessage");
        QueueClient client = messagingFactory.CreateQueueClient("chatmessage");
        APPService.Model.MessageReceived messageReceived = new Model.MessageReceived();
        messageReceived.From= e.From;
        messageReceived.Op = e.Operator;
        messageReceived.Message = e.Body;
        BrokeredMessage msg = null;

        // Use a retry policy to execute the Send action in an asynchronous and reliable fashion.
        retryPolicy.ExecuteAction
        (
            (cb) =>
            {
                // A new BrokeredMessage instance must be created each time we send it. Reusing the original BrokeredMessage instance may not 
                // work as the state of its BodyStream cannot be guaranteed to be readable from the beginning.
                msg = new BrokeredMessage(messageReceived);

                // Send the event asynchronously.
                client.BeginSend(msg, cb, null);
            },
            (ar) =>
            {
                try
                {
                    // Complete the asynchronous operation. This may throw an exception that will be handled internally by the retry policy.
                    client.EndSend(ar);
                }
                finally
                {
                    // Ensure that any resources allocated by a BrokeredMessage instance are released.
                    if (msg != null)
                    {
                        msg.Dispose();
                        msg = null;
                    }
                }
            },
            ()=>{},
            (ex) =>
            {
                // Always dispose the BrokeredMessage instance even if the send operation has completed unsuccessfully.
                if (msg != null)
                {
                    msg.Dispose();
                    msg = null;
                }

                // Always log exceptions.
                Trace.TraceError(ex.Message);
            }
                    );



    }

What do yo think could be wrong? Thanks.

Answer 1

This may be because you are disposing the msg object which is not really in your scope, before it gets sent.

Could explain the alternate pattern you are experiencing.

Answer 2

Thanks Lucifure, I'll change that and see what happens.

Answer 3

Have you solved the problem? If it is solved, It would be better if you can provide the solution? If it is not solved, please post the details of your latest progress, we will look into it.

Answer 4

No Jiang, I didn't solve it.

Whenever I see that happens, I restart the server and start working well.

Answer 5

Your code looks fine, you should be disposing the message as you are. Well I assume that's not exactly your code - you should not be running all of this

var baseAddress = RoleEnvironment.GetConfigurationSettingValue("namespaceAddress");
        var issuerName = RoleEnvironment.GetConfigurationSettingValue("issuerName");
        var issuerKey = RoleEnvironment.GetConfigurationSettingValue("issuerKey");
        var retryStrategy = new FixedInterval(5, TimeSpan.FromSeconds(2));
        var retryPolicy = new RetryPolicy<ServiceBusTransientErrorDetectionStrategy>(retryStrategy);
        Uri namespaceAddress = ServiceBusEnvironment.CreateServiceUri("sb", baseAddress, string.Empty);

        this.namespaceManager = new NamespaceManager(namespaceAddress, TokenProvider.CreateSharedSecretTokenProvider(issuerName, issuerKey));
        this.messagingFactory = MessagingFactory.Create(namespaceAddress, TokenProvider.CreateSharedSecretTokenProvider(issuerName, issuerKey));
        //namespaceManager.GetQueue("chatmessage");
        QueueClient client = messagingFactory.CreateQueueClient("chatmessage");
        APPService.Model.MessageReceived messageReceived = new Model.MessageReceived();
        messageReceived.From= e.From;
        messageReceived.Op = e.Operator;
        messageReceived.Message = e.Body;
        BrokeredMessage msg = null;

for every message you send. That all can be reused (again assuming the message sending frequency is under a minute). Just make sure you check for 

MessageSender.IsClosed

to make sure the connection to server is still live. 

Also I think the bit with disposing the msg in error handler is redundant

 (ex) =>
            {
                // Always dispose the BrokeredMessage instance even if the send operation has completed unsuccessfully.
                if (msg != null)
                {
                    msg.Dispose();
                    msg = null;
                }
...

as that already happens when the message is beeing sent in the finally block. 

If anything I'd bet my money on the code that recreates the queue every time you send a new message, try to cache all of that. 

Answer 6

Yes, that code is old. Now I changed the code as you say but it's still failing sometimes. Tha pattern is always one good, one lost until I restart the server and works well for some time again.

I will try checking the IsClosed propert.

Anyway, there's something I didnt mention:

What I need to do, is send messages from a worker role to a web role. Some event happens on worker role and sends the message to web role so it can be delivered to connected clients throught SIGNALR. The web role must be listening to the message queue.

I ended up with this solution: http://blogs.msdn.com/b/piyushjo/archive/2011/11/02/windows-azure-service-bus-push-based-programming-approach.aspx  where a WCF service is listening to the message queue.

Which would be the best way to accomplish this?

By the way, I'm running a x-small instace... could that be the cause of the failure?

For the testing I've done so far, I notice that the push component mentioned is working well. The service bus is failing to deliver the message for some reason I can't understand. I've been strugling with this for months... 

I would like some advice, please. 

Thanks.

Answer 7

Wait a minute... could it be that while I'm working on localhost, my local application is fetching the messages and is competing with the production one? Oh god............ I'll create a dev message queue.. oh god

Answer 8

The good news is that I have the same architecture - worker role sending messages to web role via service bus and web role then passing them to client using SignalR, and it works fine. 

So it is something wrong between your web and worker role. I took me a month to get this working properly.

Put more tracing into place - for when you create and send messages, when you create or delete queues and topics, when you are creating/deleting message senders and receivers, so you have a better idea of what's happening, this can get complex...   

what is your messaging frequency? Ideally you don't want to be recreating any of the above mentioned (senders, receivers, topics/queues), have once instance running at all times. 

just a note - I have not used the WCF to subscribe to the messages, I'm using the service bus classes directly.

Answer 9

:) 

Answer 10

Thank you Stevo... I'm glad we have the same architecture, I may not be so wrong then.

I'll see if creating queues for development solves the problem, I hope.

Answer 11

Hi Stevo,

Do you have some code to share? I am planning a similar architecture here.

Thanks!

Answer 12

Hi Renato,

my code is tied to my business logic, it would not make much sense, but below you can find information that I found most helpful:

• Read Best Practices with Azure Messaging and use Transient Fault Handling block to handle transient errors
• Make sure you understand and handle the non-transient exceptions correctly Appendix: Messaging Exceptions . These will occur and will cause you trouble. 
• Reuse objects, connections that can be reused for optimal performance

Regards,

Stevo