MSMQ: Messages from Windows 2008R2 server to Windows 2003 server only show up after restart MSMQ service on 2008R2 server

Question

Hi all,

We have a third party application on a Windows 2008R2 server that sends XML messages through a local MSMQ outqueue (DIRECT=http://<server>l/msmq/private$/AccQueue) to a MSMQ queue on a windows 2003 MSMQ private queue. The message do not (or incidently after a delay of more than 18 minutes!) arrive at the windows 2003 server.  When we restart the MSMQ service onthe windows 2008 server the message are delivered in the private queue on the windows 2003 server. After the restart with messages following we have the same issue.

I turned on end2end msmq eventlogging but this just states te message was send to the private queue on the windows 2003 server.

All help will be greatly appreciated.

Answer 1

Hi,

You can try this KB: http://support.microsoft.com/kb/2554746.

Hopefully this will help.

Thanks

Sudhir

Answer 2

Hi,

Thanks for your reply. when I execute a netstat command at the prompt of the new server i get:

:\Users\svc_tripleforms>netstat -ano | findstr 1801

 TCP    10.201.254.233:1801    0.0.0.0:0              LISTENING       2276
 TCP    [2002:5350:19e3:8000:0:5efe:10.201.254.233]:1801  [::]:0
 LISTENING       2276
 TCP    [fe80::5efe:10.201.254.233%13]:1801  [::]:0                 LISTENING
    2276
 TCP    [fe80::a9ce:9490:226b:dd2f%12]:1801  [::]:0                 LISTENING
    2276

So i guess this is not the issue ?

Answer 3

So messages stay in the outgoing queue waiting to be delivered until either you restart the destination queue manager or they get delivered anyway if you wait long enough.

What is the state of the outgoing queue at this time? (Connected, waiting to connect, etc.)

Are these transactional messages? If so, do you see the same delay with non-transactional messages?

Cheers

John Breakwell

Answer 4

Hi John,

You correctly described our situation above. The outqueue is created by the first XML message that comes by. (DIRECT=http://<biztalk server name>/MSMQ/Private$/LoketQueue. There is nothing to configure in MSMQ on the sending server (direct queue). The recipient server (Biztalk) is configured as transactional. Here is a screenshot what i see in the message queue. Check date/time sent en date/time received (!)

We have the same application running on a windows 2003 server that is also writing to the MSQM on the BizTalk server. No delay in message delivery then occurs. All help is welcome! Thanks!

Answer 5

Anybody any ideas ? Still can't figure this out .... :(

Answer 6

Hi,

You didn't answer my questions:

• So messages stay in the outgoing queue waiting to be delivered until either you restart the destination queue manager or they get delivered anyway if you wait long enough.  [[You said "You correctly described our situation above." so I'm assuming you mean" Yes"]]
• What is the state of the outgoing queue at this time? (Connected, waiting to connect, etc.) [[This question is important]]
• Are these transactional messages? [[You said yes]] If so, do you see the same delay with non-transactional messages?  [[Would be a useful test]]

As the messages are transactional, the sender needs to receive delivery acknowledgements from the destination.

If the acknowledgements don't arrive then the transaction can't complete.

I would normally look at network traffic to see if there are any delays in messages or acknowledgements being delivered.

Cheers

John Breakwell

Answer 7

Hi John,

• So messages stay in the outgoing queue waiting to be delivered until either you restart the destination queue manager or they get delivered anyway if you wait long enough.  [[You said "You correctly described our situation above." so I'm assuming you mean" Yes"]] CORRECT
• What is the state of the outgoing queue at this time? (Connected, waiting to connect, etc.) [[This question is important]] When a message is put in the MSMQ state changes from inactive to connected
• Are these transactional messages? [[You said yes]] If so, do you see the same delay with non-transactional messages?  [[Would be a useful test]] I Will try to describe our situation.
  
  We have an application for E-Forms that sends XML messages via MSMQ outqueue (DIRECT=http://<server>l/msmq/private$/AccQueue) to a MSMQ private queue on a windows 2003 Biztalk server. (Server B). There is nothing to see / configure on this Message Queue (Transactional/non transactional/Journal/Dead Letter etc.)
  The only thing I can see is the amount  of messages that was sent through this direct http outgoing queue. (see image below). 
• On Server B we check the journal if the message is received. We see thing like: (see image below)You can see that there is sometimes great delay between the time messages were sent and the time they were received. When we restart the MSMQ service on Server A all messages are delivered.
  

Please help.

Yours sincerely,

Thijs

Answer 8

I ran a network monitor on server B (MSMQ Receive). No http traffic is monitored (WireShark; filter http). When i restart MSMQ service on Server A (MSMQ send) i see the following (see image below): http traffic is received as expected.

The strange thing is, why do I have to restart the MSMQ service to have server A actually send the message ? The MSG count is added up en the message is not unprocessed in the Queue Explorer view? This message queuing thing really confuses me.

Answer 9

Any Ideas anybody? 

Answer 10

OK, so you are POSTing MSMQ messages to the acc13.breda.nl web server.

The MSMQ IIS extension catches the POSTs and creates messages in the loketaccqueue private queue.

As they are transactional messages, acknowledgements are sent back from MSMQ on the webserver.

In the Wireshark screenshot, I can see messages being sent to the webserver.

After half a second delay (which is normal) a POST comes from the webserver to /msmq/private$/order queue$/senderstream….{stuff}...

The original sender (10.201.254.232) responds with an error 400 (Bad Request).

As the order acknowledgement was rejected, I expect there is an amount of resending of messages until eventually an error 200 (OK) is returned.

Unfortunately we don't have the "good" response in the Wireshark trace so can't see how it differs from the "Bad request".

At the moment it looks like a webserver problem rather than an MSMQ one but we don't have enough data to confirm what's going on.

Looking at the web service logs on the sender would be a good start.

Cheers

John Breakwell

Answer 11

Hi John,

Thanks for your reply. In the meantime we tried several things. I found out that I only have issues when the receiving queue is transactional. 

When we configure a new test queue, that is not transactional , then messages do arrive in the message queue ?

I have journalling enabled but no test message is ever journalled, with the transactional as non-transactional queue.

For both queues i attached the wireshark logging (see below). As far as I can see on the network level all works fine for both queues.

Please help.

Answer 12

Hi,

I'm glad you managed to test with non-transactional queues. The big difference is that non-transactional messaging doesn't require acknowledgements to be returned from the destination.

What journaling are you referring to - in the sending app or on the destination queue?

With your new screenshot, you don't say if there is still a problem with the transactional queues - I assume there is, though.
I see 5 POSTs to loketaccqueue - is that 5 different messages, or a large message split into 5 POSTs, or the same small message re-sent 5 times? I don't see an order acknowledgement being returned from the webserver (which did happen in the previous screenshot, although it generated an error response then).

If you are going to collect network traffic, you need to do so for the whole length of the time it takes from initial sending of a single message to final arrival in the queue. Don't filter on HTTP as then you miss any TCP packet issues.

Make sure you keep an eye on the web server logs on both sending and receiving machines:

http://blogs.msdn.com/b/johnbreakwell/archive/2009/10/21/troubleshooting-msmq-over-http-nothing-in-the-web-server-log-files.aspx

Cheers

John Breakwell

Answer 13

Hi Thijs,

Are you running any more websites on the same machine/hostname? Transactional queues requires a response to the sender via port 80 with the hostname. Try to disable the other sites or run them  on a different port. Looks like your repsonse ends up at the wrong website.

Cheers,
Paul

;-)

Answer 14

Hi Paul,

Are you trying to be funny ? We solved the problem together at work .....

Answer 15

What was the solution?

Answer 16

IIS 7 works different from IIS6. Websites  can be bound to the same portnumber...i guess transactional acknowledgements arrived at the wrong website....one  website now listens on port 88 and later will be changed to a host header name, wich also works.