locked
Can't publish to Service Bus Topic - Message Size RRS feed

  • Question

  • Hi,

    I have a process where a publisher will send JSON to an Azure Event Hub, Stream Analytics is consuming that message and performing a very simple query:

    SELECT *
    INTO <Topic>
    FROM <EventHub>

    Most of our messages are small <10kb but we got a larger message sent in and got the following error message.  What does not make sense to me is that the max message size of a topic should be 256 kb.  So why is it complaining about 

    Message exceeds maximum message size:124854 Max: 122880

    Also, how can I be alerted when the ASA job stops?  In this case I understand why it stopped but how do I get informed of such event?

    Correlation ID:

    47c56cdb-425b-45de-9012-8802f1f15b0f

     

    Error:

    Message exceeds maximum message size:124854 Max: 122880

     

     

    Message:

    Service Bus Topic Send failed to topic [<topic>] error at [0/1] rows. This operation will be retried and this message may be suppressed for next few minutes.

     

    Message Time:

    2016-03-02 17:52:26Z

     

    Microsoft.Resources/EventNameV2:


     

    Microsoft.Resources/Operation:

    Send Events

     

    Microsoft.Resources/ResourceUri:


    Type:

    ServiceBusOutputAdapterTransientError


    http://kentweare.blogspot.com


    • Edited by Kent Weare Wednesday, March 2, 2016 10:24 PM
    Wednesday, March 2, 2016 10:21 PM

Answers

  • Hi Kent,

    The 256kb limit is on the entire message at the TCP level. Besides the payload, Service Bus messages have several other system properties (PartitionKey, SessionId, Timestamp and some more metadata properties)  including which it effectively restricts a ASA service bus client to send larger messages than the max limit shown above .

    Here is one thing that can help alleviate this issue - You can set alert rules on metrics in the Monitoring section of ASA job. The relevant metrics you may want to create Alert Rules on are -- "Runtime errors" and "Input event bytes". When beyond a certain limit in a given timespan, email alerts can be automatically sent .

    Also in near future, you will be able to decide what automatic action you want to take with such error events , e.g. stop the job vs drop/skip the event.


    [Disclaimer] This posting is provided "AS IS" with no warranties, and confers no rights. User assumes all.

    Thursday, March 3, 2016 2:33 AM

All replies

  • Hi Kent,

    The 256kb limit is on the entire message at the TCP level. Besides the payload, Service Bus messages have several other system properties (PartitionKey, SessionId, Timestamp and some more metadata properties)  including which it effectively restricts a ASA service bus client to send larger messages than the max limit shown above .

    Here is one thing that can help alleviate this issue - You can set alert rules on metrics in the Monitoring section of ASA job. The relevant metrics you may want to create Alert Rules on are -- "Runtime errors" and "Input event bytes". When beyond a certain limit in a given timespan, email alerts can be automatically sent .

    Also in near future, you will be able to decide what automatic action you want to take with such error events , e.g. stop the job vs drop/skip the event.


    [Disclaimer] This posting is provided "AS IS" with no warranties, and confers no rights. User assumes all.

    Thursday, March 3, 2016 2:33 AM
  • Thanks for the prompt response.  Just a clarification though when the error message says

    Message exceeds maximum message size:124854 Max: 122880

    What does Max: 122880 represent?  Is that bytes?  If so that is far smaller than 256 kb.  Fully understand the additional properties showing up but those are very small...maybe a couple kb.

    Also I did have Runtime Error Alerts configured but they got picked up by office 365 Clutter (argh...i thought they werent working).

    The features you are talking about are really welcomed...especially skipping an event.  We are going live to prod on April 24th.  Are we in that window?  If you want any private testing...will sign up for that feature.

    Thanks

    Kent


    http://kentweare.blogspot.com

    Thursday, March 3, 2016 3:08 AM
  • Hi Kent,

    We will investigate the maximum size allowed on Service Bus issue further and release improvements on it.

    Besides that, for skipping event feature, its likely we will be in that window but can't offer any guarantees.

    cheers,
    Chetan


    [Disclaimer] This posting is provided "AS IS" with no warranties, and confers no rights. User assumes all.

    Friday, March 4, 2016 10:19 PM