none
While sampling data, no data was received from '2' partitions RRS feed

  • Question

  • I am using an Logic App (LA) on Azure to query my db every 3 mins. Then the LA uses an EventHub connector to send my query result, the table, to Azure Stream Analytics (ASA).

    Normally the result table has around 100 rows, definitely many more in peak time.

    I thought sending Eventhub message one row each time, would incur so many calls, hence perhaps delay the ASA's logic(?)

    My questions are:

    • How to send multiple messages thru the LA's Eventhub Action Connector? I see there's one option: Send one or more events to Eventhub, but wasn't able to figure out what to put in the content. Looks like this is successful:

    "body": [
            {
                "ContentData": "Imp4X3R4bl9hZ2Vfb3ZlcnZpZXci",
                "Properties": [
                    {
                        "tti_IngestTime": "value1",
                        "tti_CheckOutTime": value2,
                        "AssignedTo": null,
                    },
                    {
                        "tti_IngestTime": "value3",
                        "tti_CheckOutTime": value4,
                        "AssignedTo": null,
                    }
                 ]
             }
          ]
    • But the ASA couldn't seem to be able to sample data: 
    While sampling data, no data was received from '2' partitions.
       
    • Sending 100 events one by one to ASA, is there any performance concern?
    Wednesday, September 26, 2018 5:58 AM

Answers

  • Seem to find the answer.

    (1) the JSON I am sending looks correct, and the post request to EvenHub is successful.

    Post body is [{}, {}, {}], which is the correct format

    (2) ASA couldn't read the stream is likely due to not able to deserialize the messages from EventHub.

    I happen to change how I encode the base64 string for the "ContentData" send to the EventHub. It looks like the message sent to the EH, { "ContentData": "some base64() string", "Properties": {} }, the base64() needs to encode the "Properties" value, not anything else, for ASA to be able to deserialize the message.

    It didn't work because I encoded using a random string instead of the value of the "Properties".

    • Marked as answer by ayuspark Thursday, September 27, 2018 7:46 PM
    Thursday, September 27, 2018 7:46 PM

All replies

  • Hi,

    Let me try to answer your 3 questions:

    1. ASA supports multiple events in the same message. If you use JSON, you can have line separation between each event, or using a JSON array: [event1,event2,...]
    2. Looking at your message, it is not JSON compliant, so it can't be deserialized and will be ignored. Also it seems you are using a schema in a single message, you will need to query JSON array by yourself  (more info here). Though, if you follow the recommendation in step 1, this won't be necessary.
    3. Sending 100 events one by one is not an issue, ASA is designed to scale much bigger. 

    Let me know if you have any further question.

    Wednesday, September 26, 2018 6:57 PM
  • Thank you very much for the reply!

    Sorry about bad formatting yesterday!

    So from my LA, the POST to Eventhub looks like

    {
        "host": {
            "connection": {
                "name": "/subscriptions/test/resourceGroups/test/providers/Microsoft.Web/connections/eventhubs-1"
            }
        },
        "method": "post",
        "path": "/test/events/batch",
        "queries": {
            "partitionKey": "0"
        },
        "body": [
            {
                "ContentData": "dHhuX2FnZV9yZXN1bHQ=",
                "Properties": {
                    "tti_IngestTime": "2018-09-26T20:10:55.4480047+00:00",
                    "tti_SLAThresholdMins": 330,
                    "MinsPastSla": -6
                }
            },
            {
                "ContentData": "AhuBA2FnZV9yZXN1bHQ=",
                "Properties": {
                    "tti_IngestTime": "2018-09-26T20:10:55.4480047+00:00",
                    "tti_SLAThresholdMins": 230,
                    "MinsPastSla": -5
                }
            }
        ]
    }

    I am not sure which partition key to put in the call, thence put 0. But now still getting the same error, that "

    no data was received from '2' partitions "

    I don't see a way to specify the partitions in ASA.

    Do you think the partition key was right? (When I put in the LA Evenhub Connector, I can only put 0 or 1)

    Is the JSON in the right format?

    Thanks!



    • Edited by ayuspark Thursday, September 27, 2018 2:06 AM
    Thursday, September 27, 2018 1:53 AM
  • Seem to find the answer.

    (1) the JSON I am sending looks correct, and the post request to EvenHub is successful.

    Post body is [{}, {}, {}], which is the correct format

    (2) ASA couldn't read the stream is likely due to not able to deserialize the messages from EventHub.

    I happen to change how I encode the base64 string for the "ContentData" send to the EventHub. It looks like the message sent to the EH, { "ContentData": "some base64() string", "Properties": {} }, the base64() needs to encode the "Properties" value, not anything else, for ASA to be able to deserialize the message.

    It didn't work because I encoded using a random string instead of the value of the "Properties".

    • Marked as answer by ayuspark Thursday, September 27, 2018 7:46 PM
    Thursday, September 27, 2018 7:46 PM