locked
Querying an event hub populated by another query RRS feed

  • Question

  • I have a set of entities that a job runs over to create results for each entity each minute. The results are pushed into an event hub where a tumbling 5 minute window query then outputs 5 minute aggregates to another event hub. I also have a one hour tumbling window query.

    Currently I'm pointing that one hour query at the event hub holding the 5 minute aggregate window rather than the raw results event hub. The reason for this is that the aggregates can be calculated from other aggregates and in doing so I can reduce the data flowing to the hour query. This will then also allow me to scale the queries differently as required.

    The main issue ATM is that I am not getting results through to the hour query. Is this design a good idea or would it be more reliable for the hour query to read the raw results event hub?

    Thursday, June 18, 2015 2:23 AM

Answers

  • We have recorded the request to support offsets on windowed aggregations in ASA.

    Thanks


    Zafar Abbas

    • Marked as answer by Zafar Abbas Wednesday, July 1, 2015 8:52 AM
    Wednesday, July 1, 2015 8:52 AM

All replies

  • Hi Rory,

    Your design should work. Could you confirm that the hour query is getting input events? You can see that from the Monitoring tab in the Azure portal. If you are getting events but still do not see any output, then we would like to debug further and would need more job related information from you.

    Thanks


    Zafar Abbas

    Monday, June 22, 2015 12:20 AM
  • I tweeted about it last week - https://twitter.com/roryprimrose/status/611790277628440576

    I change the query and input to use the raw event hub rather than the five minute event hub and found the same thing. It isn't so much a problem ATM because I'm going to go via SQL Azure to calculate hourly aggregates from five minute aggregates. The main reason for this is that ASA queries don't currently support offsets on tumbling windows (I need hour aggregates on the 0, 30 and 45 minute offsets).

    Monday, June 22, 2015 4:06 AM
  • We have recorded the request to support offsets on windowed aggregations in ASA.

    Thanks


    Zafar Abbas

    • Marked as answer by Zafar Abbas Wednesday, July 1, 2015 8:52 AM
    Wednesday, July 1, 2015 8:52 AM