locked
Issues with Stream Analytics jobs that use a non-default Consumer Group from an IoT Hub RRS feed

  • Question

  • I cannot get any new Stream Analytics job (tied to an IoT hub) to work properly when the Input is configured to use a non-default Consumer Group.  Something seems to have changed, because this functionality used to work.  Any ideas?

    Background:

    I have a past SA job with multiple Inputs, each of which is pointing to a dedicated Consumer Group that was configured within the IoT Hub.  This job worked as expected.  But I have recently created new Stream Analytics jobs with similar configurations, where there are multiple Inputs connected to an IoT hub, and these jobs fail to stream any data on any Input that is using a non-default Consumer Group (The list of consumer groups was configured on the General|Operations Monitoring tab of the IoT Hub).  These new jobs start working only when the Consumer Group is removed from the Inputs and the Inputs revert back to the $Default group.  The problem with this is that I am frequently exceeding the limit of 5 receivers.

    There is also an issue with the Azure UI in terms of how Consumer Groups are assigned to an SA Input.  When you create a new SA Input, after you select the desired IoT Hub to connect to, Azure reports that it is loading both the Shared Access Policies and the Consumer Groups from the selected IoT Hub.  But once this load process is complete, I do not see any of the configured Consumer Groups listed in the Consumer Group drop-down.  It only shows the $Default group.  So the only way to assign a different Consumer Group is to complete the Input creation with the $Default consumer group, then go back into the Input and edit the Consumer Group directly by typing the desired name into the Consumer Group text box (there is no drop down list on the Edit Input screen, only a text box).  So obviously the entire process of selecting Consumer Groups is not working, which may be directly related to why non-default Consumer Groups aren't working.

    Friday, January 20, 2017 9:19 PM

Answers

  • In my case, I configured the ASA jobs the same as you, each with their own CG. What I learned was in the IoT Hub, Endpoint properties Device-to-cloud settings each CG needs to be configured as well. There is a max of only 20 CG, but each CG can have 5 readers = 100 ASA jobs. 
    • Marked as answer by Brice Bucher Friday, February 10, 2017 3:03 AM
    • Unmarked as answer by Brice Bucher Friday, February 10, 2017 3:06 AM
    • Marked as answer by Brice Bucher Friday, February 10, 2017 3:06 AM
    Tuesday, February 7, 2017 11:36 PM

All replies

  • I am having the same issue.. I just submitted to premium support.. Hope to have an answer in the next few hours.

    Monday, February 6, 2017 7:05 PM
  • In my case, I configured the ASA jobs the same as you, each with their own CG. What I learned was in the IoT Hub, Endpoint properties Device-to-cloud settings each CG needs to be configured as well. There is a max of only 20 CG, but each CG can have 5 readers = 100 ASA jobs. 
    • Marked as answer by Brice Bucher Friday, February 10, 2017 3:03 AM
    • Unmarked as answer by Brice Bucher Friday, February 10, 2017 3:06 AM
    • Marked as answer by Brice Bucher Friday, February 10, 2017 3:06 AM
    Tuesday, February 7, 2017 11:36 PM
  • Hi, Brice Bucher

         Can you please following this document to retry, it contains step by step how to create Consumer group on IoT site, and make sure when you create ASA input, choose either the message or monitoring endpoint whichever you like to subscribe data from.  We verified by using default or non default consumer group and connect to the right endpoint where the consumer group created, the ASA job should work.

    Let us know if you have further question

    https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal

    Thursday, February 9, 2017 5:13 AM
  • I had configured Consumer Groups on the Operations Monitoring tab.  I was not aware that you can also configure Consumer Groups on the properties page of the built-in Events endpoint.  This was even more confusing because the Consumer Groups label is hidden on the Events properties page.  But once I configured the consumer groups here, everything started working correctly.  Thanks!

    Friday, February 10, 2017 3:14 AM