locked
Most simple query on Event Hub stream (json) constantly gives Data Conversion Errors RRS feed

  • Question

  • Hello all,

    Been playing with ASA in December and didn't have any issues, my queries kept working and outputted the data as needed.  However, since January, I created a new demo, where I now constantly get Data Conversion errors.  The scenario is described below, but I have the following questions:

    • Where can I get detailed information on the data conversion errors?  I don't get any point now (not in the operation logs and not in the table storage of my diagnostic storage account)
    • What could be wrong in my scenario and could be causing these issues

    The scenario I have implemented is the following:

    • My local devices send EventData objects, serialized through Json.Net to an Event Hub with 32 partitions.
    • I define my query input as Event Hub Stream and define the data as json/utf8.  I give it the name TelemetryReadings
    • Then I write my query as SELECT * FROM TelemetryReadings
    • In the output, I create an output on blob with CSV/UTF8 encoding
    • After that, I start the job

    The result is an empty blob container (no output written) and tons of data conversion errors in the monitoring graph.  What should I do to get this solved?

    Thanks


    Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu


    Tuesday, January 20, 2015 9:37 AM

Answers

  • So, apparently the issue was related to the incoming objects, I had.  I was sending unsupported data types (boolean and Dictionary).  I changed my code to remove these from the json and that worked out well.  There was a change that got deployed that (instead of marking the unsupported fields as null, they were throwing an exception).  That's why things worked earlier.

    So, it had to do with the limitation that I mentioned in my earlier comment:

    https://github.com/Azure/azure-content/blob/master/articles/stream-analytics-limitations.md

    Unsupported type conversions result in NULL values

    Any event vales with type conversions not supported in the Data Types section of Azure Stream Analytics Query Language Reference will result in a NULL value. In this preview release no error logging is in place for these conversion exceptions.

    I am creating a blog post on this one


    Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu

    Wednesday, January 21, 2015 1:57 PM

All replies

  • Could it have something to do with the following known limitation?

    https://github.com/Azure/azure-content/blob/master/articles/stream-analytics-limitations.md

    Unsupported type conversions result in NULL values

    Any event vales with type conversions not supported in the Data Types section of Azure Stream Analytics Query Language Reference will result in a NULL value. In this preview release no error logging is in place for these conversion exceptions.


    Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu

    Tuesday, January 20, 2015 9:54 AM
  • Hey Sam,

    I have a similar issue, jobs that worked but don't anymore with a conversion error, but with CSV input / CSV output (elsewhere in this forum).

    In my case switching to a JSON input mitigated the issue. Maybe you could try the other way around (just a sanity check that the rest of your configuration is ok).

    Tuesday, January 20, 2015 4:16 PM
  • Thanks, Florian,

    I tried everything (changed to different eventhub, changed query to only select one field of the input, changed encoding to JSON/CSV on the output). The only thing I didn't change was the input in CSV, instead of JSON. Maybe I can try that (however, I don't like to change my code from json to csv :))


    Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu


    Tuesday, January 20, 2015 4:58 PM
  • Update: I changed the input type to CSV and that seems to work.  So it's really related with the JSON encoding.  I am linking the code I am using for the creation of my EventData object.  With the ToCsv method, it works, with the ToJson, it doesn't work.

    I created a gist here: https://gist.github.com/SamVanhoutte/60d740c185adfa8b1a72#file-eventhub-cs


    Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu

    • Proposed as answer by Zafar Abbas Wednesday, January 21, 2015 2:04 AM
    Tuesday, January 20, 2015 5:52 PM
  • Hey Sam,

    I found a lead for my problem. Could you please check if that could apply to your issue?

    Thx!

    Wednesday, January 21, 2015 1:07 PM
  • So, apparently the issue was related to the incoming objects, I had.  I was sending unsupported data types (boolean and Dictionary).  I changed my code to remove these from the json and that worked out well.  There was a change that got deployed that (instead of marking the unsupported fields as null, they were throwing an exception).  That's why things worked earlier.

    So, it had to do with the limitation that I mentioned in my earlier comment:

    https://github.com/Azure/azure-content/blob/master/articles/stream-analytics-limitations.md

    Unsupported type conversions result in NULL values

    Any event vales with type conversions not supported in the Data Types section of Azure Stream Analytics Query Language Reference will result in a NULL value. In this preview release no error logging is in place for these conversion exceptions.

    I am creating a blog post on this one


    Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu

    Wednesday, January 21, 2015 1:57 PM