Output options for input bigger than 64kb RRS feed

  • Question

  • Hi, 

    We have telemetry data (around 300,000 events per day) that is coming through the Azure event hub. We are currently using the Azure Table Storage as an output so that we can do quick searches and this is working very well for us.

    We are now faced with an interesting problem where each event we have been receiving above comes with a small image (around 100kb each).

    We first tried using the Azure Table Storage but it has a column size limit of 64kb. We also tried using blob storage but Azure Stream Analytics doesnt allow custom file name and content options so it is not that useful for fast searching.

    We want to keep the existing Azure Table Storage output and create another type of output from the Azure Stream Analytics where we can store these images for fast lookup. (We have a unique id already assigned to each event.)

    Anyone have any suggestions on what I should do?

    Thanks in advance.


    Tuesday, December 27, 2016 9:02 AM

All replies

  • Hi,

    Unless you use an output that doesn't have such size limitation (like SQL or DocDB for example), the scenario you mentioned cannot be implemented seamlessly. I assume you get the image as one of the payload properties, encoded in some kind of text format. Another option to try to split the images into multiple chunks so that each chunk is <64kb, you can write the images to a different table and have a pointer to it in the main table. This can be written in just ASA query. You are right about blob, it doesn't allow custom file names.

    • Proposed as answer by Jeff Stokes Wednesday, February 1, 2017 8:43 PM
    Tuesday, January 17, 2017 6:51 PM