locked
Handling updated event data RRS feed

  • Question

  • Hi there, I am streaming data into the Event Hub. This is all fine.

    My question is around what happens if I resend the same transaction but this time with an updated field?

    At the moment I am outputting to PowerBI and a SQL DB but at the moment this just adds records as they come in. What is the best approach for updating (or upserting?) existing records within the SQL DB - realtime?

    Should this be offloaded as part of an ETL/Cleansing process post insert or is there a way of doing this when streaming the data?

    Thanks

    Immy

    Friday, December 29, 2017 3:57 PM

All replies

  • Hello, ImtiazUllah

       I think Azure Stream Analytics may be a good choice for you requirement. Azure Stream Analytics service supports doing stream processing directly from Event Hubs. It a fully managed real-time distributed stream computation service that provides low latency, scalable processing of streaming data in the cloud with an enterprise grade SLA. Stream Analytics connects directly to Azure Event Hubs and Azure IoT Hub for stream ingestion, and to Azure Blob storage service to ingest historical data.Route Stream Analytics job output in many directions, write to storage like SQL DB, or to Power BI for visualization.

    Best Regards,

    Michael


    MSDN Community Support Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, January 2, 2018 7:39 AM