locked
System.TimeStamp and Adjust RRS feed

  • Question

  • Hi everyone,

    I have a question related to the System.TimeStamp property. The situation is:
    - I'm using TIMESTAMP BY in my query.
    - The field used in TIMESTAMP BY does not contains a time zone mark, i.e. Stream Analytics considers that it's in UTC.
    - I've enabled the option "Event ordering" > "Handling other events" > Adjust.

    I've observed that if I send an event that goes behind UTC (1 hour behind, for example), the current UTC time is assigned to that event. On the other hand, if I send an event that it's ahead UTC (1 hour ahead, for example), the timestamp assigned is UTC+5minutes.

    Why exactly 5 minutes? Is that 5 minutes difference documented anywhere? How can I reprocess old data (last month data for example) if the current UTC time is assigned to every output?

    Regards.

    Guiferviz.

    Wednesday, February 8, 2017 5:18 PM

Answers

  • There is indeed 5 minutes fixed “early arrival” policy. What it means is that event timestamp (defined by TIMESTAMP BY) cannot be more than 5 minutes later than arrival time (timestamp assigned by the input source, e.g. Event Hub, Blob). It is typically not a problem for real-time systems as events are not arriving with the “future” timestamps. In case of processing historic data, timestamps are typically significantly less than arrival time so it is not a problem either (but you need to set Late Arrival policy to indicate that timestamps are from the past).

     

    If your timestamp field is not in UTC time zone, you can adjust it in TIMESTAMP BY expression itself.

    • Marked as answer by Guiferviz Thursday, February 9, 2017 10:20 AM
    Wednesday, February 8, 2017 5:49 PM

All replies

  • There is indeed 5 minutes fixed “early arrival” policy. What it means is that event timestamp (defined by TIMESTAMP BY) cannot be more than 5 minutes later than arrival time (timestamp assigned by the input source, e.g. Event Hub, Blob). It is typically not a problem for real-time systems as events are not arriving with the “future” timestamps. In case of processing historic data, timestamps are typically significantly less than arrival time so it is not a problem either (but you need to set Late Arrival policy to indicate that timestamps are from the past).

     

    If your timestamp field is not in UTC time zone, you can adjust it in TIMESTAMP BY expression itself.

    • Marked as answer by Guiferviz Thursday, February 9, 2017 10:20 AM
    Wednesday, February 8, 2017 5:49 PM
  • Ok, I got it.

    Always use datetimes with the time offset from UTC so TIMESTAMP BY can adjust to UTC (this avoids the fixed "early arrival" policy for events ahead of UTC).

    In order to process historic data: change the "arrive late" policy and the start time of the job. Those two changes will make ASA process the events correctly (System.TimeStamp returns the correct time).

    Thank you very much.
    Thursday, February 9, 2017 10:20 AM