locked
How to define date / time placeholders depending on timestamped field? RRS feed

  • Question

  • Hi everyone,

    I'm wondering if it is possible (and so how) to use the "timestamp by" field as a reference for the placeholders in the path of a Blob output.

    Here is my situation. Assuming we have :

    • an Event Hub as an input to an ASA : eventhubIn
    • a blob container as an ouput with path set to logs/{date}/{time} (date= YYYY/MM/DD ; time = HH } : blobcontainerOut

    I want to make some "recovery" functionality. I'm using ASA as an aggregator for pages view. If, for any reason, my event hub injestion process break down, I want to be able to recover data thanks to IIS logs simply by sending events to my Event Hub.

    I already wrote the log parser and so on. Because I can't use the same ASA as the running environment (otherwise my "recover" data will be identified as "late events"), I created a new Event Hub and a new ASA with the same query (and to simply avoir duplicated processed data)

    When I run the process, everything works well except that my blob container uses the EventProcessedUtcTime to generate the path. Is there any way to say to ASA "hey I want you to use that field to generate {date} and that one for {time}" (or that fields as a Date reference)?

    Here is the query used:

    SELECT * INTO blobstorageOut FROM eventhubInAS chh TIMESTAMP BY Payload.Date;

    -- Some more stuff out of interest here

    And here are some events sample:

    [
    {"Payload":{"Date":"2016-01-01T09:56:47.0000000","Value":42}},
    {"Payload":{"Date":"2016-01-02T08:58:18.0000000","Value":42}},
    {"Payload":{"Date":"2016-01-02T09:58:10.0000000","Value":42}},
    {"Payload":{"Date":"2016-01-02T09:53:46.0000000","Value":42}},
    {"Payload":{"Date":"2016-01-03T07:58:44.0000000","Value":42}},
    {"Payload":{"Date":"2016-01-03T07:58:13.0000000","Value":42}},
    {"Payload":{"Date":"2016-01-03T07:59:11.0000000","Value":42}}
    ]

    Whenever I run the ASA, I would like to have so the following directories (regarding this sample):

    • 2016/01/01/09
    • 2016/01/02/08
    • 2016/01/02/09
    • 2016/01/03/07

    How can I achieve that from ASA?

    Many thanks,



    • Edited by escandells Wednesday, January 6, 2016 7:14 AM
    Tuesday, January 5, 2016 6:35 PM

Answers

  • Hi again,

    OK, I think I found how to resolve my situation. I just increased the Late arrival tolerance window option to something "important enough" so my ASA uses my Payload.Date and it works just fine.

    Sorry for the inconvenience.

    • Marked as answer by escandells Wednesday, January 6, 2016 7:25 AM
    Wednesday, January 6, 2016 7:25 AM