How the timestamp values are sent/recorded? Are they unified UTC?


  • Question: let say I send two telemetries, 1 from CA and another from New York, at the same time, will the recorded timestamp be same? What I'm noticing is different. For example if CA is timstamped as "2017-12-19T20:57:31.071Z", the New York will be timestamped as "2017-12-19T23:57:31.071Z". Wouldn't these to be same UTC time? 

    Now, lets look at below query where I am interested in entries in last 1 minute (i.e. timestamp >= now(-1m)). The response include entries way beyond 1 minute. And to me the reason is, the timestamp is not normalized/converted to true UTC.  

    | where name == "Latency" 
    | where timestamp >= now(-1m) 
    | order by timestamp desc 
    | project timestamp , client_StateOrProvince 


      2017-12-19T23:54:39.356 Kentucky
    2017-12-19T23:52:20.916 Kentucky
    2017-12-19T23:46:10.088 Kentucky
    2017-12-19T23:30:51.600 Ontario
    2017-12-19T23:29:59.924 Ontario
    2017-12-19T23:28:36.508 Ontario
    2017-12-19T23:22:15.743 Bavaria
    2017-12-19T23:21:11.800 Bavaria
    2017-12-19T23:16:05.088 Georgia
    2017-12-19T23:13:53.535 North Carolina
    2017-12-19T23:12:08.868 North Carolina

    Or, I am missing something here? Do we support UtcNow or something?


    • Edited by AzureUser123 Wednesday, December 20, 2017 5:08 AM
    Wednesday, December 20, 2017 1:46 AM

All replies

  • You should be able to repro this easily. Steps:

    - Add Azure App Insights in a test web page
    - Add a test telemetry/event
    - Run your test page in two different time zones
    - Trigger your telemetry/event 
    - Check the recorded timestamp for above 2 entries under your Application Insight (an entry from each time zone)
    - Both timestamps should be nearly same UTC time, but it's not (at least for me). 



    Wednesday, December 20, 2017 5:14 AM
  • Hi there, 

    I've set up a test scenario where I can try and reproduce your results, but am having difficulty doing so. Perhaps a bit more information will help me understand better what the cause of this issue might be.

    I've set the following up on my account:

    2 web apps hosting a static web page instrumented with Application Insights JavaScript.

    • Static web pages, no ASP.NET or anything
    • Instrumented using the Application Insights JavaScript SDK
    • One is hosted in East US
    • The other is hosted in West Europe
    • Both are sending their telemetry to the same Application Insights component in East US

    I've not added any custom telemetry of my own, I've only added the 'pageView' event built into the SDK, which provides us with a timestamp.

    After deploying & running my application, I hit refresh on my browser on each page to generate some pageView telemetry.

    After a minute or two, I ran this query in my analytics to observe the timestamp information being skewed:

    | order by timestamp desc

    ...the results of which are:

    Results from using my setup to reproduce the issue (I have not successfully reproduced the issue) you can see I have not reproduced the issue with my setup.

    A few things I've noted about the issue:

    Your query has 'timestamp >= now(-1m)' in it, I can't understand how you are getting the results you are getting with that clause in your statement as it seems like it would cull everything that didn't happen in the last minute? Try simplifying the query to see if you can get the highlighted results in a more simplistic fashion, something like

    | where name == "Latency" 
    | order by timestamp desc

    ...and just report back the top 10 rows?

    I would love to know the exact setup you are using to create the scenario as well:

    • Web Application language/version (.NET, .NET Core, Static HTML, other?)
    • Web application platform (Azure Web App, IIS-hosted on VM, etc.?)
    • Code snippets surrounding your custom telemetry
    • Is it only reproducible using custom telemetry, or do you see the issue with things built into the SDK such as events/traces/metrics/etc.?

    • Edited by d3r3kk Thursday, December 28, 2017 5:38 PM formatting
    Thursday, December 28, 2017 5:37 PM
  • Thanks for looking into this. Btw, I have a quick question with your repro setup. I can see that you have setup your web pages in two different DCs but from where exactly are you running these web pages? You (i.e. client/browser) will need to run these pages in respective DCs. Basically, you don't need to setup your page in two DCs, you just need to run it from different DCs (i.e. client should be in different DCs) to repro what I'm seeing.  Just confirming is that what you have done?
    Wednesday, January 03, 2018 7:58 PM
  • You are exactly correct. I set up my two web apps in different data centers, one in East US and another in West EU, and then tested them by hitting the apps with my browser located in west coast North America.

    I will perform some testing using VMs located in the two data centers (and perhaps a third different data center as well) and reply shortly.

    Wednesday, January 03, 2018 8:03 PM
  • Cool. Waiting to hear your findings :-). Thanks again. 
    Wednesday, January 03, 2018 9:12 PM
  • To check this issue out further, I performed the following:

    I've created a VM in West Europe using the timezone UTC, and UTC+1 (Amsterdam) and one in Western North America (Alberta, Canada) with a timezone of UTC-7 (Mountain Standard Time).

    I then constructed a powershell script with the following contents to hit my service (located in both West Europe as well as East US) as such:

    Invoke-WebRequest -Uri "" -Method Get
    Invoke-WebRequest -Uri "" -Method Get
    Invoke-WebRequest -Uri "" -Method Get

    I run the script on both VMs, and I then head over to the analytics portal in Azure and make the following query:

    | order by timestamp desc
    | project timestamp, client_StateOrProvince, client_CountryOrRegion 

    Which results in the following expected results (note there seems to be some clock drift between the two sites):

    timestamp [UTC], client_StateOrProvince, client_CountryOrRegion
    2018-01-03T22:08:23.504, Alberta, Canada
    2018-01-03T22:08:23.096, Alberta, Canada
    2018-01-03T22:08:22.317, Alberta, Canada
    2018-01-03T22:05:09.957, North Holland, Netherlands
    2018-01-03T22:05:09.657, North Holland, Netherlands
    2018-01-03T22:05:08.843, North Holland, Netherlands

    I then altered the host in West Europe such that it's time zone was in UTC+1 (Amsterdam) and ran the same script again in both locations, the results are similar:

    timestamp [UTC], client_StateOrProvince, client_CountryOrRegion
    2018-01-03T22:13:27.662, Alberta, Canada
    2018-01-03T22:13:27.316, Alberta, Canada
    2018-01-03T22:13:26.335, Alberta, Canada
    2018-01-03T22:10:12.821, North Holland, Netherlands
    2018-01-03T22:10:12.489, North Holland, Netherlands
    2018-01-03T22:10:11.631, North Holland, Netherlands

    Lastly, I changed the time on the system clock on my West Europe VM to be an incorrect time (add 3 hours) and re-ran the script giving me the following result:

    timestamp [UTC], client_StateOrProvince, client_CountryOrRegion
    2018-01-03T23:11:36.538, Alberta, Canada
    2018-01-03T23:11:36.136, Alberta, Canada
    2018-01-03T23:11:35.217, Alberta, Canada
    2018-01-03T23:11:29.927, North Holland, Netherlands
    2018-01-03T23:11:29.605, North Holland, Netherlands
    2018-01-03T23:11:29.114, North Holland, Netherlands

    The timestamps are correct! Which leads me to believe that the timestamp code is functioning how I would expect it to, and it's getting its information independently from the system clock.

    Can you perform the tests as I have against your two test pages, and let me know what results you get (using specifically, the pageViews query I have above)?

    Thursday, January 04, 2018 12:16 AM
  • Ok, I get it now. See in my early two tests there is a slight discrepancy in the time? In the last test where I actually changed the clock, the time difference between the two client locations is much closer.

    The original clock skew was likely due to the new VM not having 100% perfect time, but slowly being corrected by the NTP service.

    The last test I performed is incorrect since I cannot seem to actually change the time on my Azure VM! It appears to change when I manually set it, then in 10 seconds or so it goes back to the 'correct' system time automatically - I hadn't noticed that originally.

    I'm going to run the test again, this time running the test from a desktop where I can control the time setting. I expect you will see the skew you expect in that case.

    Which then leads me to believe that there is something incorrect in your test setup perhaps, an incorrect time setting or an incorrect region?

    • Edited by d3r3kk Thursday, January 04, 2018 12:30 AM
    Thursday, January 04, 2018 12:29 AM
  • Something is really messed up here. Let me give you more details here:

    Query 1: 

    | where name == "ModelLookupLatency" 
    | order by timestamp desc   
    | project timestamp, client_StateOrProvince  
    | take 5


    2018-01-04T19:41:02.590 New York
    2018-01-04T19:40:55.176 South Carolina
    2018-01-04T19:39:12.123 North Carolina
    2018-01-04T19:38:26.737 California

    Query 2:

    | where name == "ModelLookupLatency" 
    | where timestamp > now(-1m)
    | order by timestamp desc   
    | project timestamp, client_StateOrProvince 
    | take 5


    2018-01-04T21:14:33.030 Illinois
    2018-01-04T21:12:40.190 Alabama
    2018-01-04T21:12:20.250 Texas
    2018-01-04T21:00:45.629 Florida
    2018-01-04T20:59:55.613 Kansas

    The current UTC time is ~ 2018-01-04T19:41:02.590

    So, the question is, why do I see entries beyond (i.e. >) current UTC in the second query? 

    Btw, do you work at msft? If so, I can reach out to you directly and show you things in detail.  

    Thursday, January 04, 2018 8:01 PM
  • Yes, that is a very odd result in the query! I would expect to see only things less than the current UTC in both your queries, or I would expect to see the same results.

    I do wonder if there is a time setup issue on your Alabama/Texas/Florida/Kansas/Illinois machines, but not on your *Carolina/California/New York machines...

    You can find me inside MS via this: please feel free to reach out. (I want to ensure we put any resolution here though, so we can help further users that encounter this issue going forward).

    Friday, January 05, 2018 7:38 PM
  • I've put this issue as 'mitigated' for now, but am very interested in finding a way to reproduce this issue. Note: I can reproduce it if I set my clock improperly, so at this point I believe the fix to be "Please ensure the time & region settings on host computers are set correctly for their location."

    If this is not the fix, if there is some other problem that cannot be corrected by fixing the time/region on your host computers please do respond and we can dig further.

    • Proposed as answer by d3r3kk Thursday, January 18, 2018 9:57 PM
    Friday, January 12, 2018 5:17 PM