none
Requests stopped logging for Functions App around 2019-04-21T17:30 (NZDT) RRS feed

  • Question

  • It appears that my App Insights instance stopped logging requests at 2019-04-21T17:30 NZDT.

    (Most of my functions run every 30 minutes and one runs every 4 minutes)

    Dependencies and Traces are still being logged as usual, but no records for requests are logged after that time.

    Interestingly manual executions of functions seem to still log requests, just not ones triggered by Timer Trigger in functions.

    Tuesday, April 23, 2019 12:46 PM

Answers

  • Update 4: 

    Azure support confirmed this is caused by a minor bug in the backend (AppService I assume.)

    Will close this issue here.

    Response from AppService/support team below.

    Our backend team managed to capture a minor bug in the platform that have caused this issue.

                                                                                                                

    For function apps in consumption plan, the function can be move to different instances from time to time. This bug prevent the function app to be fully registered after moving to new instance until one of the functions is triggered externally (with triggers such as http request).

    The backend team also provided a mitigation: if you find the logs are missing in Metrics, please visit the status page of the function app https://<APPNAME>.azurewebsites.net/. This inbound request should complete the register and mitigate the issue.

    Our backend team has prioritized this issue. We are sorry for the inconvenience caused. The fix should be released with the next platform update; until then, please mitigate the issue with the mitigation provided should the issue occurs again.


    Tuesday, July 30, 2019 9:37 AM

All replies

  • Madushan,

    Just to make sure we are covering our bases, have any changes been made prior to or around that time?  It's possible that you could hit a daily limit with Function Apps if you are trying it out for free, and there are some other options that would reduce the amount of data that you receive, but that doesn't seem to be the case here.

    Tuesday, April 23, 2019 3:13 PM
    Moderator
  • Hi,

    No I doubt it. Traces and dependencies are still working without any issues, so it's unlikely a quota issue.

    There were 2 deployments to the app, but I don't think they're related. My reasoning being

    - App doesn't have any code logging requests or dependencies, I only use the built in integration.

    - dependencies and traces still work as they should.

    - Oddly, if I manually execute a function through the Azure portal, that execution gets logged in requests, just not the executions triggered by TimerTrigger.

    I dont know if this is related, I had issues with monitoring before (not App Insights, the App Service Plan > Metrics) where it would show 0 for memory (avg working set and private bytes) and funcion execution count and function execution units. (These did not affect App Insights before)

    They would usually recover by themselves after a few days and wasn't critical to me. They also show the 0 issue around the time of the start of this issue, but they have now recovered.

    They have started showing 0 at around 2019-04-21T17:30 (NZDT) and seems to have recovered around  2019-04-22T05:00 (NZDT). I can see app insights traces has been working throughout this time and has been since. But requests stopped working around that time and has not recovered.

    Please refer to attached images for Metrics graphs.

    Wednesday, April 24, 2019 11:02 AM
  • Madushan,

    Yeah, I agree.  That does seem unlikely.  It looks like we will need to do some additional investigation to figure exactly what is going on here.  Please email the following to AzCommunity@microsoft.com and we'll get back to you promptly:

     

    • Your Subscription ID (this can be found by searching for "Subscriptions" in the portal)
    • A link to this thread so we can validate and expedite the request
    • A brief, 2-3 sentence summary of your issue
    • The person you were working with (kobulloc)

     

    That would look something like this:

     

    I was working with kobulloc on an Application Insights issue where I observed that requests stopped logging for functions on a specific date.

     

    Forum Link:

    https://social.msdn.microsoft.com/Forums/en-US/f2e15296-246d-4967-8384-3cd46ee4a172/requests-stopped-logging-for-functions-app-around-20190421t1730-nzdt?forum=ApplicationInsights

     

    Subscription ID:

    aaaaaa-bbbb-cccc-1111-222222222222

    Wednesday, April 24, 2019 7:47 PM
    Moderator
  • Thank you kobulloc. I have emailed AzCommunity and they have temporaly enabled technical support for my subscription.

    I have created a support request. Will update here with resolution.

    Thursday, April 25, 2019 4:53 AM
  • Thank you, Madushan!
    Friday, April 26, 2019 12:09 AM
    Moderator
  • Update:

    This is still an ongoing investigation although the issue is now mitigated. Requests do get logged now but is not reliable, as it still tends to skip logging certain requests. (20 of the last 48 executions/requests have been logged)

    Azure support has notified me that they have involved support engineers from the Function App team as well. (I logged it as an AppInsights issue.)

    Will keep this thread updated.

    Friday, May 3, 2019 12:04 PM
  • Thank you very much for the update, Madushan!
    Friday, May 3, 2019 9:50 PM
    Moderator
  • Update 2:

    After speaking with a colleague about this issue, I was advised to add the following to the host.json in the functions. 

    "logging": {   
        "applicationInsights": {     
            "samplingSettings": {
                "isEnabled": true,       
                "maxTelemetryItemsPerSecond":5000
             }
         }
      }

    (My friend had a similar issue with an ASP.NET (.NET Framework) application which this resolved. His fix was apply the corresponding setting to web.config)

    I'm waiting on Azure support to confirm if this is a good/bad "fix" but so far (last 24+ hours) has not dropped any request logs after applying this change.

    Metrics issue is likely unrelated and has not occurred since the start of this issue.

    Tuesday, May 7, 2019 8:23 AM
  • Update 3:

    Azure support person (App Service) has responded.

    We decided to keep monitoring the issue. App Service team is not certain that the maxTelemetryItemsPerSecond is the root cause as the default for Azure functions runtime 2.x is 20, and my app is not busy enough to hit that limit or any default caps in App Insights.



    Tuesday, May 7, 2019 10:38 AM
  • Thank you again for providing these updates, Madushan.
    Thursday, May 16, 2019 6:45 AM
    Moderator
  • Update 4: 

    Azure support confirmed this is caused by a minor bug in the backend (AppService I assume.)

    Will close this issue here.

    Response from AppService/support team below.

    Our backend team managed to capture a minor bug in the platform that have caused this issue.

                                                                                                                

    For function apps in consumption plan, the function can be move to different instances from time to time. This bug prevent the function app to be fully registered after moving to new instance until one of the functions is triggered externally (with triggers such as http request).

    The backend team also provided a mitigation: if you find the logs are missing in Metrics, please visit the status page of the function app https://<APPNAME>.azurewebsites.net/. This inbound request should complete the register and mitigate the issue.

    Our backend team has prioritized this issue. We are sorry for the inconvenience caused. The fix should be released with the next platform update; until then, please mitigate the issue with the mitigation provided should the issue occurs again.


    Tuesday, July 30, 2019 9:37 AM
  • Madushan, thank you very much for the update and thank you again for improving Azure!
    Tuesday, July 30, 2019 9:19 PM
    Moderator