none
Function App stops working RRS feed

  • Question

  • A client contacted me telling me their data was not being updated. The data is updated by a Function App. I opened up the Azure portal and the dashboard showed the Function App as running. When I clicked on it to look into the issue the UI blade for the Function would not load for almost a minute. I clicked back to the Dashboard and tried again and this time it loaded. When I checked the monitor, and the log, there was nothing for the last 4 days.

    How can I find out what happened and why?

    How can I prevent it or at least get a notification that the Function App is not operating?

    Here is what the Application Events log shows:

    


    Tuesday, April 18, 2017 5:38 AM

All replies

  • Related twitter thread.

    https://twitter.com/drewmister_/status/854345047927644162

    Tuesday, April 18, 2017 3:59 PM
  • To investigate I need a few bits of info:

    • what type of triggers is your function app using
    • are there multiple functions that were failing to trigger or just a single function
    • UTC timestamps for when the function was failing to run
    • your function app name - see here for info on how to share privately
    Tuesday, April 18, 2017 4:20 PM
    1. App uses Timer, Http, and Queue Triggers
    2. Multiple functions
    3. Failing seemed to start before April 11. Today while troubleshooting I logged in around 5am UTC.
      - First log entry: 2017-04-18T05:15:00.087
      - Last log entry before stopping: 2017-04-18T06:30:02.990
      - Logged in to Portal later
      - Next log entry: 2017-04-18T14:20:02.663
    4. 2017-04-18T16:20:00.061 Function started (Id=5d42525a-d2fa-48d1-a41e-bcfbdbfa4163) in West US
    Tuesday, April 18, 2017 4:33 PM
  • Perfect, I'm taking a look now.
    Tuesday, April 18, 2017 4:37 PM
  • Thanks. Got to head out for the day. Hopefully this can get resolved soon.
    Tuesday, April 18, 2017 5:10 PM
  • Hmm, I'm not sure what is going on exactly. On my side it appears that the functions in your function app have been executing successfully for the past 10 days (I didn't go further back), with the exception of one function that has been failing consistently. I see 1200-1300 successful executions per day and 1 failure. The name of the function that is failing starts with "GetN". It appears to be failing due to a compilation error, so you should see that in your function logs when you open it in the portal and make a change (such as touching whitespace).

    You should see the same picture as I do if you scroll back through your webjobs dashboard logs. The webjobs dashboard is basically an expanded version of the "monitor" view you get in the functions portal. Access it like this:

    https://<<yourappname>>.scm.azurewebsites.net/azurejobs/#/functions

    Be aware that the log stream you see in the portal does not contain ALL executions. The log stream is used just for development/debugging and is only active while you have the Azure portal open. So looking at the log stream and seeing nothing for several days is not an indication that your function was not running.

    Another thing you could try - we have a much better monitoring story currently in preview. It involves direct integration with App Insights. Once you turn this integration on you can go and query your executions to see the same type of data I'm reporting here (number of success/failure per day, etc). There is info about the beta here:

    https://blogs.msdn.microsoft.com/appserviceteam/2017/04/06/azure-functions-application-insights/

    I'm happy to investigate further if you still think there is an issue. If you want to be able to discuss more specifics about your individual functions you can contact me directly at pbatum@microsoft.com

    Tuesday, April 18, 2017 6:37 PM
  • Thanks for the help.

    Yes, I do know about the one failing function. I have since disabled it so it should not be causing a problem.

    So, am I to understand that logs are only ever saved (anywhere) while the Portal is open? I was under the impression that the Portal may limit what logs you can see but if you use Kudu or something else you could still access the full logs.

    Apparently, you are correct that my Functions have been running. I did more checking and I am getting data from the function regularly, just not the data I was looking for. I am trying to troubleshoot why a particular set of data is not coming through. I have no control of when that data will come through so my only option was to search through the logs to find what was happening. When I saw limited logs I believed the Function was not running. Now, that I understand that the logging is not always running, finding what I need in the logs may be impossible.

    I will need to figure out another solution.

    Thanks 

    Wednesday, April 19, 2017 6:53 AM
  • Let me answer your specific question about when logs are saved. So the behavior is as follows:

    1. Execution logs are always written to your dashboard storage account. These logs are what powers the "Monitor" view for each individual function, and you can get an expanded version of that monitor view at the webjobs dashboard which I mentioned earlier.
    2. Whether File system logs are written is dependent on the tracing section of your host.json file. Here is an example:
    {
      "id": "2dee58ce5449432ca2302e2d45f5b267",
      "tracing": { "fileLoggingMode": "debugOnly" }
    }

    The "fileLoggingMode" defaults to "debugOnly". This means "when the portal is open". If you change this to "always", then file system logs are always written. Be aware that there is a performance cost to these logs so we generally do not recommend you set it to "always" for your production app. 

    When you open the development tab for a given function, its these file system logs that power the log streaming window. 

    This blog post provides a good summary of what I've discussed above:

    http://markheath.net/post/three-ways-view-error-logs-azure-functions

    Lastly, I'll plug our new App Insights support again :) Its by far the easiest way to get data on the execution of your functions. Considering that its in beta, you would want to enable it for a non-prod instance of your function app first. But I think you'll find it addresses your needs quite nicely.

    Wednesday, April 19, 2017 4:59 PM
  • I'll take a look at App Insights when it gets out of preview. Thanks
    Thursday, April 20, 2017 5:43 AM
  • Hey, did you get any answer for this? I'm struggling as well with functions stopping working suddenly
    Monday, April 24, 2017 3:00 PM
  • Luis, you may be running into this issue. i.e. the executions are in fact happening, but the log does not show them. See that thread for workaround.

    David

    Monday, April 24, 2017 6:53 PM
  • Hey,

    I have the feeling that my function isn't triggered as should. We have a process at night that puts quite a lot of data on a service bus topic in several messages (like a thousand+). This data should be processed when it is placed on the service bus. The next day when I get in the office and look at the service bus message count I see it is high and not dropping, this is for me an indication that the function isn’t triggered. The messages gets processed after I go to the function in the Azure portal, I see the status messages in the monitor view and also see the service bus message count dropping.

    -      Trigger: Service bus topic

    -      Single (first) function is failing

    -      I have two test runs created:

    • 2017-04-26 06:40:02 to 2017-04-26 06:40:36
    • 2017-04-26 06:55:03 to 2017-04-26 06:55:16

    -      2017-04-26T12:25:24.764 Function started (Id=18361d27-6bd7-4a17-b854-c9ee4fe22de0)

    Wednesday, April 26, 2017 12:33 PM
  • JorneS, please start a new thread. Your problem is unrelated to the issue discussed in this thread. Thanks.
    Wednesday, April 26, 2017 4:27 PM
  • JorneS, indeed, we can help you in a new thread. Please make sure to specify when your files were deployed, and using what techinque (e.g. Portal UI, git, VS, script, ...).
    Wednesday, April 26, 2017 7:22 PM
  • OK, thank you, I have started a new thread:

    https://social.msdn.microsoft.com/Forums/azure/en-US/1da75fde-1177-46b4-86ae-e127088eec51/service-bus-topic-stops-triggering-after-some-time?forum=AzureFunctions
    Friday, April 28, 2017 6:31 AM
  • This issue has returned. Previously the issue was related to a bug in how application settings were read from the function.json file. The bug was being address but in the meantime the suggested workaround was not to use an application setting as the value for 'disabled' in the function.json. I set 'disabled' to false and things have been working until just recently.

    Now, my timer triggers are not working again.

    What's the problem? Help needed.

    Wednesday, June 28, 2017 10:54 AM
  • Might be related to this error:

    Unable to acquire Singleton lock (xxxxxxxxxxxxxxxxx/Host.Functions.GetTankReadings.Listener)

    I pushed through an update to the host.json file and after that it looks like a lock was acquired. The timer triggers are functioning again but I really need to know what is happening and how to prevent it.

    Wednesday, June 28, 2017 11:14 AM
  • Hi Clarity,

    Please start a separate thread, since the cause this time appears to be unrelated to the original one.

    thanks,
    David

    Wednesday, June 28, 2017 2:08 PM
  • Done.

    https://social.msdn.microsoft.com/Forums/azure/en-US/12a7fb27-f2ec-4294-9b3d-9e690aaa7999/timer-trigger-stopped-working?forum=AzureFunctions

    Wednesday, June 28, 2017 2:15 PM