locked
WebJobs not stopping at the same time RRS feed

  • Question

  • I have a single WebApp App Service with seven continuous WebJobs running. I have also setup the stopping wait time to two minutes { "stopping_wait_time": 120 } to allow for graceful shutdowns. When I stop the WebJobs, either through the portal or the kudu api, they stop in order but one WebJob must finish stopping before the next WebJob starts the stopping process.

    WebJob 1 immediately goes into a “stopping” status for two minutes then switches to the “stop” status. WebJob 2 sits in a “disabling” state for that two minutes until WebJob 1 is in the “stop” status. WebJob 2 then enters a “stopping” status for two minutes and stops. This means WebJob 7 sits in a “disabling” status for twelve minutes before it enters the “stopping” status.

    I want all the WebJobs to shut down in two minutes, not a staggering process that takes fourteen minutes. Is there any way to do this?

    Thanks, Patrick

    Thursday, February 1, 2018 6:06 PM

All replies

  • It is the current limitation that we are using single thread to go thru all job state transitions.   The workaround is for job writer to handle shutdown event in the timely manner.   Please see this graceful shutdown.   In short, job implementation should look for shutdown periodically and perform shutdown if present.

    Suwatch

    • Proposed as answer by Swikruti Bose Friday, February 2, 2018 7:46 AM
    Friday, February 2, 2018 7:21 AM
  • Thanks Suwatch,

    I am actively looking if the %WEBJOBS_SHUTDOWN_FILE% file exists. When viewing the file system via the App Service Editor I see that the file is created in the WebJob as soon as I click stop. However, File.Exists returns false until the job is in a “stopping” status.

    In the below output, I stop the job at 02/02/2018 20:09:55, where the job says it is disabling. Now the shutdown file exists. The informational messages (redacted) at 02/02/2018 20:11:35 are output from a process that is calling the File.Exists method, but returns false and continues to process. At 02/02/2018 20:22:01 the job enters a stopping status and the process detects the shutdown file and exits.

    Console output:

    App Service Editor:

    Friday, February 2, 2018 8:29 PM
  • This is puzzling.  If you can see the file from App Service Editor, File.Exists should return TRUE.   Are we sure we are looking at the same file?   BTW, the shutdown file would not be disable.job but some file with random name.

    Suwatch

    Friday, February 2, 2018 8:40 PM
  • If the shutdown file is not disable.job then I never see it in App Service Editor, regardless of the WebJob status. I only ever see the disable.job file created when I press the stop button and then it is deleted when I press the start button. I added some extra informational console messages to express what I see when remote debugging.

    The file name is something random but File.Exists is False when the WebJob is disabling. And does not return true until the WebJob enters a “stopping” status.

    Friday, February 2, 2018 9:49 PM
  • They're definitely two different files, so please ignore disable.job for this discussion. So you are seeing this file appear when the WebJob enters the stopping status. That is the correct behavior. You need to detect the appearance of the file and initiate your shutdown.

    David

    Wednesday, February 7, 2018 9:13 PM