locked
Webjob with "InactiveInstance" state RRS feed

  • Question

  • This morning I find one of my continuous webjobs WJOB_ADHB01 stuck in "InactiveInstance" state as shown below

    I've tried to "start" the webjob (only option available), nothing happens

    I've restarted the Web site, which seems to stop & restart the other two webjobs but not this one

    I've tried stopping & restarting the web site manually, but still to no avail

    Some debug info and pic below

    Can someone shed some light on this please..?

    WJOB_ADHB01.ps1
    
    Final few lines of job_log.txt 
    ==============================
    [05/31/2016 00:01:59 > a09d3a: INFO] tock 196 2016-05-31 00:01:58 UTC 0 HUB msgs sent 0.000 TotalSeconds
    [05/31/2016 00:02:03 > a09d3a: WARN] Reached maximum allowed output lines for this run, to see all of the job's logs you can enable website application diagnostics
    [05/31/2016 00:45:25 > a09d3a: SYS INFO] WebJob is still running
    [05/31/2016 08:32:00 > a09d3a: SYS INFO] Status changed to Starting
    [05/31/2016 08:32:48 > f400f8: SYS INFO] Status changed to Starting
    [05/31/2016 08:32:49 > f400f8: SYS INFO] Status changed to InactiveInstance
    [05/31/2016 08:34:45 > a09d3a: SYS INFO] Status changed to InactiveInstance
    
    
    Contents of singleton.job.lock
    ==============================
    {
      "OperationName": "Acquiring continuous WebJob singleton lock",
      "AcquiredDateTime": "2016-05-30T23:45:25.6637878Z",
      "StackTrace": "   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)\r\n   at System.Environment.get_StackTrace()\r\n   at Kudu.Core.Infrastructure.LockFile.WriteLockInfo(String operationName, Stream lockStream)\r\n   at Kudu.Core.Infrastructure.LockFile.Lock(String operationName)\r\n   at Kudu.Core.Jobs.ContinuousJobRunner.TryGetLockIfSingleton()\r\n   at Kudu.Core.Jobs.ContinuousJobRunner.<>c__DisplayClass19_0.<StartJob>b__0()\r\n   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)\r\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Threading.ThreadHelper.ThreadStart()"
    } 
    
    Contents of status_a09d3a 
    =========================
    {"Status":"InactiveInstance"}



    Simon


    • Edited by Msgwrx Tuesday, May 31, 2016 10:06 PM
    Tuesday, May 31, 2016 10:05 PM

Answers

  • I think you're hitting a known Portal issue. The status in the portal reflects the state of one (arbitrary) instance. So if your site is scaled out and your continuous job is marked as singleton, it will be inactive on all but one instance. If the portal is hitting any of the inactive instances, it will report that status (likelihood increases as instance count increases).

    But in spite of that, the WebJob is likely running fine. To see this, from Portal go to Process Explorer (under Tools). It'll list all processes for all instances, and you should see one instance running the webjob.

    David



    • Proposed as answer by David Ebbo Tuesday, May 31, 2016 10:46 PM
    • Edited by David Ebbo Tuesday, May 31, 2016 10:47 PM
    • Marked as answer by Msgwrx Wednesday, June 1, 2016 1:06 AM
    Tuesday, May 31, 2016 10:46 PM
  • Simon, note that Kudu explorer (on the scm site) has the same issue as it only hits one instance (there is an advanced way to pick which one). On the other hand, the Process Explorer in the Portal (under Tools) shows you all the instances, which will make things a lot clearer.

    David

    • Marked as answer by Msgwrx Wednesday, June 1, 2016 6:29 AM
    Wednesday, June 1, 2016 5:03 AM

All replies

  • Oh, amazing... the webjob just started working again all by itself..!

    I'm none the wiser on what "Inactive Instance" means


    Cheers Simon

    • Edited by Msgwrx Tuesday, May 31, 2016 10:15 PM
    Tuesday, May 31, 2016 10:14 PM
  • I think you're hitting a known Portal issue. The status in the portal reflects the state of one (arbitrary) instance. So if your site is scaled out and your continuous job is marked as singleton, it will be inactive on all but one instance. If the portal is hitting any of the inactive instances, it will report that status (likelihood increases as instance count increases).

    But in spite of that, the WebJob is likely running fine. To see this, from Portal go to Process Explorer (under Tools). It'll list all processes for all instances, and you should see one instance running the webjob.

    David



    • Proposed as answer by David Ebbo Tuesday, May 31, 2016 10:46 PM
    • Edited by David Ebbo Tuesday, May 31, 2016 10:47 PM
    • Marked as answer by Msgwrx Wednesday, June 1, 2016 1:06 AM
    Tuesday, May 31, 2016 10:46 PM
  • Thanks David

    I did look in KUDU Process Explorer for the WJOB_ADHB01webjob thinking I could maybe "kill" it from there, but it wasn't showing or working although the other two webjobs were.

    I think there are other factors in the Sydney DC that are affecting this, we've had a lot of problems this month


    Simon

    • Edited by Msgwrx Wednesday, June 1, 2016 1:06 AM
    Wednesday, June 1, 2016 1:06 AM
  • Simon, note that Kudu explorer (on the scm site) has the same issue as it only hits one instance (there is an advanced way to pick which one). On the other hand, the Process Explorer in the Portal (under Tools) shows you all the instances, which will make things a lot clearer.

    David

    • Marked as answer by Msgwrx Wednesday, June 1, 2016 6:29 AM
    Wednesday, June 1, 2016 5:03 AM
  • Ok cool, thanks for the tip David

    Simon 

    • Edited by Msgwrx Wednesday, June 1, 2016 6:29 AM
    Wednesday, June 1, 2016 6:29 AM