locked
waagent reporting "/opt/microsoft/omsagent/bin/service_control doesnot exist" same error over and over in waagent.log RRS feed

  • Question

  • My waagent.log file is currently over 800MB large. It is full of the same install or upgrade attempt and then error over and over again. How do I fix this azure service? My VM is Linux Ubuntu 14.04, Standard DS3. The error message from the log is below:

    [stderr]
    , duration=0
    2019/10/04 07:49:01.243633 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] Remove extension handler directory: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15
    2019/10/04 07:49:01.299042 INFO ExtHandler Deleted cgroup "/sys/fs/cgroup/cpu/walinuxagent.extensions/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.11.15".
    2019/10/04 07:49:01.312848 INFO ExtHandler Deleted cgroup "/sys/fs/cgroup/memory/walinuxagent.extensions/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.11.15".
    2019/10/04 07:49:04.441908 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] Target handler state: enabled
    2019/10/04 07:49:04.451328 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] [Enable] current handler state is: notinstalled
    2019/10/04 07:49:04.451894 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] Using existing extension package: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring__OmsAgentForLinux__1.11.15.zip
    2019/10/04 07:49:04.453877 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] Unzipping extension package: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring__OmsAgentForLinux__1.11.15.zip
    2019/10/04 07:49:05.173787 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] Initializing extension Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15
    2019/10/04 07:49:05.177439 INFO ExtHandler Created cgroup /sys/fs/cgroup/cpu/walinuxagent.extensions/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.11.15
    2019/10/04 07:49:05.179702 INFO ExtHandler Created cgroup /sys/fs/cgroup/memory/walinuxagent.extensions/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.11.15
    2019/10/04 07:49:05.181906 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.11.15] Update settings file: 0.settings
    2019/10/04 07:49:05.184382 INFO [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Disable extension [omsagent.py -disable]
    2019/10/04 07:49:05.221236 INFO ExtHandler Added PID 29580 to cgroup /sys/fs/cgroup/cpu/walinuxagent.extensions/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.7.9
    2019/10/04 07:49:05.223605 INFO ExtHandler Added PID 29580 to cgroup /sys/fs/cgroup/memory/walinuxagent.extensions/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux_1.7.9
    2019/10/04 07:49:05 OmsAgentForLinux started to handle.
    2019/10/04 07:49:05 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] cwd is /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9
    2019/10/04 07:49:05 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Change log file to /var/log/azure/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux/1.7.9/extension.log
    2019/10/04 07:49:07.260743 ERROR ExtHandler Event: name=Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux, op=Download, message=[ExtensionError] Non-zero exit code: 1, omsagent.py -disable
    [stdout]
    2019/10/04 07:49:05 [Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.7.9] Disable,failed,1,OMSAgent service control script /opt/microsoft/omsagent/bin/service_control doesnot exist. Disable cannot be called before install.

    Tuesday, October 22, 2019 2:09 PM

All replies

  • Looks like the OMS extension is failing. 

    I suggest simply removing the extension then letting it reapply itself. 

    To remove, go to the Portal -> Select your VM -> Select Extensions -> Select OMS Extension -> Remove

    Once it is removed I suggest clearing out the Waagent file to get some space back. 

    If you are having OMS installed via Azure Monitoring software it should be applied again automatically

    If not, you can manually reinstall the extension 

    https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/oms-linux

    Give that a go and let me know if it helps. 

    Tuesday, October 22, 2019 4:01 PM
  • I tried that but it fails to remove the agent and reports:

    Failed to delete the virtual machine extension 'OmsAgentForLinux'. Error: error

    On the extensions tab for that VM the status of the OmsAgentForLinux says "Transitioning". Is there a way I can stop the oms process and then remove it?

    Thursday, October 24, 2019 5:31 PM
  • Hmmm usually it will eventually either succeed or fail then you can attempt the delete again. 

    Can you email me at AzCommunity@microsoft.com and provide me with the following: 

    - SubscriptionID

    - VM Name

    - Resource Group Name

    I can take a look at the backend and see if I can unblock you. 

    Friday, October 25, 2019 7:47 PM
  • Thanks I just sent you an email.
    Friday, October 25, 2019 8:39 PM
  • Any update on this? 


    Thursday, November 7, 2019 5:24 PM
  • Worked with microsoft support over a period of about 6 weeks. They never really found the actual root cause of the problem and they still don't understand what happened. We managed to uninstall the oms agent so that the waagent.logs are not growing anymore, but the Azure portal dashboard was never fixed and still shows the agent as "Transitioning". They suggested I try just restarting the VM, but it's a production machine and I can't take it down just for some Azure issue that isn't currently affecting functionality, so I'm waiting for a regular maintenance window to restart it.

    While waiting for a maintenance window to do the restart they also sent me some instructions to run to try and uninstall the agent and re-install the agent to try and fix it's status in Azure portal dashboard. I ran the instructions on a development machine first to test them to make sure they were safe to run on prod and it broke the walinuxagent on that machine. I eventually managed to just uninstall the oms agent on that dev machine and fix it. The worst thing was microsoft support had no clue how to fix the issue they had caused and I had to figure it out myself. As a general warning to anyone who has issues with the oms agent in the future be very careful when running any instructions you get from Azure technical support. They seem to just be following scripts or googling stuff and don't really understand what they're doing, make sure you vet everything they send you first before running it on anything important.

    Monday, December 2, 2019 2:52 PM
  • Sorry to hear of that poor experience. Also sorry to hear you were not able to get a worth while solution. 

    If you like, you can share your ticket number with me and I can follow up with the feedback. 

    Monday, December 2, 2019 5:30 PM