locked
ASDK Appservice installation failed : VMExtensionProvisioningError RRS feed

  • Question

  • Hi 

    ASDK app service installation failed.Please help

    All the VM's except CN01 completed the deployment successfully.

    Deployment log shows:

    {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'OnStart'. Error message: Finished executing command\"\r\n }\r\n ]\r\n }\r\n}"}]}

    CN0-VM

    All the managed servers shows ready

    C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Status\0.status at CN0-VM shows

    .......

    -tenantId \"base64:cG9pbnRuZXh0Lm9ubWljcm9zb2Z0LmNvbQ==\" -ZipFiles Websites.cr","operation":"Command Execution Finished","status":"error","substatus":[{"code":0,"formattedMessage":{"lang":"en-US","message":":20 ---- Waiting for servers to be ready...\\n2017.08.23 04:36:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:37:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:38:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:39:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:40:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:41:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:42:20 ---- Waiting for servers to be ready...\\n2017.08.23 04:43:20 ---- Waiting for servers to be .......

    ready...\\n2017.08.23 05:40:22 ---- Waiting for servers to be ready...\\n2017.08.23 05:41:24 ---- Error while waiting for server to become ready. Exit code: 1."},"name":"StdOut","status":"success"},{"code":0,"formattedMessage":{"lang":"en-US","message":"Wait for servers has timed out\\nAt C:\\Packages\\Plugins\\Microsoft.Compute.CustomScriptExtension\\1.8\\Downloads\\0\\\\nCommon\\common.ps1:448 char:13\\n+             throw New-Object System.TimeoutException \"Wait for server ...\\n+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\n    + CategoryInfo          : OperationStopped: (:) [], TimeoutException\\n    + FullyQualifiedErrorId : Wait for servers has timed out\\n \\nC:\\Packages\\Plugins\\Microsoft.Compute.CustomScriptExtension\\1.8\\Downloads\\0\\OnS\\ntartAzureVirtualMachineController.ps1 : Error while waiting for server to \\nbecome ready. Exit code: 1.\\nAt line:1 char:1\\n+ .\\OnStartAzureVirtualMachineController.ps1 -controllerIndex 0 -feed h ...\\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\n    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorExcep \\n   tion\\n    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio \\n   n,OnStartAzureVirtualMachineController.ps1\\n "},"name":"StdErr","status":"success"}]},"timestampUTC":"\/Date(1503510089106)\/","version":"1"}]

    DB Access

    Installed SQL Studio in CN0-VM and confirmed that studio can access the SQL server hosting the App service databases

    Host Status

    CPU utilization is less than 15% and Memory utilization is less than 32%

    Host system config

    12x2 CPU

    336 GB RAM

    200 GB+4*600 GB  Disks


    Wednesday, August 23, 2017 11:50 PM

Answers

  • If you see your roles - controller and management are ready in Web Cloud Management Console on CN0-VM then  it is timeout issue mostly because of Windows Updates. Try to refresh your Windows server 2016 image from gallery to have all or most recent updates pre-installed. This will reduce deployment time and will avoid timeout. 

    For current installation - if your controller and management roles are in ready state, You can do following as work around.

    1. Do not close/exit  App Service installer. Keep it open at retry step.
    2. RDP to CN VM
    3. Ensure Controller and Management roles are in ready state (rest of roles can be in other state)

    This is to ensure, setup is failed because of timeout issue.

    1. Go to “C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Status” and replace statue file with the one with successful status.

    This is to manually update status to successful for VM custom extension scripts

    1. Now comeback to App Service installer and hit retry. This should move forward and execute remaining steps.
    Thursday, August 24, 2017 4:59 PM
  • Sample 0.status file. Replace the content of this file with the one below, which indicates the successful execution of VM extension script and can now move to next steps.

    [{"status":{"code":0,"formattedMessage":{"lang":"en-US","message":"Finished executing command"},"name":"powershell.exe -ExecutionPolicy Unrestricted -Command .\\OnStartAzureVirtualMachineController.ps1","operation":"Command Execution Finished","status":"success","substatus":[{"code":0,"formattedMessage":{"lang":"en-US","message":"Waiting for servers to be ready...\\n2017.05.26 04:59:26 ---- Servers are ready!\\n2017.05.26 04:59:27 ---- All servers are ready!\\n2017.05.26 04:59:27 ---- App Service controller installation and configuration has completed successfully."},"name":"StdOut","status":"success"},{"code":0,"formattedMessage":{"lang":"en-US","message":""},"name":"StdErr","status":"success"}]},"timestampUTC":"\/Date(1495774767271)\/","version":"1"}]

    Thursday, August 24, 2017 5:00 PM

All replies

  • Since you are getting a timeout, can I ask where you got your Windows Server image? We have seen issues with timeouts since app service will install updates, and these have gotten progressively larger (and slower).
    Thursday, August 24, 2017 4:30 PM
  • If you see your roles - controller and management are ready in Web Cloud Management Console on CN0-VM then  it is timeout issue mostly because of Windows Updates. Try to refresh your Windows server 2016 image from gallery to have all or most recent updates pre-installed. This will reduce deployment time and will avoid timeout. 

    For current installation - if your controller and management roles are in ready state, You can do following as work around.

    1. Do not close/exit  App Service installer. Keep it open at retry step.
    2. RDP to CN VM
    3. Ensure Controller and Management roles are in ready state (rest of roles can be in other state)

    This is to ensure, setup is failed because of timeout issue.

    1. Go to “C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Status” and replace statue file with the one with successful status.

    This is to manually update status to successful for VM custom extension scripts

    1. Now comeback to App Service installer and hit retry. This should move forward and execute remaining steps.
    Thursday, August 24, 2017 4:59 PM
  • Sample 0.status file. Replace the content of this file with the one below, which indicates the successful execution of VM extension script and can now move to next steps.

    [{"status":{"code":0,"formattedMessage":{"lang":"en-US","message":"Finished executing command"},"name":"powershell.exe -ExecutionPolicy Unrestricted -Command .\\OnStartAzureVirtualMachineController.ps1","operation":"Command Execution Finished","status":"success","substatus":[{"code":0,"formattedMessage":{"lang":"en-US","message":"Waiting for servers to be ready...\\n2017.05.26 04:59:26 ---- Servers are ready!\\n2017.05.26 04:59:27 ---- All servers are ready!\\n2017.05.26 04:59:27 ---- App Service controller installation and configuration has completed successfully."},"name":"StdOut","status":"success"},{"code":0,"formattedMessage":{"lang":"en-US","message":""},"name":"StdErr","status":"success"}]},"timestampUTC":"\/Date(1495774767271)\/","version":"1"}]

    Thursday, August 24, 2017 5:00 PM
  • Hi Alok,

    I t worked.

    Completed the app service install.

    But got into another issue during create-identit-Apps script.

    verified that its not a certificate and password issue.I dont think its related to my earlier issue.But just want to let you know.If you have come across anything similar please let me know

    Thank you for your help.

    PS D:\bkp\AppServiceHelperScripts> .\Create-IdentityApp.ps1

    cmdlet Create-IdentityApp.ps1 at command pipeline position 1
    Supply values for the following parameters:
    (Type !? for Help.)
    DirectoryTenantName: $$$$$$$$$$$$$$$$$$$$$
    AzureStackCredential
    CertificateFilePath: ".\sso.appservice.local.azurestack.external.pfx"
    CertificatePassword: "$$$$$$$$$$$$$$"
    TenantArmEndpoint: management.local.azurestack.external
    DomainName: local.azurestack.external
    VERBOSE: Resource Provider FQDN resolved to: 'sso.appservice.pointnext.onmicrosoft.com'
    VERBOSE: GET https://management.local.azurestack.external/metadata/endpoints?api-version=1.0 with 0-byte payload
    VERBOSE: received 337-byte response of content type application/json; charset=utf-8
    VERBOSE: ARM resource Uri: 'https://management.pointnext.onmicrosoft.com/36e1e6f0-3e81-4bb6-bdb5-091ab8b38622'
    36e1e6f0-3e81-4bb6-bdb5-091ab8b38622
    New-Object : Exception calling ".ctor" with "3" argument(s): "Illegal characters in path."
    At D:\bkp\AppServiceHelperScripts\Create-IdentityApp.ps1:86 char:24
    + ... rtificate = New-Object System.Security.Cryptography.X509Certificates. ...
    +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidOperation: (:) [New-Object], MethodInvocationException
        + FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand


    • Edited by johnpaultthomas Saturday, August 26, 2017 12:02 PM making it more correct
    Saturday, August 26, 2017 11:59 AM
  • I am still using the 2016 ISO to create the default image that I downloaded during the TP2 release 

    How can I update the VM Image that I created using this ISO?


    johnpaul

    Saturday, August 26, 2017 12:05 PM
  • There are a few methods you can use.

    1) You could build a new VHD using the ISO and the latest rollup release. Note that the New-AzsServer2016VMImage has a -IncludeLatestCU  but I strongly recommend you not use that. The CU is hard-coded in the script and is old - it also will fail now. So, you can update the script to use the real latest one:

    https://support.microsoft.com/en-us/help/4000825/windows-10-windows-server-2016-update-history

    Then you click on the topmost version on the left under 1607, and when you get to that page, scroll down “How to get this update” which gives you a link to the Update Catalog for this item. You can also provide the URI as a parameter for the -ManulCUUri option once you determine it (there is also a -ManualCUPath option if you have downloaded the msu file). Today the URI is 

    http://download.windowsupdate.com/d/msdownload/update/software/updt/2017/08/windows10.0-kb4034661-x64_f776ef2c5c18168cb0e7a1b2d69d7e785cdcbf51.msu

    Note that it updates twice a month: 2nd Tuesday is "Patch Tuesday" and includes new security updates and features from the past month; 3rd Tuesday includes everything from Patch Tuesday and some new features.

    2) You can perform offline servicing of the VHD you already have a copy. You can follow the logic in the PSM1 module, or use the dism command if you are familiar with offline servicing. You will have to remove the existing image or create a new version.

    3) Again, if you have the VHD, you can copy it to a system, use it for a New VM using Hyper-V Manager, start it up and use Windows Update to download/apply whatever packages you need (assuming it has network access). Then sysprep, shut down, convert the disk to a fixed gen 1 and use that for your new VHD. You will have to remove the existing image or create a new version.

    4) Use marketplace syndication to d/l the latest version. The image there is about 1 month old and will be updated to August Patch Tuesday sometime this week. That image has the patches available as of August 8th, plus .NET 3.5 and .NET 4.7. Using that image makes App Service install much faster! When you download the image from syndication, it will be used in future VM deployments as it's version number will be greater than what you probably supplied already. You can delete the existing 2016 image before downloading if you don't need it (e.g., not used with an existing scale set).


    Saturday, August 26, 2017 2:39 PM
  • Thank you Jeff

    johnpaul

    Sunday, August 27, 2017 8:55 AM
  • Interesting.

    Can you try passing the arguments to powershell directly (or provide full SSO cert path)

    Try below command for example  -

    .\Create-IdentityApp.ps1 -DirectoryTenantName 'test1.onmicrosoft.com' -CertificateFilePath "C:\AppServiceRP\AppServiceHelperScripts\sso.appservice.local.azurestack.external.pfx" -TenantArmEndpoint 'management.local.azurestack.external' -CertificatePassword 'xxxxxx' -DomainName 'local.azurestack.external'


    Monday, August 28, 2017 5:08 PM
  • Passing arguments to power shell directly worked worked without any error!

    Ran the script,updated the secret key from azure portal and completed the deployment successfully

    all the managed server roles are ready.

    but while trying to create web app it shows

    Oops!

    Could not create Web App (preview)

    Something went wrong while creating Web App (preview).

    Tried restarting and repairing the managed server roles without any effect.Do you think any of the work around we did missed any thing


    johnpaul

    Wednesday, August 30, 2017 1:40 AM
  • Just look here: https://rzetelnekursy.pl/azure-stack-web-mobile-web-app-oops-could-not-create-web-app/

    mafamafamafa

    Sunday, December 30, 2018 10:44 AM
  • https://rzetelnekursy.pl/azure-stack-app-service-install-problems/ - Step By Step solution based on answers bellow.

    mafamafamafa

    Sunday, December 30, 2018 10:45 AM