none
Azure Cloud Witness

    Question

  • Hello,

    I am trying to set up an Azure Cloud Witness to my 2-node cluster. The cluster has access to the internet through a proxy.

    I created my storage account in my Azure subscription and obtained a pair of keys. When entering the credentials in Failover Cluster Manager to set up the witness, I get:

    An error occurred while validating access to Azure from cluster node 'NODENAME'.

    Please check your storage account name, endpoint, and access key.

    Thanks,

    Alberto

    Tuesday, December 6, 2016 3:12 AM

Answers

  • Hello Alberto,

    According to my experience, this problem is caused by the missing of proxy settings in the PowerShell. You can execute the following commands in PowerShell, and then try set up Cloud Witness again.

    Run PowerShell as administrator, and execute the commands below.

    netsh winhttp import proxy source=ie

    If the proxy server needs authentication, please also execute the followings.

     $webclient=New-Object System.Net.WebClient
     $creds=Get-Credential
     $webclient.Proxy.Credentials=$creds


    Best Regards,
    Andy Liu

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    • Marked as answer by Alberto Hsu Monday, December 19, 2016 9:34 AM
    Tuesday, December 6, 2016 9:30 AM
  • Hi Alberto,

    I checked this with my internal team and it appears to be a proxy issue. Could you try (re)setting up the proxy using the below command.

    netsh winhttp set proxy proxy-server="https=SERVER:PORT"

     

    This will enable the cloud witness to work and be setup via WMI (locally or remotely) or remotely (using powershell & UI[from a node that has internet access or the proxy configured for the UI]).
    Ex: 
    $clus = Get-CimInstance  -ClassName MSCluster_ClusterService -Namespace "root\mscluster"
    Invoke-CimMethod -InputObject $clus -MethodName "CreateCloudWitness" -Arguments @{AccountKey="SOMEKEY"; AccountName = "SOMEACCOUNT"}

    If you also want to configure the settings to enable powershell or the UI to work, you need to enable the .net proxy settings. The easiest way to do this is by setting the proxy settings in IE.

    Please let us know how it goes.

    Regards.

    Md. Shihab

    • Marked as answer by Alberto Hsu Monday, December 19, 2016 9:34 AM
    Thursday, December 15, 2016 5:25 AM

All replies

  • Hello Alberto,

    Thanks for posting here!

    I attempted to set up Cloud Witness as the cluster quorum witness and was successfully able to do so without encountering any errors.

    The error you mentioned indicates that there could be some issue with the Storage Account Key. Please note as per this documentation we need to use the Primary Key when setting up the cluster witness the first time. If feasible, kindly try re-generating the Primary Key and configure the witness again and see if it works. Also, please make sure your endpoint is in the correct format. A typical endpoint looks like this “core.windows.net”.

    Alternatively, you could use the below PowerShell cmdlet to configure the Cloud Witness.

    Please feel free to get back to us if you have any further questions. We’d be happy to assist.

    Regards.

    Md. Shihab

    ***************************************************************************

    Please remember to click "Mark as Answer" on the post that helps you as this can be beneficial to other community members reading the thread. And vote as helpful.

    • Proposed as answer by Zirbesma Tuesday, October 2, 2018 2:24 PM
    Tuesday, December 6, 2016 8:07 AM
  • Hello Alberto,

    According to my experience, this problem is caused by the missing of proxy settings in the PowerShell. You can execute the following commands in PowerShell, and then try set up Cloud Witness again.

    Run PowerShell as administrator, and execute the commands below.

    netsh winhttp import proxy source=ie

    If the proxy server needs authentication, please also execute the followings.

     $webclient=New-Object System.Net.WebClient
     $creds=Get-Credential
     $webclient.Proxy.Credentials=$creds


    Best Regards,
    Andy Liu

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    • Marked as answer by Alberto Hsu Monday, December 19, 2016 9:34 AM
    Tuesday, December 6, 2016 9:30 AM
  • Hello Md. and Andy,

    Thanks for your help, I had followed the documentation in the MS site,

    I then ran the netsh proxy command, and I still get the same error.

    Is it related to the Azure Storage account? When creating the account, there are options (Resource Manager/Classic) deployment model, and (Standard/Premium) performance. Do these have effect on the storage account?

    Also, do I need to make any Storage account setting inside the Azure Portal after the account is created?

    i.e. make settings to the blog, container?

    Thanks a lot!

    Tuesday, December 6, 2016 10:44 AM
  • Hi Alberto,

    Microsoft recommendation in this regard is to use Resource Manager deployment and in the case of Cloud Witness the Standard blob storage account would do the job.

    Also, there are no separate settings needed within the portal. Please let us know if you have any further questions/clarifications.

    Regards.

    Md. Shihab

    Wednesday, December 7, 2016 2:52 AM
  • Hello Alberto,

    Please make sure you are using standard storage, NOT premium.

    Best Regards,

    Andy Liu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, December 7, 2016 5:36 AM
  • Thanks for your continued assistance. I am still not able to set up correctly. Below is the log I obtained from the Cluster.log file.

    Does this point to a internet connectivity/proxy error OR an Azure storage account error ?

    Thanks again!

    00001190.00001828::2016/12/12-17:52:59.299 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00001190.00001828::2016/12/12-17:54:34.768 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001190.00001828::2016/12/12-17:54:34.768 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001190.00001828::2016/12/12-17:54:34.768 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001190.00001828::2016/12/12-17:54:34.768 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001190.00001828::2016/12/12-17:54:34.769 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.
    000016d8.00001788::2016/12/12-17:55:19.728 INFO  [GUM] Node 2: Executing locally gumId: 274, updates: 1, first action: /dm/update
    000016d8.00001570::2016/12/12-17:55:59.839 INFO  [API] s_ApiUnblockGetNotifyCall: for the HDL( 5 )
    000016d8.0000162c::2016/12/12-17:56:24.508 INFO  [GUM] Node 2: Executing locally gumId: 275, updates: 1, first action: /dm/update
    00001190.000015ec::2016/12/12-17:56:57.867 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00001190.000015ec::2016/12/12-17:58:22.058 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001190.000015ec::2016/12/12-17:58:22.058 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001190.000015ec::2016/12/12-17:58:22.058 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001190.000015ec::2016/12/12-17:58:22.058 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001190.000015ec::2016/12/12-17:58:22.058 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.
    000016d8.0000162c::2016/12/12-18:00:19.747 INFO  [GUM] Node 2: Processing RequestLock 2:23
    000016d8.00001550::2016/12/12-18:00:19.748 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 275)
    000016d8.0000162c::2016/12/12-18:00:19.748 INFO  [GUM] Node 2: executing request locally, gumId:276, my action: /dm/update, # of updates: 1
    00001190.00001824::2016/12/12-18:02:38.911 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00001190.00001824::2016/12/12-18:04:14.641 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001190.00001824::2016/12/12-18:04:14.641 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001190.00001824::2016/12/12-18:04:14.641 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001190.00001824::2016/12/12-18:04:14.641 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001190.00001824::2016/12/12-18:04:14.641 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.
    00001190.00001824::2016/12/12-18:04:49.469 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00001190.00000ca0::2016/12/12-18:05:22.770 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00001190.00001824::2016/12/12-18:06:13.591 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001190.00001824::2016/12/12-18:06:13.591 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001190.00001824::2016/12/12-18:06:13.591 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001190.00001824::2016/12/12-18:06:13.591 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001190.00001824::2016/12/12-18:06:13.591 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.
    00001190.00000ca0::2016/12/12-18:06:46.872 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001190.00000ca0::2016/12/12-18:06:46.872 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001190.00000ca0::2016/12/12-18:06:46.872 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001190.00000ca0::2016/12/12-18:06:46.872 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001190.00000ca0::2016/12/12-18:06:46.872 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.

    Monday, December 12, 2016 6:36 AM
  • Hi Alberto,

    Just want to confirm if you were able to implement the below suggestion that was provided earlier.

    "If feasible, kindly try re-generating the Primary Key and configure the witness again and see if it works".

    I say this because the error message states "[RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359".

    Meanwhile, I will check this with my internal team and get back to you.

    Regards.

    Md. Shihab

    Monday, December 12, 2016 6:59 AM
  • Hi Md.,

    Yes, I had tried it the first time when you suggested it, and tried it again just now.

    Below is the log from the try I just performed regenerating the primary key:

    00000ba4.000012b8::2016/12/13-00:51:14.770 ERR   [API] ApipGetLocalCallerInfo: Error 3221356570 calling RpcBindingInqLocalClientPID.
    00001320.00000530::2016/12/13-00:51:17.049 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00000ba4.000012b8::2016/12/13-00:52:14.745 ERR   [API] ApipGetLocalCallerInfo: Error 3221356570 calling RpcBindingInqLocalClientPID.
    00001320.00000530::2016/12/13-00:52:41.076 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001320.00000530::2016/12/13-00:52:41.076 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001320.00000530::2016/12/13-00:52:41.076 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001320.00000530::2016/12/13-00:52:41.076 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001320.00000530::2016/12/13-00:52:41.077 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.
    00000ba4.00001a08::2016/12/13-00:53:14.771 ERR   [API] ApipGetLocalCallerInfo: Error 3221356570 calling RpcBindingInqLocalClientPID.
    00000ba4.00001a08::2016/12/13-00:54:00.975 INFO  [VSAM] BuildSpaceExtentListFromDiskOffsets(exit): total entries 0, 0 milliseconds
    00000ba4.00001a08::2016/12/13-00:54:00.975 INFO  [VSAM] RepairMissingExtents: total units to repair 0
    00000ba4.00001a08::2016/12/13-00:54:00.975 INFO  [VSAM] RepairMissingExtents(exit): time taken 0 milliseconds
    00001320.000018f0::2016/12/13-00:54:05.518 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    00000ba4.00001a08::2016/12/13-00:54:14.750 ERR   [API] ApipGetLocalCallerInfo: Error 3221356570 calling RpcBindingInqLocalClientPID.
    00000410.00001494::2016/12/13-00:54:18.251 INFO  [CAM] Substituting Token Owner: BUILTIN\Administrators, Original: NT AUTHORITY\SYSTEM
    00000410.00001494::2016/12/13-00:54:18.251 INFO  [CAM] Token Created, Client Handle: b64
    00000410.00001494::2016/12/13-00:54:19.815 INFO  [CAM] Substituting Token Owner: BUILTIN\Administrators, Original: NT AUTHORITY\SYSTEM
    00000410.00001494::2016/12/13-00:54:19.815 INFO  [CAM] Token Created, Client Handle: 12c0
    00000ba4.000012ac::2016/12/13-00:55:14.747 ERR   [API] ApipGetLocalCallerInfo: Error 3221356570 calling RpcBindingInqLocalClientPID.
    00001320.000018f0::2016/12/13-00:55:29.541 WARN  [RES] Cloud Witness: Calling AzureWrappers:: TestBlobExists returned an error Error in: WinHttpSendRequest with http status 0 and retry status 0
    00001320.000018f0::2016/12/13-00:55:29.541 ERR   [RES] Cloud Witness: Failed to validate Primary Key with errorCode 1359
    00001320.000018f0::2016/12/13-00:55:29.541 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 1359
    00001320.000018f0::2016/12/13-00:55:29.541 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    00001320.000018f0::2016/12/13-00:55:29.541 WARN  [RHS] Error 1359 from resource type control for restype Cloud Witness.

    Monday, December 12, 2016 10:00 AM
  • Hi Alberto,

    I checked this with my internal team and it appears to be a proxy issue. Could you try (re)setting up the proxy using the below command.

    netsh winhttp set proxy proxy-server="https=SERVER:PORT"

     

    This will enable the cloud witness to work and be setup via WMI (locally or remotely) or remotely (using powershell & UI[from a node that has internet access or the proxy configured for the UI]).
    Ex: 
    $clus = Get-CimInstance  -ClassName MSCluster_ClusterService -Namespace "root\mscluster"
    Invoke-CimMethod -InputObject $clus -MethodName "CreateCloudWitness" -Arguments @{AccountKey="SOMEKEY"; AccountName = "SOMEACCOUNT"}

    If you also want to configure the settings to enable powershell or the UI to work, you need to enable the .net proxy settings. The easiest way to do this is by setting the proxy settings in IE.

    Please let us know how it goes.

    Regards.

    Md. Shihab

    • Marked as answer by Alberto Hsu Monday, December 19, 2016 9:34 AM
    Thursday, December 15, 2016 5:25 AM

  • Hi Md.

    I used the proxy netsh setting, tried the WMI method (Get-CimInstance + Invoke-CimMethod)
    and I get the following output:

    ReturnValue PSComputerName
    ----------- --------------
             87


    However in FailoverCluster Manager I can't verify if the Azure Witness is set up correctly. There is no status.

    Friday, December 16, 2016 6:19 AM
  • I checked the cluster log and 87 corresponds to the new error code, as opposed to the previous error #1359

    000014ac.000010e8::2016/12/16-21:21:11.985 ERR   [RES] Cloud Witness: Caught exception Error in: WinHttpSendRequest while trying to validate new properties
    000014ac.000010e8::2016/12/16-21:21:11.985 ERR   [RES] Cloud Witness: Failed to validate new properties due to error code 87
    000014ac.000010e8::2016/12/16-21:21:11.985 INFO  [RES] Cloud Witness: Successfully validated parameters
    000014ac.000010e8::2016/12/16-21:21:11.985 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 87
    000014ac.000010e8::2016/12/16-21:21:11.985 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    000014ac.000010e8::2016/12/16-21:21:11.985 WARN  [RHS] Error 87 from resource type control for restype Cloud Witness.
    00000fa8.00001364::2016/12/16-21:21:42.061 INFO  [API] s_ApiUnblockGetNotifyCall: for the HDL( 7 )
    00000fa8.000017a0::2016/12/16-21:24:16.196 INFO  [GUM] Node 2: Executing locally gumId: 269, updates: 1, first action: /dm/update
    00000fa8.000017a0::2016/12/16-21:24:16.259 INFO  [GUM] Node 2: Processing RequestLock 2:21
    00000fa8.00001380::2016/12/16-21:24:16.292 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 269)
    00000fa8.000017a0::2016/12/16-21:24:16.292 INFO  [GUM] Node 2: executing request locally, gumId:270, my action: /dm/update, # of updates: 1
    00000fa8.000017a0::2016/12/16-21:24:23.665 INFO  [API] s_ApiGetQuorumResource final status 0.
    000014ac.000009e4::2016/12/16-21:24:23.666 ERR   [RES] Cloud Witness: Cloud witness type validating credentials
    000014ac.000009e4::2016/12/16-21:25:47.887 ERR   [RES] Cloud Witness: Caught exception Error in: WinHttpSendRequest while trying to validate new properties
    000014ac.000009e4::2016/12/16-21:25:47.887 ERR   [RES] Cloud Witness: Failed to validate new properties due to error code 87
    000014ac.000009e4::2016/12/16-21:25:47.887 INFO  [RES] Cloud Witness: Successfully validated parameters
    000014ac.000009e4::2016/12/16-21:25:47.887 ERR   [RES] Cloud Witness: Cloud witness type failed completed validation of credentials with error code 87
    000014ac.000009e4::2016/12/16-21:25:47.887 ERR   [RES] Cloud Witness: Cloud witness - Veriy connectivity, storage account name, and access key.
    000014ac.000009e4::2016/12/16-21:25:47.887 WARN  [RHS] Error 87 from resource type control for restype Cloud Witness.

    Friday, December 16, 2016 6:34 AM
  • I suggest you create a support ticket for this as it is likely going to require deeper technical analysis beyond the scope of forums.

    Regards.

    Md Shihab

    Monday, December 19, 2016 4:55 AM
  • Dear Md. and Andy,

    I appreciate your help greatly. I am now able to set up Cloud Witness correctly.

    It was indeed an issue with our proxy server. Thank you very much!

    Monday, December 19, 2016 9:34 AM
  • Glad to hear your issue is resolved, Alberto. 

    Regards.

    Md. Shihab

    Tuesday, December 20, 2016 5:53 AM
  • Hello Alberto,

    Glad to hear the issue is resolved.

    Regards,

    Andy Liu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, December 21, 2016 9:25 AM
  • Hi Md.,

    I have another question... I noticed that if I shutdown the 2 cluster nodes completely and Restart them, the Azure Cloud will be shown as "offline" in Failover Cluster Manager and I need to run the cmdlets again to make the Cloud Witness "Online" again. Is this normal behavior? is it related to the proxy issue as well?

    Thanks!

    Alberto.

    Thursday, January 12, 2017 9:57 AM
  • Hi Alberto,

    Could you check the cluster event logs to see if any related events were logged for the Cluster Witness going offline and not reconnecting again. Also, I'd recommend running the cluster validation set-up to ensure your cluster has been configured correctly.

    Regards.

    Md. Shihab

    Saturday, January 14, 2017 9:38 AM
  • regenerating the primary key worked for me. There must have been some problem with the original key.

    MZ

    Tuesday, October 2, 2018 2:24 PM