none
当用powershell创建HDInsight Cluster时,使HDInsight使用之前创建好的存储,报错。 RRS feed

  • 问题

  • 使用powershell创建了两个 storageAccount ,一个做HDIisght的默认存储,另一个做hive的udf的存储。然后将默认的存储配置到AzureHDInsightDefaultStorage 。在用这个AzureHDInsightDefaultStorage 新建HDInsight cluster时,会报 <Message>UnableToVerifyLocationOfResource</Message>的错误。

    我的powershell脚本:

    #1.Name HDInsight and Storage & container #2.Create A Storage Acount with Two Container,One Works as Primary Storage for HDInsight, Another Works as UDF Storage for Hive #3.Upload UDF to UDF Storage, Hive Script & Mahout-core-job.jar to Primary Storage #4.Make HDInsight Configure and Create One param( [string]$SubscriptionName="Visual Studio Ultimate with MSDN", # Replace <Your Subscription Name> with your Azure subscription name [string]$HDInsightClusterLocation="West US", # This is the data center where you are going to provision this HDInsight cluster [string]$HDInsightClusterName="myclusterName", # This is the name that you want for the HDInsight cluster [int]$NumNodesInCluster = 1, # This is the number of data nodes that you want for this cluster # The storage accounts and containers need to be created on the same data center as your HDInsight cluster prior to the execution of this PS script [string]$PrimaryStorageAccount="myprimary", # This is the name of your primary storage account [string]$PrimaryStorageContainer="install", # This is the name of the primary storage container [string]$HiveLibStorageAccount="myhivestoragesea", # This is the name of the additional WASB storage account that will hold the Hive libs. [string]$HiveLibStorageContainer="hivecustomlibs", # This is the name of the default storage container for Hive libs. # This is the SQL Azure Sever name that will host the metastore databases, this needs to be created on the same data center as your cluster prior to executing this PS script [string]$SQLAzureServerName = "myseverName", # This is the SQL Azure Database that will host the Hive Metastore DB. Again, this needs to be created on the SQL Azure server above prior to executing this PS script. [string]$HiveMetaStoreDB = "mydbstorage" ) Select-AzureSubscription -SubscriptionName $SubscriptionName ############################################################################ ##New Storage Account And Container ############################################################################ Write-Verbose "Creating new storage account $PrimaryStorageAccount." $PrimaryStorage = New-AzureStorageAccount –StorageAccountName $PrimaryStorageAccount -Location $HDInsightClusterLocation -Verbose Write-Verbose "Creating new storage account $HiveLibStorageAccount." $HiveStorage = New-AzureStorageAccount –StorageAccountName $HiveLibStorageAccount -Location $HDInsightClusterLocation -Verbose ############################################################################ ##Generate Storage Account Keys ############################################################################ $PrimaryStorageAccountKey = Get-AzureStorageKey $PrimaryStorageAccount | %{ $_.Primary} $HiveLibStorageAccountKey = Get-AzureStorageKey $HiveLibStorageAccount | %{ $_.Primary} $PrimaryStorageContext = New-AzureStorageContext –StorageAccountName $PrimaryStorageAccount -StorageAccountKey $PrimaryStorageAccountKey #(Get-AzureStorageKey $PrimaryStorageAccount).Primary $HiveStorageContext = New-AzureStorageContext –StorageAccountName $HiveLibStorageAccount -StorageAccountKey $HiveLibStorageAccountKey #(Get-AzureStorageKey $HiveLibStorageAccount).Primary Write-Verbose "Creating new storage container $PrimaryStorageContainer." New-AzureStorageContainer -Name $PrimaryStorageContainer -Context $PrimaryStorageContext Write-Verbose "Creating new storage container $HiveLibStorageContainer." New-AzureStorageContainer -Name $HiveLibStorageContainer -Context $HiveStorageContext ############################################################################ ##End Create Storage Account And Container ############################################################################ #$context = New-AzureStorageContext -StorageAccountName $PrimaryStorageAccount -StorageAccountKey $PrimaryStorageAccountKey #Set-AzureStorageBlobContent -Blob "hive_udf.jar" -Container $HiveLibStorageContainer -File "d:\azure_stack\hive_udf.jar" -Context $HiveStorageContext -Force ls "d:\azure_stack\file" -File -Recurse | Set-AzureStorageBlobContent -Container $HiveLibStorageContainer -Context $HiveStorageContext -Force ls "d:\azure_stack\hive_scripts" -File -Recurse | Set-AzureStorageBlobContent -Container $PrimaryStorageContainer -Context $PrimaryStorageContext -Force $PrimarySubscriptionID = (Get-AzureSubscription $SubscriptionName).SubscriptionId # This will be the credential used for the admin account of the HDInsight Cluster $password = "mypassword" | ConvertTo-SecureString -asPlainText -Force $username = "myusername" $creds = New-Object System.Management.Automation.PSCredential($username,$password) # This will be the credential account used for accessing your SQL Azure Hive MetaStore Databases $password = "mypassword" | ConvertTo-SecureString -asPlainText -Force $username = "myusername" $MetaStoreCreds = New-Object System.Management.Automation.PSCredential($username,$password) ######################################################### ### This section adds the new JAR to the Hive library ### ######################################################### $configvalues = new-object 'Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.DataObjects.AzureHDInsightHiveConfiguration' $configvalues.AdditionalLibraries = new-object 'Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.DataObjects.AzureHDInsightDefaultStorageAccount' $configvalues.AdditionalLibraries.StorageAccountName = "$HiveLibStorageAccount.blob.core.windows.net" $configvalues.AdditionalLibraries.StorageAccountKey = $HiveLibStorageAccountKey $configvalues.AdditionalLibraries.StorageContainerName = $HiveLibStorageContainer $NumNodesInCluster = '1' New-AzureHDInsightClusterConfig -ClusterSizeInNodes $NumNodesInCluster ` | Set-AzureHDInsightDefaultStorage -StorageAccountName "$PrimaryStorageAccount.blob.core.windows.net" -StorageAccountKey $PrimaryStorageAccountKey -StorageContainerName $PrimaryStorageContainer ` | Add-AzureHDInsightMetastore -SqlAzureServerName $SQLAzureServerName -DatabaseName $HiveMetaStoreDB -Credential $MetaStoreCreds -MetastoreType HiveMetastore ` | Add-AzureHDInsightConfigValues -Hive $configvalues ` | New-AzureHDInsightCluster -Subscription $PrimarySubscriptionID -Credential $creds -Name $HDInsightClusterName -Location $HDInsightClusterLocation -debug 我的error LOG是

    调试: 10:46:04 - NewAzureHDInsightClusterCmdlet begin processing with ParameterSet 'Cluster By Config (with Specific
    Subscription Credential)'.
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    警告: The -Subscription parameter is deprecated, Please use Select-AzureSubscription -Current to select a subscription
    to use.
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    调试: Severity: Informational
    Create Cluster Started : 2014/5/5 10:46:06
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    调试: Severity: Informational
      Type: HDInsightClient
    Member: MoveNext
       UTC: 2014-05-05T02:46:06.8018052Z
    Validating Cluster Versions
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    调试: Severity: Informational
      Type: HDInsightClient
    Member: MoveNext
       UTC: 2014-05-05T02:46:06.8048054Z
    Sending Cluster Create Request
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    调试: Severity: Informational
      Type: HDInsightManagementPocoClient
    Member: MoveNext
       UTC: 2014-05-05T02:46:06.8078055Z
    Create Cluster Requested
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    调试: Severity: Informational
      Type: HttpClientAbstraction
    Member: LogRequestResponseDetails
       UTC: 2014-05-05T02:46:42.4628449Z
     HTTP Request:
      SDK Version: 1.1.0.7
              Uri:
    https://management.core.windows.net/48e7656d-0790-4af6-ac68-841968be9abb/resourceproviders/hdinsight/Properties?resourc
    eType=containers
           Method: GET
     Content Type: application/xml
          Headers: x-ms-version = 2012-08-01
                   accept = application/xml
    
          Content:
    
    
    HTTP Response:
      Status Code: OK
          Headers: x-ms-servedbyregion = ussouth2
                   x-ms-request-id = 1e7643cb5ded943eb82dd9dc48784dcd
                   Cache-Control = no-cache
                   Date = Mon, 05 May 2014 02:46:41 GMT
                   Server = 1.0.6198.70
                            (rd_rdfe_stable.140426-2318)
                            Microsoft-HTTPAPI/2.0
    
          Content:
    <?xml version="1.0" encoding="utf-8"?>
    <ResourceProviderProperties xmlns="http://schemas.microsoft.com/windowsazure"
    xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
      <ResourceProviderProperty>
        <Key>CAPABILITY_FEATURE_CUSTOM_ACTIONS_V2</Key>
        <Value>Custom Actions Feature V2</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_REGION_EAST_US</Key>
        <Value>East US</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_REGION_NORTH_EUROPE</Key>
        <Value>North Europe</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_REGION_SOUTHEAST_ASIA</Key>
        <Value>Southeast Asia</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_REGION_WEST_EUROPE</Key>
        <Value>West Europe</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_REGION_WEST_US</Key>
        <Value>West US</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_VERSION_1.6</Key>
        <Value>HdInsight version 1.6</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_VERSION_2.1</Key>
        <Value>HdInsight version 2.1</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CAPABILITY_VERSION_3.0</Key>
        <Value>Version 3.0.0.0.497904</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CONTAINERS_Count</Key>
        <Value>0</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CONTAINERS_CoresUsed</Key>
        <Value>0</Value>
      </ResourceProviderProperty>
      <ResourceProviderProperty>
        <Key>CONTAINERS_MaxCoresAllowed</Key>
        <Value>170</Value>
      </ResourceProviderProperty>
    </ResourceProviderProperties>
    
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”):
    调试: Severity: Informational
      Type: HttpClientAbstraction
    Member: LogRequestResponseDetails
       UTC: 2014-05-05T02:47:01.7889503Z
     HTTP Request:
      SDK Version: 1.1.0.7
              Uri:
    https://management.core.windows.net/48e7656d-0790-4af6-ac68-841968be9abb/cloudservices/hdinsightZ5I5DWQOTDYUYQCNJBAT5OI
    KQ55NKWUM3VVYA5MLGDB35JB2EA2Q-Southeast-Asia
           Method: GET
     Content Type: application/xml
          Headers: x-ms-version = 2012-08-01
                   accept = application/xml
    
          Content:
    
    
    HTTP Response:
      Status Code: OK
          Headers: x-ms-servedbyregion = ussouth2
                   x-ms-request-id = b7c95a849468954b98363ad37fb74401
                   Cache-Control = no-cache
                   Date = Mon, 05 May 2014 02:47:00 GMT
                   ETag = 7cdf6b92-a8f0-4cca-bd5a-2f75b130090b
                   Server = 1.0.6198.70
                            (rd_rdfe_stable.140426-2318)
                            Microsoft-HTTPAPI/2.0
    
          Content:
    <?xml version="1.0" encoding="utf-8"?>
    <CloudService xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
      <Name>hdinsightZ5I5DWQOTDYUYQCNJBAT5OIKQ55NKWUM3VVYA5MLGDB35JB2EA2Q-Southeast-Asia</Name>
      <Label>HdInsight CloudService</Label>
      <Description>HdInsight clusters for subscription 48e7656d-0790-4af6-ac68-841968be9abb</Description>
      <GeoRegion>Southeast Asia</GeoRegion>
      <Resources />
    </CloudService>
    
    
    确认
    是否继续执行此操作?
    [Y] 是(Y)  [A] 全是(A)  [H] 终止命令(H)  [S] 挂起(S)  [?] 帮助 (默认值为“Y”): A
    调试: Severity: Informational
      Type: HttpClientAbstraction
    Member: LogRequestResponseDetails
       UTC: 2014-05-05T02:47:23.2181760Z
     HTTP Request:
      SDK Version: 1.1.0.7
              Uri:
    https://management.core.windows.net/48e7656d-0790-4af6-ac68-841968be9abb/cloudservices/hdinsightZ5I5DWQOTDYUYQCNJBAT5OI
    KQ55NKWUM3VVYA5MLGDB35JB2EA2Q-Southeast-Asia/resources/hdinsight/containers/zdwechatclustersea
           Method: PUT
     Content Type: application/xml
          Headers: x-ms-version = 2012-08-01
                   schemaversion = 2.0
                   accept = application/xml
    
          Content:
    <?xml version="1.0" encoding="utf-8"?>
    <Resource xmlns="http://schemas.microsoft.com/windowsazure">
      <SchemaVersion>2.0</SchemaVersion>
      <IntrinsicSettings>
        <ClusterContainer xmlns="http://schemas.microsoft.com/hdinsight/2013/05/management">
          <Deployment>
            <AdditionalStorageAccounts />
            <ClusterPassword>triceedInfo*2014</ClusterPassword>
            <ClusterUsername>wechatuser</ClusterUsername>
            <Roles>
              <ClusterRole>
                <Count>1</Count>
                <RoleType>HeadNode</RoleType>
                <VMSize>ExtraLarge</VMSize>
              </ClusterRole>
              <ClusterRole>
                <Count>1</Count>
                <RoleType>DataNode</RoleType>
                <VMSize>Large</VMSize>
              </ClusterRole>
            </Roles>
            <Version>default</Version>
          </Deployment>
          <DeploymentAction>Create</DeploymentAction>
          <ClusterName>zdwechatclustersea</ClusterName>
          <Region>Southeast Asia</Region>
          <StorageAccounts>
            <BlobContainerReference>
              <AccountName>zdwechatprimaryse.blob.core.windows.net</AccountName>
              <BlobContainerName>install</BlobContainerName>
              <Key>njn20Vs6NC4Rh39fcqdf86zhy6l7IMAw6fbUpWsEQLzxEYeiDoqG1lE6c2/kgLVfc9p1rwl0PMSBwoF+Ony9+w==</Key>
            </BlobContainerReference>
          </StorageAccounts>
          <Settings>
            <Core />
            <Hdfs />
            <MapReduce />
            <Hive>
              <AdditionalLibraries>
                <AccountName>zdwechathivestoragesea.blob.core.windows.net</AccountName>
                <BlobContainerName>hivecustomlibs</BlobContainerName>
                <Key>oDJwj9s2UWU2Usih6aEp+45m/ve/2LSaCFJ9XsNARM+ohCy+uhHGdECXzDHSL84wBCRO3ksf8NPIFUr2sT13pA==</Key>
              </AdditionalLibraries>
              <Catalog>
                <DatabaseName>zdwechathive</DatabaseName>
                <Password>Server2008!</Password>
                <Server>zdwechat.cloudapp.net</Server>
                <Username>wechatuser</Username>
              </Catalog>
            </Hive>
            <Oozie />
            <Yarn />
          </Settings>
        </ClusterContainer>
      </IntrinsicSettings>
    </Resource>
    
    HTTP Response:
      Status Code: Accepted
          Headers: x-ms-servedbyregion = ussouth2
                   x-ms-request-id = 95fb81bddf8194bc8c46e0f01e4a8521
                   Cache-Control = no-cache
                   Date = Mon, 05 May 2014 02:47:19 GMT
                   ETag = 8a2fd2b6-8ddc-45ec-b174-9daa92da9729
                   Server = 1.0.6198.70
                            (rd_rdfe_stable.140426-2318)
                            Microsoft-HTTPAPI/2.0
    
          Content:
    
    
    调试: Severity: Informational
      Type: HttpClientAbstraction
    Member: LogRequestResponseDetails
       UTC: 2014-05-05T02:47:27.6474293Z
     HTTP Request:
      SDK Version: 1.1.0.7
              Uri: https://management.core.windows.net/48e7656d-0790-4af6-ac68-841968be9abb/cloudservices
           Method: GET
     Content Type: application/xml
          Headers: x-ms-version = 2012-08-01
                   accept = application/xml
                   useragent = HDInsight .NET SDK
    
          Content:
    
    
    HTTP Response:
      Status Code: OK
          Headers: x-ms-servedbyregion = ussouth2
                   x-ms-request-id = 85e0e96ccc1d98be80bf5280c03b2d6c
                   Cache-Control = no-cache
                   Date = Mon, 05 May 2014 02:47:26 GMT
                   Server = 1.0.6198.70
                            (rd_rdfe_stable.140426-2318)
                            Microsoft-HTTPAPI/2.0
    
          Content:
    <?xml version="1.0" encoding="utf-8"?>
    <CloudServices xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
      <CloudService>
        <Name>hdinsightZ5I5DWQOTDYUYQCNJBAT5OIKQ55NKWUM3VVYA5MLGDB35JB2EA2Q-Southeast-Asia</Name>
        <Label>HdInsight CloudService</Label>
        <Description>HdInsight clusters for subscription 48e7656d-0790-4af6-ac68-841968be9abb</Description>
        <GeoRegion>Southeast Asia</GeoRegion>
        <Resources>
          <Resource>
            <ResourceProviderNamespace>hdinsight</ResourceProviderNamespace>
            <Type>containers</Type>
            <Name>zdwechatclustersea</Name>
            <ETag>8a2fd2b6-8ddc-45ec-b174-9daa92da9729</ETag>
            <State>Unknown</State>
            <OperationStatus>
              <Type>Create</Type>
              <Result>Failed</Result>
              <Error>
                <HttpCode>400</HttpCode>
                <Message>UnableToVerifyLocationOfResource</Message>
                <ExtendedCode />
              </Error>
            </OperationStatus>
          </Resource>
        </Resources>
      </CloudService>
      <CloudService>
        <Name>hdinsightZ5I5DWQOTDYUYQCNJBAT5OIKQ55NKWUM3VVYA5MLGDB35JB2EA2Q-West-US</Name>
        <Label>HdInsight CloudService</Label>
        <Description>HdInsight clusters for subscription 48e7656d-0790-4af6-ac68-841968be9abb</Description>
        <GeoRegion>West US</GeoRegion>
        <Resources />
      </CloudService>
    </CloudServices>
    
    调试: Severity: Informational
      Type: HDInsightManagementPocoClient
    Member: PollSignal
       UTC: 2014-05-05T02:47:27.6794311Z
    Current State Unknown -> waiting for one state of Operational,Running
    调试: Severity: Error
    发生一个或多个错误。
       在 System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
       在 System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
       在 Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.NewAzureHDInsightClusterCmdlet.EndProcessing() 位置
    c:\workspace\workspace\build-azure-sdk-tools-msi\WindowsAzurePowershell\src\Commands\HDInsight\NewAzureHDInsightCluster
    Cmdlet.cs:行号 269
    已取消一个任务。
    
    
    New-AzureHDInsightCluster : 已取消一个任务。
    所在位置 行:5 字符: 3
    + | New-AzureHDInsightCluster -Subscription $PrimarySubscriptionID -Credential $cr ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [New-AzureHDInsightCluster], TaskCanceledException
        + FullyQualifiedErrorId : System.Threading.Tasks.TaskCanceledException,Microsoft.WindowsAzure.Management.HDInsight
       .Cmdlet.PSCmdlets.NewAzureHDInsightClusterCmdlet

    请问各位大侠,报<Message>UnableToVerifyLocationOfResource</Message> 是为什么呢?
    2014年5月6日 2:31

答案