none
powershell script to launch hdinsight in existing vnet RRS feed

  • Question

  •  Hi All
    below is the error i am getting while i use powershell script to launch hdinsight cluster in existing vnet using the below script :


    $vnetName = "acctvnet"
    $subnetName = "subnet2"
    $resourceGroupName = "test"
    $location = "West US"
    $vnet = Get-AzVirtualNetwork `
        -Name $vnetName `
        -ResourceGroupName $resourceGroupName
    $subnet = $vnet.Subnets | Where-Object Name -eq $subnetName
    $defaultStorageAccountName = Read-Host -Prompt "Enter the name of the storage account"

    # Create an Az.Storage account and container
    New-AzStorageAccount `
        -ResourceGroupName $resourceGroupName `
        -Name $defaultStorageAccountName `
        -Type Standard_LRS `
        -Location $location
    $defaultStorageAccountKey = (Get-AzStorageAccountKey `
                                    -ResourceGroupName $resourceGroupName `
                                    -Name $defaultStorageAccountName)[0].Value
    $defaultStorageContext = New-AzStorageContext `
                                    -StorageAccountName $defaultStorageAccountName `
                                    -StorageAccountKey $defaultStorageAccountKey

    # Get information for the HDInsight cluster
    $clusterName = Read-Host -Prompt "Enter the name of the HDInsight cluster"
    # Cluster login is used to secure HTTPS services hosted on the cluster
    $httpCredential = Get-Credential -Message "Enter Cluster login credentials" -UserName "admin"
    # SSH user is used to remotely connect to the cluster using SSH clients
    $sshCredentials = Get-Credential -Message "Enter SSH user credentials"

    # Default cluster size (# of worker nodes), version, type, and OS
    $clusterSizeInNodes = "1"
    $clusterVersion = "3.6"
    $clusterType = "Hadoop"
    $clusterOS = "Linux"
    # Set the storage container name to the cluster name
    $defaultBlobContainerName = $clusterName

    # Create a blob container. This holds the default data store for the cluster.
    New-AzStorageContainer `
        -Name $clusterName -Context $defaultStorageContext 

    # Create the HDInsight cluster
         New-AzHDInsightCluster `
        -ResourceGroupName $resourceGroupName `
        -VirtualNetworkId  $vnet `
        -SubnetName $subnetName `
        -ClusterName $clusterName `
        -Location $location `
        -ClusterSizeInNodes $clusterSizeInNodes `
        -ClusterType $clusterType `
        -OSType $clusterOS `
        -Version $clusterVersion `
        -HttpCredential $httpCredential `
        -DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
        -DefaultStorageAccountKey $defaultStorageAccountKey `
        -DefaultStorageContainer $clusterName `
        -SshCredential $sshCredentials

    ERROR:

    New-AzHDInsightCluster : Operation returned an invalid status code 'BadRequest'
    At line:1 char:1
    + New-AzHDInsightCluster `

    Also,     -VirtualNetworkId  $vnet `
        -SubnetName $subnetName ` are not part of cmdlet function is the errro ----- i installed AZ modules even then i get this error 

    Kindly help me resolve this issue and really appreciate you all.

        
    Friday, December 13, 2019 1:31 PM

All replies

  • Hi All
    below is the error i am getting while i use powershell script to launch hdinsight cluster in existing vnet using the below script :


    $vnetName = "acctvnet"
    $subnetName = "subnet2"
    $resourceGroupName = "test"
    $location = "West US"
    $vnet = Get-AzVirtualNetwork `
        -Name $vnetName `
        -ResourceGroupName $resourceGroupName
    $subnet = $vnet.Subnets | Where-Object Name -eq $subnetName
    $defaultStorageAccountName = Read-Host -Prompt "Enter the name of the storage account"

    # Create an Az.Storage account and container
    New-AzStorageAccount `
        -ResourceGroupName $resourceGroupName `
        -Name $defaultStorageAccountName `
        -Type Standard_LRS `
        -Location $location
    $defaultStorageAccountKey = (Get-AzStorageAccountKey `
                                    -ResourceGroupName $resourceGroupName `
                                    -Name $defaultStorageAccountName)[0].Value
    $defaultStorageContext = New-AzStorageContext `
                                    -StorageAccountName $defaultStorageAccountName `
                                    -StorageAccountKey $defaultStorageAccountKey

    # Get information for the HDInsight cluster
    $clusterName = Read-Host -Prompt "Enter the name of the HDInsight cluster"
    # Cluster login is used to secure HTTPS services hosted on the cluster
    $httpCredential = Get-Credential -Message "Enter Cluster login credentials" -UserName "admin"
    # SSH user is used to remotely connect to the cluster using SSH clients
    $sshCredentials = Get-Credential -Message "Enter SSH user credentials"

    # Default cluster size (# of worker nodes), version, type, and OS
    $clusterSizeInNodes = "1"
    $clusterVersion = "3.6"
    $clusterType = "Hadoop"
    $clusterOS = "Linux"
    # Set the storage container name to the cluster name
    $defaultBlobContainerName = $clusterName

    # Create a blob container. This holds the default data store for the cluster.
    New-AzStorageContainer `
        -Name $clusterName -Context $defaultStorageContext 

    # Create the HDInsight cluster
         New-AzHDInsightCluster `
        -ResourceGroupName $resourceGroupName `
        -VirtualNetworkId  $vnet `
        -SubnetName $subnetName `
        -ClusterName $clusterName `
        -Location $location `
        -ClusterSizeInNodes $clusterSizeInNodes `
        -ClusterType $clusterType `
        -OSType $clusterOS `
        -Version $clusterVersion `
        -HttpCredential $httpCredential `
        -DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
        -DefaultStorageAccountKey $defaultStorageAccountKey `
        -DefaultStorageContainer $clusterName `
        -SshCredential $sshCredentials

    ERROR:

    New-AzHDInsightCluster : Operation returned an invalid status code 'BadRequest'
    At line:1 char:1
    + New-AzHDInsightCluster `

    also  -VirtualNetworkId  $vnet `
        -SubnetName $subnetName ` are not part of cmdlet functions is the error is am getting.

    Kindly provide a solution for this and appreciate your help in this regard.

        
    Friday, December 13, 2019 11:06 AM
  • Hi Rahul,

    In the below script, you are missing “-VirtualNetworkId  $vnet.Id” and “-SubnetName $subnet.Id” in the script.

    # Create the HDInsight cluster

         New-AzHDInsightCluster `

        -ResourceGroupName $resourceGroupName `

        -VirtualNetworkId  $vnet.Id `

        -SubnetName $subnet.Id `

        -ClusterName $clusterName `

        -Location $location `

        -ClusterSizeInNodes $clusterSizeInNodes `

        -ClusterType $clusterType `

        -OSType $clusterOS `

        -Version $clusterVersion `

        -HttpCredential $httpCredential `

        -DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `

        -DefaultStorageAccountKey $defaultStorageAccountKey `

        -DefaultStorageContainer $clusterName `

        -SshCredential $sshCredentials

    And also, please do remember to add the network security group in the script.

    For more details, refer “Create virtual networks for Azure HDInsight clusters”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members

    Monday, December 16, 2019 6:50 AM
    Moderator
  • Thank you so much for the reply.

    I have made the changes suggested by you and below is the script that i am running which is giving the error as highlighted below:

    $vnetName = "acctvnet"
    $subnetName = "subnet2"
    $resourceGroupName = "test"
    $location = "West US"
    $vnet = Get-AzVirtualNetwork `
        -Name $vnetName `
        -ResourceGroupName $resourceGroupName
    $subnet = $vnet.Subnets | Where-Object Name -eq $subnetName
    $defaultStorageAccountName = Read-Host -Prompt "Enter the name of the storage account"

    # Create an Az.Storage account and container
    New-AzStorageAccount `
        -ResourceGroupName $resourceGroupName `
        -Name $defaultStorageAccountName `
        -Type Standard_LRS `
        -Location $location
    $defaultStorageAccountKey = (Get-AzStorageAccountKey `
                                    -ResourceGroupName $resourceGroupName `
                                    -Name $defaultStorageAccountName)[0].Value
    $defaultStorageContext = New-AzStorageContext `
                                    -StorageAccountName $defaultStorageAccountName `
                                    -StorageAccountKey $defaultStorageAccountKey

    # Get information for the HDInsight cluster
    $clusterName = Read-Host -Prompt "Enter the name of the HDInsight cluster"
    # Cluster login is used to secure HTTPS services hosted on the cluster
    $httpCredential = Get-Credential -Message "Enter Cluster login credentials" -UserName "admin"
    # SSH user is used to remotely connect to the cluster using SSH clients
    $sshCredentials = Get-Credential -Message "Enter SSH user credentials"

    # Default cluster size (# of worker nodes), version, type, and OS
    $clusterSizeInNodes = "1"
    $clusterVersion = "3.6"
    $clusterType = "Hadoop"
    $clusterOS = "Linux"
    # Set the storage container name to the cluster name
    $defaultBlobContainerName = $clusterName

    # Create a blob container. This holds the default data store for the cluster.
    New-AzStorageContainer `
        -Name $clusterName -Context $defaultStorageContext
    # Create the HDInsight cluster
    New-AzHDInsightCluster `
        -ResourceGroupName $resourceGroupName `
        -VirtualNetworkId  $vnet.Id `
        -SubnetName $subnet.Id `
        -ClusterName $clusterName `
        -Location $location `
        -ClusterSizeInNodes $clusterSizeInNodes `
        -ClusterType $clusterType `
        -OSType $clusterOS `
        -Version $clusterVersion `
        -HttpCredential $httpCredential `
        -DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
        -DefaultStorageAccountKey $defaultStorageAccountKey `
        -DefaultStorageContainer $clusterName `
        -SshCredential $sshCredentials

    Error:

    ------------------------------------------------------------------------------------------------

    New-AzHDInsightCluster : Operation returned an invalid status code 'BadRequest'
    At line:1 char:1
    + New-AzHDInsightCluster `
    + ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [New-AzHDInsightCluster], ErrorResponseException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.HDInsight.NewAzureHDInsightClusterCommand

    --------------------------------------------------------------------------------------------------------------------

    Monday, December 16, 2019 9:13 AM
  • Hi Rahul,

    Could you please install latest PowerShell module and retry?

    In case, the error persists please do share the screenshot of the error message.

    Monday, December 16, 2019 10:23 AM
    Moderator
  • Hi Pradeep,

    Installed the latest powershell module and still i get the below error:

    please let me know the reason for this error and i was using the above script which i shared in the last reply.

    Thanks-Rahul.!

    

    • Marked as answer by Rahul Akkula Monday, December 16, 2019 12:21 PM
    • Unmarked as answer by Rahul Akkula Monday, December 16, 2019 12:34 PM
    Monday, December 16, 2019 10:39 AM
  • Hi Rahul,

    This could be the issue with the Azure PowerShell module, could you please uninstall and install Azure PowerShell, Azure HDInsight module, and retry the same.

    I have tried the same script from my end and works without any issue.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members

    Tuesday, December 17, 2019 5:26 AM
    Moderator
  • Hi 

    I tried doing the same but still ending up with the same error.(attached the error)..i am sharing with you the VM details which you can access and try the same:

    UserName:RahulXXXX

    Password:XXXXXX

    Public ip :1x.8x.1x.1xx (3389)

    error:



       Blob End Point: https://tapasyaammu.blob.core.windows.net/

    Name                 PublicAccess         LastModified
    ----                 ------------         ------------
    tapasyaammu          Off                  12/17/2019 9:40:03 AM +00:00


    PS C:\Users\Rahul> # Create the HDInsight cluster
    PS C:\Users\Rahul> New-AzHDInsightCluster `
    >>     -ResourceGroupName $resourceGroupName `
    >>     -VirtualNetworkId  $vnet.Id `
    >>     -SubnetName $subnet.Id `
    >>     -ClusterName $clusterName `
    >>     -Location $location `
    >>     -ClusterSizeInNodes $clusterSizeInNodes `
    >>     -ClusterType $clusterType `
    >>     -OSType $clusterOS `
    >>     -Version $clusterVersion `
    >>     -HttpCredential $httpCredential `
    >>     -DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
    >>     -DefaultStorageAccountKey $defaultStorageAccountKey `
    >>     -DefaultStorageContainer $clusterName `
    >>     -SshCredential $sshCredentials
    New-AzHDInsightCluster : Operation returned an invalid status code 'BadRequest'
    At line:1 char:1
    + New-AzHDInsightCluster `
    + ~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : CloseError: (:) [New-AzHDInsightCluster], ErrorResponseException
        + FullyQualifiedErrorId : Microsoft.Azure.Commands.HDInsight.NewAzureHDInsightClusterCommand

    PS C:\Users\Rahul> Get-InstalledModule -Name Az -AllVersions | select Name,Version

    Name Version
    ---- -------
    Az   3.2.0


    PS C:\Users\Rahul> Install-Module -Name Az.HDInsight -RequiredVersion 3.0.0
    PS C:\Users\Rahul>





    Tuesday, December 17, 2019 9:51 AM
  • As you can see in my last reply i installed modules which are below :

    PS C:\Users\Rahul> Get-InstalledModule -Name Az -AllVersions | select Name,Version

    Name Version
    ---- -------
    Az   3.2.0


    PS C:\Users\Rahul> Install-Module -Name Az.HDInsight -RequiredVersion 3.0.0
    PS C:\Users\Rahul>

    i have shared the VM login details which you can access and please help me out with the solution.

    Tuesday, December 17, 2019 10:05 AM
  • Hi Pradeep,

    Could you please let me know the reason for this error : "New-AzHDInsightCluster : Operation returned an invalid status code 'BadRequest'
    At line:1 char:1
    + New-AzHDInsightCluster `
    + ~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : CloseError: (:) [New-AzHDInsightCluster], ErrorResponseException"

    Kindly reply and i am awaiting for your response.

    Thanks in Advance and appreciate your help.



    • Edited by Rahul Akkula Wednesday, December 18, 2019 2:34 PM waiting for reply
    Wednesday, December 18, 2019 8:24 AM
  • Hi Rahul,

    You are using Azure free trial Azure Subscription; you cannot create any HDInsight cluster because which requires more than 8 cores.

    Reason: Azure Free Trail has a limit of 4 cores, and you cannot create HDInsight cluster using a Free Trial Subscription.

    You need to upgrade to Pay-As-You-Go subscription to create Azure HDInsight clusters.

    Note: Free Trial subscriptions aren't eligible for limit or quota increases. If you have a Free Trial subscription, you can upgrade to a Pay-As-You-Go subscription.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Thursday, December 19, 2019 6:18 AM
    Moderator
  • Hi Pradeep,

    If thats the case then i should not be able to create it through portal as well but i am able to create it.\

    Please let me know on this

    Thanks-Rahul.!

    Thursday, December 19, 2019 2:06 PM
  • Hi Rahul,

    This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you send an email to AzCommunity@Microsoft.com with your Subscription ID and thread link to this post, and I will enable a one-time free support request for your subscription. 

    Please reference this forum thread in the subject: “powershell script to launch hdinsight in existing vnet”. Thank you for your persistence.

    Friday, December 20, 2019 4:51 AM
    Moderator
  • Hi Pradeep

    Thank You very much for helping me in resolving this issue and as suggested by you have sent an email to "AzCommunity@Microsoft.com" .

    Thanks-Rahul.!

    +31-617854586

    Friday, December 20, 2019 9:41 AM
  • Hi Rahul,

    Thank you for reaching out.

    We have enabled for one-time free technical support.

    I hope you have received one on one support to work towards a resolution on this matter.

    Friday, December 20, 2019 9:48 AM
    Moderator
  • Yes i have received it and the support request is "119122022000449 " .

    Please let me know if anything is required.

    I'd be awaiting for your reply.

    Thanks-Rahul.!

    +31-617854586

    Friday, December 20, 2019 10:08 AM
  • Hi Rahul,

    Thanks for the update, Once the issue is sorted out with the support, please do share the resolution, which might be beneficial to other community members reading this thread. 

    Friday, December 20, 2019 10:22 AM
    Moderator