locked
HDInsight Deployment using CLI with ADLS Gen2 Storage. RRS feed

  • Question

  • Hi,

    I am trying to create a HDInsight Cluster with version 4.0. When I give a storage account that have enabled hierarchical namespace (Azure Data Lake Storage Gen2), I get the below error:

    DeploymentDocument 'AmbariConfiguration_1_7' failed the validation. Error: 'Blob storage account API 'acoehdiadlsgen2wus2dev01.blob.core.windows.net' is not supported for accounts that have enabled hierarchical namespace (Azure Data Lake Storage Gen2). Please refer https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues

    Below is the storage configuration I am using:

    Account kindStorageV2 (general purpose v2)

    Performance: Standard

    Secure transfer required: Disabled

    Access tier (default): Hot

    Azure Active Directory Domain Service (AAD DS): Enabled

    Hierarchical namespace(Data Lake Storage Gen2): Enabled

    There are no issues when I am using a storage account that have disabled enabled hierarchical namespace (Azure Data Lake Storage Gen2).




    • Edited by BiNiku Thursday, April 9, 2020 7:02 AM
    Thursday, April 9, 2020 6:57 AM

Answers

  • Hello,

    When you are using Azure Data Lake Gen2 storage account, you need to use the below parameter.

    --storage-account-managed-identity

    User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.

    For more details, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Wednesday, April 15, 2020 5:29 AM

All replies

  • Hello,

    In order to investigate further:

    • Could you please share the steps which you are following to create HDInsight cluster with ADLS Gen 2 account using Azure CLI?
    • Could you please share the Azure CLI code snippet for the reference?

    Meanwhile, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Thursday, April 9, 2020 12:12 PM
  • Hello,

    Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

    Monday, April 13, 2020 11:38 AM
  • Hello,

    Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

    Tuesday, April 14, 2020 3:45 PM
  • Sorry for the late Reply.. Below is my CLI command:

    az hdinsight create -t spark -l westus2 -g kp-rg-cdo-casazcoe-sandbox-01 -n kumarsparktest --version 4.0 --storage-account acoehdiadlsgen2wus2dev01 --storage-account-key $(az storage account keys list --account-name acoehdiadlsgen2wus2dev01 --query [0].value --output tsv) --cluster-tier Premium --subnet $(az network vnet subnet show -g kp-rg-networking-acoe-devtest --vnet-name kp-vn-usw2-acoe-devtest-01 -n ACOE02-10.10.194.224_27 --query id --output tsv) --http-user kpadmuser --http-password HttpPassword1234! --ssh-password sshPassword1234! --ssh-user kpsshuser --esp  --assign-identity $(az identity show -g kp-rg-cdo-casazcoe-sandbox-01 -n acoehdiadlsgen2wus2dev01msi --query id --output tsv) --domain /subscriptions/e91d0e75-bfa6-4aed-a5ff-4f73e6b4bf1a/resourceGroups/kp-rg-cto-aadds-comsvc/providers/Microsoft.AAD/domainServices/kpaaddsprod.onmicrosoft.com --ldaps-urls ldaps://kpaaddsprod.onmicrosoft.com:636 --cluster-admin-account svccoeazdashsb@KP.ORG --cluster-users-group-dns cap-hdi-svc)

    My storage account "
    acoehdiadlsgen2wus2dev01" has adls enabled.

    I am trying to move to ARM Templates due to couple of missing items in azure CLI.

    Tuesday, April 14, 2020 5:51 PM
  • Hello,

    When you are using Azure Data Lake Gen2 storage account, you need to use the below parameter.

    --storage-account-managed-identity

    User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.

    For more details, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Wednesday, April 15, 2020 5:29 AM
  • Hello,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Thursday, April 16, 2020 11:48 AM