HDInsight Deployment using CLI with ADLS Gen2 Storage.

Question

Hi,

I am trying to create a HDInsight Cluster with version 4.0. When I give a storage account that have enabled hierarchical namespace (Azure Data Lake Storage Gen2), I get the below error:

DeploymentDocument 'AmbariConfiguration_1_7' failed the validation. Error: 'Blob storage account API 'acoehdiadlsgen2wus2dev01.blob.core.windows.net' is not supported for accounts that have enabled hierarchical namespace (Azure Data Lake Storage Gen2). Please refer https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues

Below is the storage configuration I am using:

Account kind: StorageV2 (general purpose v2)

Performance: Standard

Secure transfer required: Disabled

Access tier (default): Hot

Azure Active Directory Domain Service (AAD DS): Enabled

Hierarchical namespace(Data Lake Storage Gen2): Enabled

There are no issues when I am using a storage account that have disabled enabled hierarchical namespace (Azure Data Lake Storage Gen2).

Answer 1

Hello,

In order to investigate further:

• Could you please share the steps which you are following to create HDInsight cluster with ADLS Gen 2 account using Azure CLI?
• Could you please share the Azure CLI code snippet for the reference?

Meanwhile, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.

Hope this helps. Do let us know if you any further queries.

----------------------------------------------------------------------------------------

Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

Answer 2

Hello,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

Answer 3

Hello,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Answer 4

Sorry for the late Reply.. Below is my CLI command:

az hdinsight create -t spark -l westus2 -g kp-rg-cdo-casazcoe-sandbox-01 -n kumarsparktest --version 4.0 --storage-account acoehdiadlsgen2wus2dev01 --storage-account-key $(az storage account keys list --account-name acoehdiadlsgen2wus2dev01 --query [0].value --output tsv) --cluster-tier Premium --subnet $(az network vnet subnet show -g kp-rg-networking-acoe-devtest --vnet-name kp-vn-usw2-acoe-devtest-01 -n ACOE02-10.10.194.224_27 --query id --output tsv) --http-user kpadmuser --http-password HttpPassword1234! --ssh-password sshPassword1234! --ssh-user kpsshuser --esp  --assign-identity $(az identity show -g kp-rg-cdo-casazcoe-sandbox-01 -n acoehdiadlsgen2wus2dev01msi --query id --output tsv) --domain /subscriptions/e91d0e75-bfa6-4aed-a5ff-4f73e6b4bf1a/resourceGroups/kp-rg-cto-aadds-comsvc/providers/Microsoft.AAD/domainServices/kpaaddsprod.onmicrosoft.com --ldaps-urls ldaps://kpaaddsprod.onmicrosoft.com:636 --cluster-admin-account [e-mail] --cluster-users-group-dns cap-hdi-svc)

My storage account "acoehdiadlsgen2wus2dev01" has adls enabled.

I am trying to move to ARM Templates due to couple of missing items in azure CLI.

Answer 5

Hello,

When you are using Azure Data Lake Gen2 storage account, you need to use the below parameter.

--storage-account-managed-identity

User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.

For more details, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.

Hope this helps. Do let us know if you any further queries.

----------------------------------------------------------------------------------------

Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

Answer 6

Hello,

Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.