Answered by:
HDInsight Deployment using CLI with ADLS Gen2 Storage.

Question
-
Hi,
I am trying to create a HDInsight Cluster with version 4.0. When I give a storage account that have enabled hierarchical namespace (Azure Data Lake Storage Gen2), I get the below error:
DeploymentDocument 'AmbariConfiguration_1_7' failed the validation. Error: 'Blob storage account API 'acoehdiadlsgen2wus2dev01.blob.core.windows.net' is not supported for accounts that have enabled hierarchical namespace (Azure Data Lake Storage Gen2). Please refer https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issuesBelow is the storage configuration I am using:
Account kind: StorageV2 (general purpose v2)
Performance: Standard
Secure transfer required: Disabled
Access tier (default): Hot
Azure Active Directory Domain Service (AAD DS): EnabledHierarchical namespace(Data Lake Storage Gen2): Enabled
There are no issues when I am using a storage account that have disabled enabled hierarchical namespace (Azure Data Lake Storage Gen2).
- Edited by BiNiku Thursday, April 9, 2020 7:02 AM
Thursday, April 9, 2020 6:57 AM
Answers
-
Hello,
When you are using Azure Data Lake Gen2 storage account, you need to use the below parameter.
--storage-account-managed-identity
User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.
For more details, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.
Hope this helps. Do let us know if you any further queries.
----------------------------------------------------------------------------------------
Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.
- Proposed as answer by CHEEKATLAPRADEEP-MSFTMicrosoft employee Wednesday, April 15, 2020 5:29 AM
- Marked as answer by BiNiku Thursday, April 16, 2020 3:20 PM
Wednesday, April 15, 2020 5:29 AM
All replies
-
Hello,
In order to investigate further:
- Could you please share the steps which you are following to create HDInsight cluster with ADLS Gen 2 account using Azure CLI?
- Could you please share the Azure CLI code snippet for the reference?
Meanwhile, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.
Hope this helps. Do let us know if you any further queries.
----------------------------------------------------------------------------------------
Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.
Thursday, April 9, 2020 12:12 PM -
Hello,
Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.
Monday, April 13, 2020 11:38 AM -
Hello,
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Tuesday, April 14, 2020 3:45 PM -
Sorry for the late Reply.. Below is my CLI command:
az hdinsight create -t spark -l westus2 -g kp-rg-cdo-casazcoe-sandbox-01 -n kumarsparktest --version 4.0 --storage-account acoehdiadlsgen2wus2dev01 --storage-account-key $(az storage account keys list --account-name acoehdiadlsgen2wus2dev01 --query [0].value --output tsv) --cluster-tier Premium --subnet $(az network vnet subnet show -g kp-rg-networking-acoe-devtest --vnet-name kp-vn-usw2-acoe-devtest-01 -n ACOE02-10.10.194.224_27 --query id --output tsv) --http-user kpadmuser --http-password HttpPassword1234! --ssh-password sshPassword1234! --ssh-user kpsshuser --esp --assign-identity $(az identity show -g kp-rg-cdo-casazcoe-sandbox-01 -n acoehdiadlsgen2wus2dev01msi --query id --output tsv) --domain /subscriptions/e91d0e75-bfa6-4aed-a5ff-4f73e6b4bf1a/resourceGroups/kp-rg-cto-aadds-comsvc/providers/Microsoft.AAD/domainServices/kpaaddsprod.onmicrosoft.com --ldaps-urls ldaps://kpaaddsprod.onmicrosoft.com:636 --cluster-admin-account svccoeazdashsb@KP.ORG --cluster-users-group-dns cap-hdi-svc)
My storage account "acoehdiadlsgen2wus2dev01" has adls enabled.
I am trying to move to ARM Templates due to couple of missing items in azure CLI.Tuesday, April 14, 2020 5:51 PM -
Hello,
When you are using Azure Data Lake Gen2 storage account, you need to use the below parameter.
--storage-account-managed-identity
User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.
For more details, I would request you to go through the documentation “Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters” and “Creating a cluster with ADLS Gen2 through the Azure CLI”.
Hope this helps. Do let us know if you any further queries.
----------------------------------------------------------------------------------------
Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.
- Proposed as answer by CHEEKATLAPRADEEP-MSFTMicrosoft employee Wednesday, April 15, 2020 5:29 AM
- Marked as answer by BiNiku Thursday, April 16, 2020 3:20 PM
Wednesday, April 15, 2020 5:29 AM -
Hello,
Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.
Thursday, April 16, 2020 11:48 AM