none
Issue in creating HDInsight with Data Lake Using Template Deployment.

    Question

  • Hi,

    I'm trying to create Azure HDInsight with Data lake using Template Deployment. But I'm Facing an issue in executing the template because of what i think the reason would be the "ClusterIdentity". Please find below screenshot for more details.

    ERROR Snapshot

    $ az group deployment create \
    >     --name demostack \
    >     --resource-group eastus \
    >     --template-file template.json \
    >     --parameters @parameters.json
    ERROR: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details. {
      "status": "Failed",
      "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The resource operation completed with terminal provisioning state 'Failed'.",
        "details": [
          {
            "code": "InvalidDocumentErrorCode",
            "message": "DeploymentDocument 'AmbariConfiguration_1_7' failed the validation. Error: 'Error while getting access to the datalake storage account neodls: Error while getting the OAuth token from AAD for AppPrincipalId 5546f8ca-84a0-4083-8f9d-99ba726d7cef, ResourceUri https://management.core.windows.net/, AADTenantId https://login.windows.net/0a425771-4fcc-460c-bce9-dd6139a133c9, ClientCertificateThumbprint 6416EC0D76E3F7AE11CB959C8B8DA8DCE3F60188.'"
          }
        ]
      }
    }  Correlation ID: cfxxx5d-92xx5-4xx8-bxxa-e7c07xxecxx

    Here I have few doubts like

    1. Is "SecureString" values like clusterpassword,sshpassword in "parameter.json" should be given as plaintext or i have to convert it into Securestring and give the secure string value to it?

    2. The field "identityCertificate" should be "base64" encoded of "Certificate.pfx" file content or I'll have to convert it as Base64 -> SecureString and give it in parameter.json?

    3. How do i assign permissions of this "Service Principle Name" to Data Lake store when Creating it? 

    Help appreciated much ! Thanks

    Regards,
    Karthick



    Thursday, April 27, 2017 9:49 AM

Answers

  • cleaned up and got it up and running, attached template and PS code for others to use.

    cls
    $templatefilepath = "C:\Users\****\Desktop\hdinsighttest\hdinsWithDatalakeStoreTemplate.Json"
    $SSHpass = ConvertTo-SecureString -String "*********" -AsPlainText -Force
    $rg = "*******"
    $datelakestore = "datalakestore"
    
    $certFolder = "C:\Users\****\Documents\testjson\certificates"
    $certFilePath = "$certFolder\certFile.pfx"
    $certStartDate = (Get-Date).Date
    $certStartDateStr = $certStartDate.ToString("04/28/2017")
    $certEndDate = $certStartDate.AddYears(1)
    $certEndDateStr = $certEndDate.ToString("05/28/2017")
    $certName = "testhdinsight16"
    $certPassword = "*******"
    $certPasswordSecureString = ConvertTo-SecureString $certPassword -AsPlainText -Force
     
    mkdir $certFolder
     
    $cert = New-SelfSignedCertificate -DnsName $certName -CertStoreLocation cert:\CurrentUser\My -KeySpec KeyExchange -NotAfter $certEndDate -NotBefore $certStartDate
    $certThumbprint = $cert.Thumbprint
    $cert = (Get-ChildItem -Path cert:\CurrentUser\My\$certThumbprint)
     
    Export-PfxCertificate -Cert $cert -FilePath $certFilePath -Password $certPasswordSecureString
    
     
    $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certFilePath, $certPasswordSecureString)
    $credential = [System.Convert]::ToBase64String($certificatePFX.GetRawCertData())
     
    $application = New-AzureRmADApplication -DisplayName $certName `
                             -HomePage "https://$certName.azurehdinsight.net" -IdentifierUris "https://$certName.azurehdinsight.net" `
                             -CertValue $credential `
                             -StartDate $certStartDate -EndDate $certEndDate
    #wait of ad
    Start-Sleep -Seconds 20
                             
    $servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $application.ApplicationId
    
    Set-AzureRmDataLakeStoreItemAclEntry -AccountName $datelakestore -Path / -AceType User -Id $servicePrincipal.Id -Permissions All
    
    
    $secureCert = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))
    
    New-AzureRmResourceGroupDeployment `
        -ResourceGroupName $rg `
        -TemplateFile $templatefilepath `
        -identityCertificate $secureCert `
        -identityCertificatePassword $certPasswordSecureString `
        -clusterName  $certName `
        -clusterLoginPassword $SSHpass `
        -sshPassword $SSHpass `
        -servicePrincipalApplicationId $application.ApplicationId

    Template:

    {
      "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
      "contentVersion": "0.9.0.0",
      "parameters": {
        "clusterName": {
          "type": "string",
          "metadata": {
            "description": "The name of the HDInsight cluster to create."
          }
        },
        "clusterLoginUserName": {
          "type": "string",
          "defaultValue": "admin",
          "metadata": {
            "description": "These credentials can be used to submit jobs to the cluster and to log into cluster dashboards."
          }
        },
        "clusterLoginPassword": {
          "type": "securestring",
          "metadata": {
            "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
          }
        },
        "location": {
          "type": "string",
          "defaultValue": "northeurope",
          "metadata": {
            "description": "The location where all azure resources will be deployed."
          }
        },
        "clusterVersion": {
          "type": "string",
          "defaultValue": "3.5",
          "metadata": {
            "description": "HDInsight cluster version."
          }
        },
        "clusterWorkerNodeCount": {
          "type": "int",
          "defaultValue": 2,
          "metadata": {
            "description": "The number of nodes in the HDInsight cluster."
          }
        },
        "clusterKind": {
          "type": "string",
          "defaultValue": "HADOOP",
          "metadata": {
            "description": "The type of the HDInsight cluster to create."
          }
        },
        "sshUserName": {
          "type": "string",
          "defaultValue": "sshuser",
          "metadata": {
            "description": "These credentials can be used to remotely access the cluster."
          }
        },
        "sshPassword": {
          "type": "securestring",
          "metadata": {
            "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
          }
        },
        "identityCertificate": {
          "type": "string"
        },
        "identityCertificatePassword": {
          "type": "securestring"
        },
        "servicePrincipalApplicationId": {
          "type": "string"
        }
      },
      "resources": [
        {
          "apiVersion": "2015-03-01-preview",
          "name": "[parameters('clusterName')]",
          "type": "Microsoft.HDInsight/clusters",
          "location": "[parameters('location')]",
          "dependsOn": [],
          "properties": {
            "clusterVersion": "[parameters('clusterVersion')]",
            "osType": "Linux",
            "tier": "standard",
            "clusterDefinition": {
              "kind": "[parameters('clusterKind')]",
              "configurations": {
                "gateway": {
                  "restAuthCredential.isEnabled": true,
                  "restAuthCredential.username": "[parameters('clusterLoginUserName')]",
                  "restAuthCredential.password": "[parameters('clusterLoginPassword')]"
                },
                "core-site": {
                  "fs.defaultFS": "adl://home",
                  "dfs.adls.home.hostname": "datalakestore.azuredatalakestore.net",
                  "dfs.adls.home.mountpoint": "/clusterslake/"
                },
                "clusterIdentity": {
                  "clusterIdentity.applicationId": "[parameters('servicePrincipalApplicationId')]",
                  "clusterIdentity.certificate": "[parameters('identityCertificate')]",
                  "clusterIdentity.aadTenantId": "https://login.windows.net/***********",
                  "clusterIdentity.resourceUri": "https://management.core.windows.net/",
                  "clusterIdentity.certificatePassword": "[parameters('identityCertificatePassword')]"
                }
              }
            },
            "storageProfile": {
              "storageaccounts": []
            },
            "computeProfile": {
              "roles": [
                {
                  "name": "headnode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 2,
                  "hardwareProfile": {
                    "vmSize": "Standard_D12_V2"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                },
                {
                  "name": "workernode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 2,
                  "hardwareProfile": {
                    "vmSize": "Standard_D4_V2"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                },
                {
                  "name": "zookeepernode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 3,
                  "hardwareProfile": {
                    "vmSize": "Small"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                }
              ]
            }
          }
        }
      ]
    }


    Friday, April 28, 2017 1:45 PM

All replies

  • Ya getting exact same error, tried using the following links as reference:

    https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-hdinsight-hadoop-use-powershell

    https://github.com/Azure/azure-quickstart-templates/tree/master/201-hdinsight-datalake-store-azure-storage

    What i have tried

    i create a SPI and add give this account permission to data lake store using the following code:

     $cert = New-SelfSignedCertificate -DnsName $certName -CertStoreLocation cert:\CurrentUser\My -KeySpec KeyExchange -NotAfter $certEndDate -NotBefore $certStartDate
     $certThumbprint = $cert.Thumbprint
     $cert = (Get-ChildItem -Path cert:\CurrentUser\My\$certThumbprint)
     
     Export-PfxCertificate -Cert $cert -FilePath $certFilePath -Password $certPasswordSecureString
    
     
     $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certFilePath, $certPasswordSecureString)
     $credential = [System.Convert]::ToBase64String($certificatePFX.GetRawCertData())
     
     $application = New-AzureRmADApplication -DisplayName $certName `
                             -HomePage "https://$clusterName.azurehdinsight.net" -IdentifierUris "https://$clusterName.azurehdinsight.net" `
                             -CertValue $credential `
                             -StartDate $certStartDate -EndDate $certEndDate
                             
     $servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $application.ApplicationId
    
     Set-AzureRmDataLakeStoreItemAclEntry -AccountName testdatalakestore -Path /clusters -AceType User -Id $servicePrincipal.Id -Permissions All

    but no matter  what i do it wont accept my Cert giving me:

    "message": "DeploymentDocument 'AmbariConfiguration_1_7' failed the validation. Error: 'Error while getting access to the datalake storage account testaccount: The spec
    ified network password is not correct.\r\n.'"

    I load my cert using the following code:

    $cert = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))

    and then convert to securestring using following, before i pass it to template:

    ConvertTo-SecureString -String $cert -AsPlainText -Force

    Also tried removing the Securestring type in template and then pass it as $cert directly, but keeps giving me same error!

    any ideas would be appreciated

    Friday, April 28, 2017 12:12 PM
  •  Hi Dahund,

       It seems in your case the password used for certificate might not be correct. If you see the second part of the error the description differs. "The Specified Network Password not correct". 

    And mine says "Error while getting the OAuth token from AAD for AppPrincipalId". Try giving the password you have used for certificate. also consider password rules (Size 13, Alphanumeric, Atleast one Uppercase).

    Regards,

    Karthick

    Friday, April 28, 2017 12:28 PM
  • cleaned up and got it up and running, attached template and PS code for others to use.

    cls
    $templatefilepath = "C:\Users\****\Desktop\hdinsighttest\hdinsWithDatalakeStoreTemplate.Json"
    $SSHpass = ConvertTo-SecureString -String "*********" -AsPlainText -Force
    $rg = "*******"
    $datelakestore = "datalakestore"
    
    $certFolder = "C:\Users\****\Documents\testjson\certificates"
    $certFilePath = "$certFolder\certFile.pfx"
    $certStartDate = (Get-Date).Date
    $certStartDateStr = $certStartDate.ToString("04/28/2017")
    $certEndDate = $certStartDate.AddYears(1)
    $certEndDateStr = $certEndDate.ToString("05/28/2017")
    $certName = "testhdinsight16"
    $certPassword = "*******"
    $certPasswordSecureString = ConvertTo-SecureString $certPassword -AsPlainText -Force
     
    mkdir $certFolder
     
    $cert = New-SelfSignedCertificate -DnsName $certName -CertStoreLocation cert:\CurrentUser\My -KeySpec KeyExchange -NotAfter $certEndDate -NotBefore $certStartDate
    $certThumbprint = $cert.Thumbprint
    $cert = (Get-ChildItem -Path cert:\CurrentUser\My\$certThumbprint)
     
    Export-PfxCertificate -Cert $cert -FilePath $certFilePath -Password $certPasswordSecureString
    
     
    $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certFilePath, $certPasswordSecureString)
    $credential = [System.Convert]::ToBase64String($certificatePFX.GetRawCertData())
     
    $application = New-AzureRmADApplication -DisplayName $certName `
                             -HomePage "https://$certName.azurehdinsight.net" -IdentifierUris "https://$certName.azurehdinsight.net" `
                             -CertValue $credential `
                             -StartDate $certStartDate -EndDate $certEndDate
    #wait of ad
    Start-Sleep -Seconds 20
                             
    $servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $application.ApplicationId
    
    Set-AzureRmDataLakeStoreItemAclEntry -AccountName $datelakestore -Path / -AceType User -Id $servicePrincipal.Id -Permissions All
    
    
    $secureCert = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))
    
    New-AzureRmResourceGroupDeployment `
        -ResourceGroupName $rg `
        -TemplateFile $templatefilepath `
        -identityCertificate $secureCert `
        -identityCertificatePassword $certPasswordSecureString `
        -clusterName  $certName `
        -clusterLoginPassword $SSHpass `
        -sshPassword $SSHpass `
        -servicePrincipalApplicationId $application.ApplicationId

    Template:

    {
      "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
      "contentVersion": "0.9.0.0",
      "parameters": {
        "clusterName": {
          "type": "string",
          "metadata": {
            "description": "The name of the HDInsight cluster to create."
          }
        },
        "clusterLoginUserName": {
          "type": "string",
          "defaultValue": "admin",
          "metadata": {
            "description": "These credentials can be used to submit jobs to the cluster and to log into cluster dashboards."
          }
        },
        "clusterLoginPassword": {
          "type": "securestring",
          "metadata": {
            "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
          }
        },
        "location": {
          "type": "string",
          "defaultValue": "northeurope",
          "metadata": {
            "description": "The location where all azure resources will be deployed."
          }
        },
        "clusterVersion": {
          "type": "string",
          "defaultValue": "3.5",
          "metadata": {
            "description": "HDInsight cluster version."
          }
        },
        "clusterWorkerNodeCount": {
          "type": "int",
          "defaultValue": 2,
          "metadata": {
            "description": "The number of nodes in the HDInsight cluster."
          }
        },
        "clusterKind": {
          "type": "string",
          "defaultValue": "HADOOP",
          "metadata": {
            "description": "The type of the HDInsight cluster to create."
          }
        },
        "sshUserName": {
          "type": "string",
          "defaultValue": "sshuser",
          "metadata": {
            "description": "These credentials can be used to remotely access the cluster."
          }
        },
        "sshPassword": {
          "type": "securestring",
          "metadata": {
            "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
          }
        },
        "identityCertificate": {
          "type": "string"
        },
        "identityCertificatePassword": {
          "type": "securestring"
        },
        "servicePrincipalApplicationId": {
          "type": "string"
        }
      },
      "resources": [
        {
          "apiVersion": "2015-03-01-preview",
          "name": "[parameters('clusterName')]",
          "type": "Microsoft.HDInsight/clusters",
          "location": "[parameters('location')]",
          "dependsOn": [],
          "properties": {
            "clusterVersion": "[parameters('clusterVersion')]",
            "osType": "Linux",
            "tier": "standard",
            "clusterDefinition": {
              "kind": "[parameters('clusterKind')]",
              "configurations": {
                "gateway": {
                  "restAuthCredential.isEnabled": true,
                  "restAuthCredential.username": "[parameters('clusterLoginUserName')]",
                  "restAuthCredential.password": "[parameters('clusterLoginPassword')]"
                },
                "core-site": {
                  "fs.defaultFS": "adl://home",
                  "dfs.adls.home.hostname": "datalakestore.azuredatalakestore.net",
                  "dfs.adls.home.mountpoint": "/clusterslake/"
                },
                "clusterIdentity": {
                  "clusterIdentity.applicationId": "[parameters('servicePrincipalApplicationId')]",
                  "clusterIdentity.certificate": "[parameters('identityCertificate')]",
                  "clusterIdentity.aadTenantId": "https://login.windows.net/***********",
                  "clusterIdentity.resourceUri": "https://management.core.windows.net/",
                  "clusterIdentity.certificatePassword": "[parameters('identityCertificatePassword')]"
                }
              }
            },
            "storageProfile": {
              "storageaccounts": []
            },
            "computeProfile": {
              "roles": [
                {
                  "name": "headnode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 2,
                  "hardwareProfile": {
                    "vmSize": "Standard_D12_V2"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                },
                {
                  "name": "workernode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 2,
                  "hardwareProfile": {
                    "vmSize": "Standard_D4_V2"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                },
                {
                  "name": "zookeepernode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 3,
                  "hardwareProfile": {
                    "vmSize": "Small"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                }
              ]
            }
          }
        }
      ]
    }


    Friday, April 28, 2017 1:45 PM
  •  Hi Duhand,

    I tried with Powershell script, But still won't work. I get the error saying "Access Denied". Please look into the powershell script attached.

    //To Create Resources
     $resourceGroupName = "demoesprg"
     New-AzureRmResourceGroup -Name $resourceGroupName -Location "East US 2"
     $dataLakeStoreName = "demoespdls"
     New-AzureRmDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeStoreName -Location "East US 2"
     Test-AzureRmDataLakeStoreAccount -Name $dataLakeStoreName
     $myrootdir = "/"
     New-AzureRmDataLakeStoreItem -Folder -AccountName $dataLakeStoreName -Path $myrootdir/clusters/demoespcluster
     
     $templatefilepath = "C:\Azure-saml\template.json"
     $SSHpass = ConvertTo-SecureString -String "Demoesp1234$" -AsPlainText -Force
      
      //Create .pfx certificate
     $certFolder = "C:\Azure-saml\certs"
     $certFilePath = "$certFolder\demoespcert.pfx"
     $certStartDate = (Get-Date).Date
     $certStartDateStr = $certStartDate.ToString("MM/dd/yyyy")
     $certEndDate = $certStartDate.AddYears(1)
     $certEndDateStr = $certEndDate.ToString("MM/dd/yyyy")
     $certName = "demoespcert"
     $certPassword = "democert123$"
     $certPasswordSecureString = ConvertTo-SecureString $certPassword -AsPlainText -Force 
     $cert = New-SelfSignedCertificate -DnsName $certName -CertStoreLocation cert:\CurrentUser\My 
     $certThumbprint = $cert.Thumbprint
     $cert = (Get-ChildItem -Path cert:\CurrentUser\My\$certThumbprint) 
     Export-PfxCertificate -Cert $cert -FilePath $certFilePath -Password $certPasswordSecureString 
     $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certFilePath, $certPasswordSecureString)
     $credential = [System.Convert]::ToBase64String($certificatePFX.GetRawCertData())
     
     //create Actice Directory Application
     $application = New-AzureRmADApplication `
         -DisplayName "ESPSPN" `
         -HomePage "https://demoespcluster.hdinsight.net" `
         -IdentifierUris "https://demoespcluster.hdinsight.net" `
         -CertValue $credential  `
         -StartDate $certificatePFX.NotBefore  `
         -EndDate $certificatePFX.NotAfter 
     Start-Sleep -Seconds 20
     
     //Create Service Principla
     $applicationId = $application.ApplicationId
     $servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $applicationId
     $objectId = $servicePrincipal.Id
     
     //Assign Permissions
     Set-AzureRmDataLakeStoreItemAclEntry -AccountName $dataLakeStoreName -Path / -AceType User -Id $objectId -Permissions All
     Set-AzureRmDataLakeStoreItemAclEntry -AccountName $dataLakeStoreName -Path /clusters -AceType User -Id $objectId -Permissions All
     Set-AzureRmDataLakeStoreItemAclEntry -AccountName $dataLakeStoreName -Path /clusters/demoespcluster -AceType User -Id $objectId -Permissions All
     
     
     //Execute Scripts
     $tenantID = (Get-AzureRmContext).Tenant.TenantId
     $secureCert = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))
     //$dsecureCert = ConvertTo-SecureString $secureCert -AsPlainText -Force
     
     New-AzureRmResourceGroupDeployment `
        -ResourceGroupName $resourceGroupName `
        -TemplateFile $templatefilepath `
        -identityCertificate $secureCert `
        -identityCertificatePassword $certPasswordSecureString `
        -clusterName  $certName `
        -clusterLoginPassword $SSHpass `
        -sshPassword $SSHpass `
        -servicePrincipalApplicationId $applicationId

    Error:

    New-AzureRmResourceGroupDeployment : 11:15:00 PM - DeploymentDocument 'AmbariConfiguration_1_7' failed the validation. Error: 'Error while getting access to the datalake
    storage account demoespdls: Access denied.
    .'

    What am i missing here?

    Regards,

    Karthick


    Monday, May 1, 2017 5:46 PM
  • cleaned up and got it up and running, attached template and PS code for others to use.

    cls
    $templatefilepath = "C:\Users\****\Desktop\hdinsighttest\hdinsWithDatalakeStoreTemplate.Json"
    $SSHpass = ConvertTo-SecureString -String "*********" -AsPlainText -Force
    $rg = "*******"
    $datelakestore = "datalakestore"
    
    $certFolder = "C:\Users\****\Documents\testjson\certificates"
    $certFilePath = "$certFolder\certFile.pfx"
    $certStartDate = (Get-Date).Date
    $certStartDateStr = $certStartDate.ToString("04/28/2017")
    $certEndDate = $certStartDate.AddYears(1)
    $certEndDateStr = $certEndDate.ToString("05/28/2017")
    $certName = "testhdinsight16"
    $certPassword = "*******"
    $certPasswordSecureString = ConvertTo-SecureString $certPassword -AsPlainText -Force
     
    mkdir $certFolder
     
    $cert = New-SelfSignedCertificate -DnsName $certName -CertStoreLocation cert:\CurrentUser\My -KeySpec KeyExchange -NotAfter $certEndDate -NotBefore $certStartDate
    $certThumbprint = $cert.Thumbprint
    $cert = (Get-ChildItem -Path cert:\CurrentUser\My\$certThumbprint)
     
    Export-PfxCertificate -Cert $cert -FilePath $certFilePath -Password $certPasswordSecureString
    
     
    $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certFilePath, $certPasswordSecureString)
    $credential = [System.Convert]::ToBase64String($certificatePFX.GetRawCertData())
     
    $application = New-AzureRmADApplication -DisplayName $certName `
                             -HomePage "https://$certName.azurehdinsight.net" -IdentifierUris "https://$certName.azurehdinsight.net" `
                             -CertValue $credential `
                             -StartDate $certStartDate -EndDate $certEndDate
    #wait of ad
    Start-Sleep -Seconds 20
                             
    $servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $application.ApplicationId
    
    Set-AzureRmDataLakeStoreItemAclEntry -AccountName $datelakestore -Path / -AceType User -Id $servicePrincipal.Id -Permissions All
    
    
    $secureCert = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))
    
    New-AzureRmResourceGroupDeployment `
        -ResourceGroupName $rg `
        -TemplateFile $templatefilepath `
        -identityCertificate $secureCert `
        -identityCertificatePassword $certPasswordSecureString `
        -clusterName  $certName `
        -clusterLoginPassword $SSHpass `
        -sshPassword $SSHpass `
        -servicePrincipalApplicationId $application.ApplicationId

    Template:

    {
      "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
      "contentVersion": "0.9.0.0",
      "parameters": {
        "clusterName": {
          "type": "string",
          "metadata": {
            "description": "The name of the HDInsight cluster to create."
          }
        },
        "clusterLoginUserName": {
          "type": "string",
          "defaultValue": "admin",
          "metadata": {
            "description": "These credentials can be used to submit jobs to the cluster and to log into cluster dashboards."
          }
        },
        "clusterLoginPassword": {
          "type": "securestring",
          "metadata": {
            "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
          }
        },
        "location": {
          "type": "string",
          "defaultValue": "northeurope",
          "metadata": {
            "description": "The location where all azure resources will be deployed."
          }
        },
        "clusterVersion": {
          "type": "string",
          "defaultValue": "3.5",
          "metadata": {
            "description": "HDInsight cluster version."
          }
        },
        "clusterWorkerNodeCount": {
          "type": "int",
          "defaultValue": 2,
          "metadata": {
            "description": "The number of nodes in the HDInsight cluster."
          }
        },
        "clusterKind": {
          "type": "string",
          "defaultValue": "HADOOP",
          "metadata": {
            "description": "The type of the HDInsight cluster to create."
          }
        },
        "sshUserName": {
          "type": "string",
          "defaultValue": "sshuser",
          "metadata": {
            "description": "These credentials can be used to remotely access the cluster."
          }
        },
        "sshPassword": {
          "type": "securestring",
          "metadata": {
            "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
          }
        },
        "identityCertificate": {
          "type": "string"
        },
        "identityCertificatePassword": {
          "type": "securestring"
        },
        "servicePrincipalApplicationId": {
          "type": "string"
        }
      },
      "resources": [
        {
          "apiVersion": "2015-03-01-preview",
          "name": "[parameters('clusterName')]",
          "type": "Microsoft.HDInsight/clusters",
          "location": "[parameters('location')]",
          "dependsOn": [],
          "properties": {
            "clusterVersion": "[parameters('clusterVersion')]",
            "osType": "Linux",
            "tier": "standard",
            "clusterDefinition": {
              "kind": "[parameters('clusterKind')]",
              "configurations": {
                "gateway": {
                  "restAuthCredential.isEnabled": true,
                  "restAuthCredential.username": "[parameters('clusterLoginUserName')]",
                  "restAuthCredential.password": "[parameters('clusterLoginPassword')]"
                },
                "core-site": {
                  "fs.defaultFS": "adl://home",
                  "dfs.adls.home.hostname": "datalakestore.azuredatalakestore.net",
                  "dfs.adls.home.mountpoint": "/clusterslake/"
                },
                "clusterIdentity": {
                  "clusterIdentity.applicationId": "[parameters('servicePrincipalApplicationId')]",
                  "clusterIdentity.certificate": "[parameters('identityCertificate')]",
                  "clusterIdentity.aadTenantId": "https://login.windows.net/***********",
                  "clusterIdentity.resourceUri": "https://management.core.windows.net/",
                  "clusterIdentity.certificatePassword": "[parameters('identityCertificatePassword')]"
                }
              }
            },
            "storageProfile": {
              "storageaccounts": []
            },
            "computeProfile": {
              "roles": [
                {
                  "name": "headnode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 2,
                  "hardwareProfile": {
                    "vmSize": "Standard_D12_V2"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                },
                {
                  "name": "workernode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 2,
                  "hardwareProfile": {
                    "vmSize": "Standard_D4_V2"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                },
                {
                  "name": "zookeepernode",
                  "minInstanceCount": 1,
                  "targetInstanceCount": 3,
                  "hardwareProfile": {
                    "vmSize": "Small"
                  },
                  "osProfile": {
                    "linuxOperatingSystemProfile": {
                      "username": "[parameters('sshUserName')]",
                      "password": "[parameters('sshPassword')]"
                    }
                  },
                  "virtualNetworkProfile": null,
                  "scriptActions": []
                }
              ]
            }
          }
        }
      ]
    }


    Thanks, the following line helped solve my issue:

    $secureCert = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))
    

    Previously I was using the GetRawCertData() instead, which was not working (error returned said no private key was found):

    $identityCertificate = [System.Convert]::ToBase64String($certificatePFX.GetRawCertData())

    Wednesday, July 5, 2017 3:22 PM