none
Cannot create a Data Lake catalogue

    Question

  • Hello gurus,

    I am trying to automate deployment to Azure using the code below under my free subscription.

    $array = @("DEV","TST","PROD")
    Do {
    # PROMPT FOR USER INPUT
    $env = Read-Host -Prompt "Please enter an environment type"
    }
    Until ($env -in $array)

     

    $DLStorageAccount = "XY" + $Env.ToLower() + "datalake"
    $Location = "North Europe"

    Get-AzResourceGroup -Name $ResourceGroupName -ErrorVariable notPresent -ErrorAction SilentlyContinue
    if ($notPresent)
    {
    # ResourceGroup doesn't exist
    New-AzResourceGroup -Name $ResourceGroupName -Location $Location
    }

    Get-AzDataLakeStoreAccount -ResourceGroupName $ResourceGroupName -Name $DLStorageAccount -ErrorVariable notPresent -ErrorAction SilentlyContinue
    if ($notPresent)
    {
    # Data Lake Storage Account doesn't exist
    New-AzDataLakeStoreAccount -Name $DLStorageAccount -ResourceGroupName $ResourceGroupName -Location $Location -DisableEncryption
    Test-AzDataLakeStoreAccount -Name $DLStorageAccount
    Write-Output "New Data Lake Storage account Gen1 has been created"
    }
    else
    {
    Write-Output "Data Lake Storage account Gen1 already exists"
    Remove-AzDataLakeStoreAccount -Name $DLStorageAccount -ResourceGroupName $ResourceGroupName -Force
    New-AzDataLakeStoreAccount -ResourceGroupName $ResourceGroupName -Name $DLStorageAccount -Location $Location -DisableEncryption
    Write-Output "New Data Lake Storage account Gen1 has been created"
    }
    Get-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data" -ErrorVariable notPresent -ErrorAction SilentlyContinue
    $rootfolder = "/data"
    if ($notPresent)
    {
    # Data Lake Catalogue doesn't exist
    New-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data" -Folder -Force
    New-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data/calculated_data/" -Folder -Force
    New-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data/advertisers/" -Folder -Force
    New-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data/data_volume/" -Folder -Force
    Write-Output "New Data Lake catalogue structure has been created"
    }
    else
    {
    Write-Output "Data Lake Catalogue already exists"
    Remove-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data" -Force
    New-AzDataLakeStoreItem -AccountName $DLStorageAccount -Path "/data" -Folder -Force
    Write-Output "New Data Lake catalogue structure has been created"
    }

    But as a result I get the following error:

    New Data Lake Storage account Gen1 has been created
    New-AzDataLakeStoreItem : Error in creating Directory /data.
    Operation: MKDIRS failed with   Unknown Error: The 'User-Agent' header must be modified using the appropriate property or method.
    Parameter name: name Source: System StackTrace:    at System.Net.WebHeaderCollection.ThrowOnRestrictedHeader(String headerName)
       at System.Net.WebHeaderCollection.Set(String name, String value)
       at Microsoft.Azure.DataLake.Store.WebTransport.AssignCommonHttpHeaders(HttpWebRequest webReq, AdlsClient client, RequestOptions req, String token, String opMethod, IDictionary`2 customHeaders, Int32 
    postRequestLength)
       at Microsoft.Azure.DataLake.Store.WebTransport.<MakeSingleCallAsync>d__22.MoveNext().
    .
    Last encountered exception thrown after 5 tries. [The 'User-Agent' header must be modified using the appropriate property or method.
    Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
    Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
    Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
    Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
    Parameter name: name]
    [ServerRequestId:]

    Would you please help me with a piece of advice?

    Tuesday, April 16, 2019 4:10 PM

All replies

  • Hi Sinbat,

    You may refer SO thread which addressing similar issue.

    Hope this helps.

    Wednesday, April 17, 2019 5:44 AM
    Moderator
  • Hi Cheekatlapradeep,

    Thank you for trying to solve my problem.

    But unfortunately neither installing Powershell Core nor usage of AzureRM module helped me.

    In both cases I get errors.

    Looks like the only workaround is to overwrite the whole script using AzureRM commands since Az and AzureRM modules cannot be used together.
    Thursday, April 18, 2019 7:08 AM
  • Hi Andrei,

    You may refer GitHub issue which addressing similar issue.

    In case, if the referred GitHub issue doesn’t resolve issue, please do comment on the same GitHub issue.

    OR

    You can file a new GitHub issue in our GitHub issues page to provide the Azure PowerShell team direct feedback.

    Hope this helps.

    Tuesday, April 23, 2019 11:31 AM
    Moderator
  • Hi Andrei,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Friday, April 26, 2019 10:18 AM
    Moderator