locked
Trying to mount data lake gen 2 storage account in Azure Databricks RRS feed

  • Question

  • keep getting this error:

    abfss://https:/testsa747.dfs.core.windows.net has invalid authority.

    Wednesday, March 18, 2020 7:41 PM

Answers

  • Hi Svitesh,

    In order investigate further, could you please provide more details on the code which you are executing and also share the complete error message?

    Meanwhile, you may try the below steps to mount Azure Data Lake Gen2 storage account in Azure Databricks.

    To mount an Azure Data Lake Storage Gen2 or a folder inside a container, use the following command:

    configs = {"fs.azure.account.auth.type": "OAuth",
           "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
           "fs.azure.account.oauth2.client.id": "0xxxxx7-cxxxxe-af4c-9xxxxxxxxxef", #Enter <appId> = Application ID
           "fs.azure.account.oauth2.client.secret": "AxxxxxxxxxMt]*", #Enter <password> = Client Secret created in AAD
           "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/72f9xxxf-8xxx-41xx-91ab-2dxxx47/oauth2/token", #Enter <tenant> = Tenant ID
           "fs.azure.createRemoteFileSystemDuringInitialization": "true"}
    
    dbutils.fs.mount(
    source = "abfss://filesystem@chepragen2.dfs.core.windows.net/flightdata", #Enter <container-name> = filesystem name <storage-account-name> = storage name
    mount_point = "/mnt/flightdata",
    extra_configs = configs)

    Access files in your filesystem as if they were local files, for example:

    df = spark.read.csv("/mnt/flightdata/flightdata.csv", header="true")
    display(df)

    For more details, refer “Azure Databricks – Azure Data Lake Storage Gen2”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Thursday, March 19, 2020 6:57 AM

All replies

  • Hi Svitesh,

    In order investigate further, could you please provide more details on the code which you are executing and also share the complete error message?

    Meanwhile, you may try the below steps to mount Azure Data Lake Gen2 storage account in Azure Databricks.

    To mount an Azure Data Lake Storage Gen2 or a folder inside a container, use the following command:

    configs = {"fs.azure.account.auth.type": "OAuth",
           "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
           "fs.azure.account.oauth2.client.id": "0xxxxx7-cxxxxe-af4c-9xxxxxxxxxef", #Enter <appId> = Application ID
           "fs.azure.account.oauth2.client.secret": "AxxxxxxxxxMt]*", #Enter <password> = Client Secret created in AAD
           "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/72f9xxxf-8xxx-41xx-91ab-2dxxx47/oauth2/token", #Enter <tenant> = Tenant ID
           "fs.azure.createRemoteFileSystemDuringInitialization": "true"}
    
    dbutils.fs.mount(
    source = "abfss://filesystem@chepragen2.dfs.core.windows.net/flightdata", #Enter <container-name> = filesystem name <storage-account-name> = storage name
    mount_point = "/mnt/flightdata",
    extra_configs = configs)

    Access files in your filesystem as if they were local files, for example:

    df = spark.read.csv("/mnt/flightdata/flightdata.csv", header="true")
    display(df)

    For more details, refer “Azure Databricks – Azure Data Lake Storage Gen2”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Thursday, March 19, 2020 6:57 AM
  • Thanks for the reply I made a silly mistake at source. I pasted the primary ADLS file system endpoint from properties instead of looking at what it was asking.

    Thanks again.

    Thursday, March 19, 2020 5:15 PM
  • Hi Svitesh,

    Glad to know that it helped.

    If the above answers were helpful, click “Mark as Answer” or “Up-Vote”, which might be beneficial to other community members reading this thread. 

    Friday, March 20, 2020 3:57 AM