none
adl,wasb and webhdfs

    Question

  • Hi,

    we have just started exploring HDInsight/Azure Data Lake and i am trying to understand the key terms around adl:, wasb: and webhdfs.

    Is adl:// and wasb:// built on top of webhdfs? or there is no relationship at all between this three?

    Thanks

    MS

    Friday, November 17, 2017 9:07 PM

Answers

  • Hi,

    Yes, there is a relationship.  ADLS exposes a webhdfs-compatible REST interface so applications or services that use WebHDFS APIs can easily integrate with ADLS.  However, there is a new filesystem (adl://) built on top of webhdfs that has more performance optimization not currently available in WebHDFS.  So we recommend you use adl:/// but if you are maintaining existing code you can continue to use WebHDFS APIs directly.

    wasb:// is the filesystem for Azure Storage blobs and is also an extension built on top of the HDFS APIs.

    Thanks,

    Elizabeth, Program Manager on the Azure Data Lake Store team


    • Marked as answer by MS_Prog Wednesday, December 6, 2017 11:17 PM
    Tuesday, December 5, 2017 9:11 PM
  • You can access your adls data using adl:// with any ADLS tools and SDKs (.NET, Java, etc.) anywhere.  Azure CLI supports non-windows OS for example.  There is also a filesystem class for adl:// committed to Apache to support the ADLS filesystem.  But if you want to use open-source software as-is over ADLS, then you can use the webhdfs APIs directly.

    Thanks,

    Elizabeth

    • Marked as answer by MS_Prog Wednesday, December 6, 2017 11:17 PM
    Wednesday, December 6, 2017 11:14 PM

All replies

  • Hi,

    Yes, there is a relationship.  ADLS exposes a webhdfs-compatible REST interface so applications or services that use WebHDFS APIs can easily integrate with ADLS.  However, there is a new filesystem (adl://) built on top of webhdfs that has more performance optimization not currently available in WebHDFS.  So we recommend you use adl:/// but if you are maintaining existing code you can continue to use WebHDFS APIs directly.

    wasb:// is the filesystem for Azure Storage blobs and is also an extension built on top of the HDFS APIs.

    Thanks,

    Elizabeth, Program Manager on the Azure Data Lake Store team


    • Marked as answer by MS_Prog Wednesday, December 6, 2017 11:17 PM
    Tuesday, December 5, 2017 9:11 PM
  • Thanks Elizabeth So if I understand right : I use adl:// from within the Azure ecosystem. If I want to write files from on premises systems (windows based or not) to Adls then I use the webhdfs api?
    Tuesday, December 5, 2017 10:14 PM
  • You can access your adls data using adl:// with any ADLS tools and SDKs (.NET, Java, etc.) anywhere.  Azure CLI supports non-windows OS for example.  There is also a filesystem class for adl:// committed to Apache to support the ADLS filesystem.  But if you want to use open-source software as-is over ADLS, then you can use the webhdfs APIs directly.

    Thanks,

    Elizabeth

    • Marked as answer by MS_Prog Wednesday, December 6, 2017 11:17 PM
    Wednesday, December 6, 2017 11:14 PM