none
List a blob container from HDInsight Cluster Head Node shell

    Question

  • Hi,

    I'm connected on the Head node shell of a HDInsight Cluster. I have a blob container named "working" inside a storage account and I'm trying to access this container in order to get a file. I'm performing this command :

    user@hn0-lambda:~$ ls asv://working@storageaccount.blob.core.windows.net/

    ls: cannot access asv://working@storageaccount.blob.core.windows.net/: No such file or directory


    Please help me to understand what's wrong.

    Many thanks.

    Rene

    Saturday, November 5, 2016 10:28 PM

All replies

  • Can you try using wasb:// protocol instead? asv:// protocol is already depreacated and I guess the latest version does not support it.

    Where do you see the asv:// come from? Maybe we need to update the content here...

    Sunday, November 6, 2016 7:01 AM
  • Hi Xiaoyong Zhu.

    Thank you for your update.

    Indeed I was following old MS Azure blogs that were teaching me to use asv://.

    The documentation (https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/) shows this:

    Note:
    In versions of HDInsight earlier than 3.0, asv:// was used instead of wasb://. asv:// should not be used with HDInsight clusters 3.0 or higher, as it will result in an error.

    But, even I use the wasb protocol it is not possible to have access to the blob container:

    user@hn0-lambda:~$ ls wasb://working@storageaccount.blob.core.windows.net/
    ls: cannot access wasb://working@storageaccount.blob.core.windows.net/: No such file or directory

    This storage account is the cluster's default storage account. Both, the storage account and the HDInsight cluster are located at East US 2 location and are under the same resource group. The storage's replication type is "Locally-redundant storage (LRS)".

    Thank you in advance for any help.

    Best,
    Rene

    Sunday, November 6, 2016 11:16 AM
  • Oops - sorry I didn't notice you were using ls command. So ls command is part of Linux system, which does not support HDFS/WASB protocols. So in order to list files in WASB file or HDFS file, you need to use the hadoop file system command.

    In this case, you need to use:

    hadoop fs -ls wasb://working@storageaccount.blob.core.windows.net/

    For a full list of hadoop file system commands, you can refer to the doc here (not all of them are supported in WASB though but most of them should work as expected):

    https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html

    Feel free to let me know (xiaoyzhu at microsoft dot com) if you have further questions.

    Sunday, November 6, 2016 12:33 PM
  • Hi  Xiaoyong Zhu,

    Many thanks for your help and kindness.

    "hadoop fs -ls wasb://working@storageaccount.blob.core.windows.net/ " worked fine.

    Best,

    Rene

    Sunday, November 6, 2016 1:34 PM