none
HDInsight SDK: Conenct to local Cluster Error

    Question

  • Hi there,

    I followed http://social.msdn.microsoft.com/Forums/windowsazure/de-de/8179e16d-a874-41ec-b253-1021f4c0ed6c/hdinsight-sdk-connecting-to-local-cluster, but did not managed to connect to a local cluster. I specified my Azure Storage Account. A container was created successfully, but my job failed with the following error:

    
    
    org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container 4347d0cc-04c6-469c-be2d-9049d60aa1e1 in account hadooppartner not found, and we can't create  it using anoynomous credentials.
    	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:682)
    	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:252)

    I have run the following .NET code:

    var hiveConnection = new HiveConnection(new Uri("http://hadoop-poc.cloudapp.net:50111"), "hadoop", "PASS", "hadoop", "StorageAccountName", "StorageKey");
                var metaData = hiveConnection.GetMetaData().Result;
                var result = hiveConnection.ExecuteQuery(@"select * from customer limit 1");
                result.Wait();

    Tuesday, July 02, 2013 9:38 AM

Answers

  • In the "StorageAccountName" in your HiveConnection constructor, are you using the full name, i.e. hadooppartner.blob.windows.core.net? If I don't do that I get the error you are seeing, but otherwise was able to get it to work on my local install. The only change I made was to the core-site.xml as you noted, restarted the services, then ran your code with my own query.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, July 03, 2013 11:02 PM

All replies

  • Please note that I have installed HDInsight Server on a Azure VM myself, and I am not using Azure HDInsight Service.

    In the meantime I tried to add the credentials to the core-site.xml, but without success. The job is failing with the same error.

    <property>
    <name>fs.azure.account.key.hadooppartner.blob.core.windows.net</name>
    <value>Key here</value>
    </property>

    However I can list the contents of the directory using the command line. I have also reastarted all services on the hadoop machine.

    c:\Hadoop\hadoop-1.1.0-SNAPSHOT>hadoop fs -ls asv://40781c2b-639d-4caf-88dd-9f7e765f07d6@hadooppartner.blob.core.windows.net/

    Wednesday, July 03, 2013 12:28 PM
  • In the "StorageAccountName" in your HiveConnection constructor, are you using the full name, i.e. hadooppartner.blob.windows.core.net? If I don't do that I get the error you are seeing, but otherwise was able to get it to work on my local install. The only change I made was to the core-site.xml as you noted, restarted the services, then ran your code with my own query.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, July 03, 2013 11:02 PM
  • Sarah, thank you for taking time to reproduce my issue. You actually fixed the problem. 
    Thursday, July 04, 2013 6:09 AM
  • Do you have an idea why this doesn't work for me?

    I'm trying to query an external table which is outside the default HDInsight container. I've added the credentials to core-site.xml and restarted Hive's daemons, but I'm still getting this error:

    Container ... in account ... not found, and we can't create  it using anoynomous credentials.

    I've been struggling with this for hours... Any ideas?


    • Edited by IlyaKogan Sunday, December 01, 2013 3:00 PM
    Sunday, December 01, 2013 2:42 PM