none
How to use SAS token in core-site.xml in Azure VM

    Question

  • I have create Red hat Linux VM in Azure and installed my Java application in it.

    I have created Storage account with container.

    Also I have created SAS token for storage account on Azure Portal.

      

    core-site.xml

      <property>         
    
    <name>fs.azure.sas.CONTAINER_NAME.ACCOUNT_NAME.blob.core.windows.net</name>    
    
    <value>sv%3D2015-12-11%26ss%3Db%26srt%3Dsco%26sp%3Drwdlac%26se%3D2017-01-05T12%3A04%3A00Z%26st%3D2017-01-05T11%3A54%3A00Z%26spr%3Dhttps%26sig%3DEUWHyuVzv5CRlvSmg6jRuEfa8RnTNi7FLsdfcdspL6EurLMzY%253D</value>  
      </property>   
    
    
      <property>         
    
     <name>fs.defaultFS</name>          
    
    <value>wasb://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net</value>          <final>true</final>        
    
    </property>

    I am getting following error when I ran my application:-

    Caused by: org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details.    at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.mkdirs(NativeAzureFileSystem.java:1868)    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.mkdirs(NativeAzureFileSystem.java:1834)    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)    at com.myapp.hadoop.common.HdfsFileSystem$1.run(HdfsFileSystem.java:98)    at com.myapp.hadoop.common.HdfsFileSystem$1.run(HdfsFileSystem.java:91)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)    at com.myapp.hadoop.common.HdfsFileSystem.__initialize(HdfsFileSystem.java:91)    at com.myapp.hadoop.common.SimpleHdfsFileSystem.initialize(SimpleHdfsFileSystem.java:40)    ... 47 more
    
    Caused by: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details.    at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)    at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)    at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)    ... 57 more
    
    Caused by: com.microsoft.azure.storage.StorageException: The specified resource does not exist.

    Friday, January 6, 2017 11:00 AM

All replies

  • Hi,

    Thank you for contacting Microsoft forums. We are pleased to answer your query.

    Can you make sure our storage account supports page blobs?  The library he’s supporting only supports page blobs (see https://hadoop.apache.org/docs/stable2/hadoop-azure/index.html). 

    The make directory command that isn’t running isn’t an Azure function, it’s a function of the library he’s using that updates a set of metadata stored in an XML file in a blob in Azure.  It’s most likely a problem with manipulating that file, again possibly because the storage account may be the wrong type (some storage accounts only support block blobs).

    See; if you can find which is familiar with that particular library or use a tool like Azure Storage Explorer to look at the account and see what data (if any) has been stored.


    I hope that the reply will assist you in getting your query addressed. In case you require further assistance, please do reply to the thread as we are always available to your queries.

    Regards,
    Sumanth BM

    Friday, January 6, 2017 4:04 PM
    Moderator
  • I am connecting WASB storage account directly with Azure VM .To authenticate storage accounts, I am using the following methods to achieve it:-

    Type of authentication methods for storage account:-

    1. Storage Access Key (two keys available)

    I have used storage access key like below in core-site.xml and its works well.

    <property>
      <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
      <value>YOUR ACCESS KEY</value>
    </property>

    <property>
      <name>fs.defaultFS</name>
      <value>wasb://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net</value>
      <final>true</final>
    </property>

    2. Shared Access Signatures (SAS)

    I have generated SAS token in Azure Portal and trying to use generated SAS token in core-site.xml. But I am not able to succeed with it. I posted my error in question above.

    <property>         

        <name>fs.azure.sas.CONTAINER_NAME.ACCOUNT_NAME.blob.core.windows.net</name>    

        <value>sv%3D2015-12-11%26ss%3Db%26srt%3Dsco%26sp%3Drwdlac%26se%3D2017-01-05T12%3A04%3A00Z%26st%3D2017-01-05T11%3A54%3A00Z%26spr%3Dhttps%26sig%3DEUWHyuVzv5CRlvSmg6jRuEfa8RnTNi7FLsdfcdspL6EurLMzY%253D</value>

    </property>   


    <property>         

        <name>fs.defaultFS</name>          

        <value>wasb://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net</value>
        <final>true</final>        

    </property>

    I have used following jars to connect my application on VM with WASB storage:-

    • http://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/2.2.0
    • http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/2.7.3

    Let me know if you need clarity on my question.

    Kindly help to solve this ?

    • Edited by KaranM Monday, January 9, 2017 8:42 AM
    Monday, January 9, 2017 7:52 AM
  • Hi Karan,

    Thank you for contacting Microsoft forums. We are pleased to answer your query.

    We are checking on the query and will get back to you soon on this.

    I apologize for the inconvenience and appreciate your time and patience in this matter.

    Regards,
    Sumanth BM

    Tuesday, January 10, 2017 2:55 PM
    Moderator
  • I notice that your SAS token is only valid for 10 minutes. Is it possible that it's expired by the time you use it? If you find the failing request ID in your storage analytics logs and provide it then we can check the exact reason for the failure.
    Tuesday, January 17, 2017 9:26 AM
  • I have uploaded image to /CONTAINERNAME/sample/ via Storage Explorer

    I have generated SAS token with time from Start Thu, 05 Jan 2017 11:54:00 GMT - Expiry Thu, 05 Jan 2017 12:04:00 GMT with Blob Service alone on Storage Account in Azure Portal.

    I am able to see the uploaded image by this below URL:-

    https://STORAGEACCOUNTNAME.blob.core.windows.net/CONTAINERNAME/sample/logo.png?sv=2015-12-11&ss=b&srt=sco&sp=rwdlac&se=2017-01-05T12:04:00Z&st=2017-01-05T11:54:00Z&spr=https&sig=EUWHyuVzv5CRlvSmg6jRuEfa8RnTNi7FLpL6EurLMzY%3D
    

    As my SAS key is valid for ten minutes, After 10 minutes, I get below error as follows:-

    <Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
    RequestId:f66e72a7-0001-0048-7152-6714e2000000
    Time:2017-01-05T12:51:22.5600588Z</Message><AuthenticationErrorDetail>Signature not valid in the specified time frame: Start [Thu, 05 Jan 2017 11:54:00 GMT] - Expiry [Thu, 05 Jan 2017 12:04:00 GMT] - Current [Thu, 05 Jan 2017 12:51:22 GMT]</AuthenticationErrorDetail></Error>

    My SAS token is working as expected. I don't have any problem in it.

    Will I be able to use SAS token in xml file to access Storage account. Refer xml below:-

    <property>         

        <name>fs.azure.sas.CONTAINER_NAME.ACCOUNT_NAME.blob.core.windows.net</name>    

        <value>sv%3D2015-12-11%26ss%3Db%26srt%3Dsco%26sp%3Drwdlac%26se%3D2017-01-05T12%3A04%3A00Z%26st%3D2017-01-05T11%3A54%3A00Z%26spr%3Dhttps%26sig%3DEUWHyuVzv5CRlvSmg6jRuEfa8RnTNi7FLsdfcdspL6EurLMzY%253D</value>

    </property>   


    <property>         

        <name>fs.defaultFS</name>          

        <value>wasb://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net</value>
        <final>true</final>        

    </property>

    Tuesday, January 17, 2017 11:06 AM
  • Thanks for explaining. I misunderstood the question. We are engaging the appropriate team to investigate whether SAS is supported in this configuration.
    Tuesday, January 17, 2017 7:26 PM