none
How to encrypt Storage Access Key and use it

    Question

  • I have created Azure VM and installed my Java application in it and then connected to WASB storage.

    I have added following jars and core-site.xml to access WASB storage from Java application.

     - [azure-storage][1]
     -  [hadoop-azure][2]

    **core-site.xml**

        <configuration>
        
            <property>
              <name>fs.AbstractFileSystem.wasb.impl</name>
              <value>org.apache.hadoop.fs.azure.Wasb</value>
            </property>
        
        
           <property>
              <name>fs.azure.account.key.STORAGE_ACCOUNT_NAME .blob.core.windows.net</name>
               <value>STORAGE ACCESS KEY</value>
           </property>
        
            <property>
              <name>fs.azure.io.copyblob.retry.max.retries</name>
              <value>60</value>
            </property>
        
            <property>
              <name>fs.azure.io.read.tolerate.concurrent.append</name>
              <value>true</value>
            </property>
        
            <property>
              <name>fs.azure.page.blob.dir</name>
              <value>/mapreducestaging,/atshistory,/tezstaging,/ams/hbase/WALs,/ams/hbase/oldWALs,/ams/hbase/MasterProcWALs</value>
            </property>
        
            <property>
              <name>fs.defaultFS</name>
              <value>wasb://STORAGE_CONTAINER_NAME@STORAGE_ACCOUNT_NAME.blob.core.windows.net</value>
              <final>true</final>
            </property>
        
            <property>
              <name>fs.trash.interval</name>
              <value>360</value>
            </property>
        
          </configuration>


      http://mvnrepository.com/artifact/com.microsoft.azure/azure-storage
      http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure


    I have used Storage Access Key directly in core-site.xml. But I want my Access key to be encrypted.

    When I search about it, I got to know about below script:-

        <property>
          <name>fs.azure.account.keyprovider.youraccount</name>
          <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value>
        </property>
        
        <property>
          <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
          <value>YOUR ENCRYPTED ACCESS KEY</value>
        </property>
        
        <property>
          <name>fs.azure.shellkeyprovider.script</name>
          <value>PATH TO DECRYPTION PROGRAM</value>
        </property>


    How do I encrypt my Key and use it in above xml and decrypt it ?

    Note:- I am connecting Azure VM directly to WASB storage without using HDInsight Cluster.
    Tuesday, January 17, 2017 11:16 AM

All replies

  • In addition, I have tried the following:-

    I have created a HDInsight Cluster as cluster1 with storage account as storage1

    I have created Azure VM as vm1 and installed my Java application in it.

    But I want to use storage1 in to my application without need of cluster1.

    I have configured following in core-site.xml to access WASB from VM with encrypted key.

    <property>      <name>fs.azure.account.key.palmstorage4.blob.core.windows.net</name>   
    <value>ENCRYPTED KEY</value>
    </property>
    
    <property>      <name>fs.azure.account.keyprovider.palmstorage4.blob.core.windows.net
    </name>      <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value>
    </property>
    
    <property>      
    <name>fs.azure.shellkeyprovider.script
    </name>      <value>PATH TO DECRYPTION PROGRAM</value>
    </property> 
    
    <property>     
     <name>fs.defaultFS</name>      <value>wasbs://CONTAINERNAME@STORAGENAME.blob.core.windows.net</value>     
     <final>true</final>  
    </property>


    ENCRYPTED KEY -  For testing, I get this generated encrypted key from core-site.xml from cluster1 which is associated with storage1

    PATH TO DECRYPTION PROGRAM - I have used decryption program shipped with cluster1 (/usr/lib/python2.7/dist-packages/hdinsight_common/decrypt.sh).

    Now I will created new storage account and I need to encrypt the access key. How do I encrypt it ? I don't want to create unnecessary cluster. For decryption, I have decrypt.sh mentioned above.


    Tuesday, January 17, 2017 1:28 PM
  • Greetings Karan,
    Thank you for contacting Microsoft forums. We are pleased to answer your query.

    You can try using Shared Access Signature. A SAS is a secure way of sharing your Azure Storage resources without compromising your storage account keys.

    I hope that the reply will assist you in getting your query addressed. In case you require further assistance, please do reply to the thread as we are always available to your queries.


    Regards.

    Md. Shihab

    **********************************************************************

    Please remember to click "Mark as Answer" on the post that helps you as this can be beneficial to other community members reading the thread. And vote as helpful.

    Wednesday, January 18, 2017 5:31 AM
  • I need to encrypt and decrypt my Storage Access Key. For decryption, I use this /usr/lib/python2.7/dist-packages/hdinsight_common/decrypt.sh .

    How do I encrypt my Storage Access Key ?

    Wednesday, January 18, 2017 7:24 AM
  • Hi,


    You may try the steps in this article to encrypt storage access keys and see if it would work. Please note that we haven’t tested this set-up.

    Disclaimer:This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.


    Regards.

    Md. Shihab

    **********************************************************************

    Please remember to click "Mark as Answer" on the post that helps you as this can be beneficial to other community members reading the thread. And vote as helpful.

    • Proposed as answer by Md Shihab Wednesday, January 25, 2017 7:20 AM
    Sunday, January 22, 2017 8:35 AM