locked
Hive query against Table Storage RRS feed

  • Question

  • It would appear from this blog entry (http://blogs.msdn.com/b/mostlytrue/archive/2014/04/04/analyzing-azure-table-storage-data-with-hdinsight.aspx) that it is certainly possible to get an HDInsight cluster to use data directly from Table Storage rather than just Blob Storage in a hive query. Is this possible to do this via Data Factories or must we do a Copy Activity first to get the data from table storage into a blob?

    Thx,
    Ben

    Monday, April 13, 2015 2:19 AM

Answers

  • Both might work.

    You could to the Copy Activity from Table to Blob to simplify and keep it distinct from the Hive query to keep if needed. Table to Blob is supported as per http://azure.microsoft.com/en-us/documentation/articles/data-factory-copy-activity/

    The key to the Mostafa's blog post is that you are writing a .Jar to deploy to Azure Blob storage, then that HDInsight can be configured in a custom approach to point to that .Jar file in storage and reference it in the Hive query. The .jar you are compiling from the .git has the magic to talk to Table storage from Hive.

    The ADF caller doesn't know much about what is inside your .hql hive script, so that should be agnostic.

    If you are going to Bring your own HDInsight cluster (outside of ADF) then you would follow the blog code sample to provision the customized Jar path in HDInsight.

    If you are going to use ADF's capability to do on-demand HDInsight, the ADF caller code would have to know how to do the custom Hive configuration steps in your HDInsight cluster. The JSON reference here mentions the hiveConfiguration example https://msdn.microsoft.com/en-us/library/azure/dn893526.aspx I don't see a clear example on setting additional storage paths and AdditionaLibraries options.

    Thanks, Jason


    Didn't get enough help here? Submit a case with the Microsoft Customer Support teams for deeper investigation - Azure service support: https://manage.windowsazure.com/?getsupport=true For on Premise software support go here instead: http://support.microsoft.com/select/default.aspx?target=assistance

    Friday, April 24, 2015 11:39 PM