Dremio access to HDInsight cluster and Hive

Question

We are exploring the option to access a hive table or underlying data in hdfs on HDInsight cluster on Azure from Dremio client. We are currently able to access the other resources like MSSQLServer, ADLS Gen2, Blob Storage from Dremio without any issues. Could you please let me know how would we access the data on hdfs on hdinsight either as raw data or through hive (Dremio has both plugins) from Dremio accessing HDinsight ? I went through Dremio documentation of accessing hive and HDFS but couldn't get it working. Any help is much appreciated. Thank you.

Answer 1

Hello,

I’m working with the product team and get back to you when I have more information.

Meanwhile, you can raise the same question on Dremio Community.

Answer 2

Hello,

Unfortunately, Dramio does not certifies on HDInsight. 

Could you please go ahead and ask in Dremio community and share the solution, which might be beneficial to other community members reading this thread. 

Hope this helps.      

----------------------------------------------------------------------------------------

Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

Answer 3

So we are able to access ADLS gen2, Blob, SQL DB etc. hosted on Azure from Dremio. May I know if you are saying only HDInsight cannot be accessed ? Thank you.

Answer 4

Hello,

Yes, Dremio cannot be accessed on Azure HDInsight.

Dremio provisioning currently supports only on-premise hadoop installation.

For more details, refer “Dremio on Hadoop Deployment”.

Hope this helps.      

----------------------------------------------------------------------------------------

Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

Answer 5

Hello,

Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

Answer 6

We indeed were able to access HDInsight cluster from Dremio as there is a custom deployment option available.

https://www.dremio.com/tutorials/building-a-cloud-datalake-dremio-adls/

Please refer to above for details. Marking the post as completed for now as we were able to resolve it.

Answer 7

Hello,

Glad to know that your issue has resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.