Difference in HDInsight clusters - Insight requested RRS feed

  • Question

  • We have multiple options available for the HDInsight clusters.

    I am aware of the below documentation but still need some insights.

    For example, creating a Hadoop cluster means we cannot run spark commands on the hive tables ? Should we have a spark cluster to be able to run spark ? How would we access the hive tables created in the Hadoop cluster using another spark cluster and how would these clusters interact with each other ? Any help is appreciated.

    Monday, August 26, 2019 4:46 PM


  • Hello,

    To interact clusters with each other, you should deploy Hadoop and Spark cluster using the custom metastore.

    Custom Metastore lets you attach multiple clusters and cluster types to same Metastore. Example – Single Metastore can be shared across Interactive Hive, Hive and Spark clusters in HDInsight.


    For more details, refer “Use external metadata stores in Azure HDInsight”.  

    And also, you may refer the SO thread, which addressing similar issue.

    Hope this helps.

    Tuesday, August 27, 2019 5:24 AM