none
HDInsight ESP Spark cluster 3.6 RRS feed

  • Question

  • Can anyone please confirm the role of LLAP in ESP Spark 3.6. Does it only provides methodology to enforce policies for security and does not actually cache data.


    • Edited by Rahul092 Monday, October 14, 2019 2:57 PM
    Monday, October 14, 2019 2:56 PM

All replies

  • Hi Rahul,

    I’m working with the product team and get back to you when I have more information.

    Tuesday, October 15, 2019 9:53 AM
    Moderator
  • Thank you very much.
    Tuesday, October 15, 2019 11:35 AM
  • Hi Rahul,

    Yes. This is correct. The LLAP on ESP Spark clusters is configured with very little memory. It can still cache on the SSD but even that is much smaller than on a regular LLAP cluster.

    Hope this helps.      

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Wednesday, October 16, 2019 9:23 AM
    Moderator
  • Thank you very much for you response.

    In this case, do we need to assign 50% of cluster memory to LLAP daemons? If not, can you please check with the product team how much memory is sufficient to assign to LLAP daemons per worker node?

    Wednesday, October 16, 2019 1:01 PM
  • Hi Rahul,

    How many memory need to assign to LLAP daemons depending on your workload. You can give it a try to see how it works. However, we don't recommend you to use LLAP directly on Spark cluster, it may impact both your iteractive Hive and Spark worklaod due to resource conflict. You could create an LLAP cluster specifically for your interactive Hive workloads. 

    Thursday, October 24, 2019 3:50 PM
  • Hi CHEEKATLAPRADEEP,

    Thank you very much for your response.

    As per my understanding, LLAP comes by default whenever we create Spark ESP cluster. Is there any way to disable this functionality on Spark ESP cluster? And if it is then would it disable Apache ranger policies.

    Friday, October 25, 2019 8:50 AM
  • Hi Rahul,

    You cannot disable the LLAP functionality on Spark ESP cluster as we use it to implement security related features, like Ranger policies. But we keep it running with minimum resource so that your Spark workload will not be impacted by this. Our suggestion is not to disable this on ESP Spark cluster as it will break security features. The best practice is to keep the configs for LLAP on ESP Spark cluster as it is, and not use it for your interactive workload. If you need interactive workload, creating a separate LLAP cluster would be a better choice.

    Hope this helps.      

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Wednesday, November 6, 2019 4:24 AM
    Moderator
  • Hi Rahul,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Tuesday, November 12, 2019 6:27 AM
    Moderator