How to run shell commands from Jupyter Notebook? RRS feed

  • Question

  • Is there any way to run shell commands such as conda from Jupyter Notebook in Azure HDInsight?


    Thursday, October 10, 2019 8:23 AM

All replies

  • Hi Maryam_Lewan,

    You can use script actions to install external python packages for Jupyter notebooks in Apache Spark on HDInsight.

    This article helps how to use Script Actions to configure an Apache Spark cluster on HDInsight to use external, community-contributed python packages that are not included out-of-the-box in the cluster.

    Hope this helps.      


    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Thursday, October 10, 2019 10:07 AM
  • Thanks. But it's extremely slow. I have started the following in one cluster for more than two hours, but nothing happening: 

    It is same for any other package. 

    Thursday, October 10, 2019 11:12 AM
  • Hi Maryam_lewan,

    You are expected to install as a custom script action.

    • From the Azure portal, navigate to your cluster.
    • With your cluster selected, from the left pane, under Settings, select Script actions.
    • Select + Submit new.
    • Enter the following values for the Submit script action window:

    • Select Create. Visit the documentation on how to use custom script actions.
    • Wait for the script to complete. The Script actions pane will state New script actions can be submitted after the current cluster operation finishes while the script is executing. A progress bar can be viewed from the Ambari UI Background Operations window.

    Hope this helps.

    Friday, October 11, 2019 9:01 AM
  • Hi Maryam_Lewan,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Monday, October 14, 2019 8:24 AM
  • Hi Maryam_lewan,

    Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

    Tuesday, November 5, 2019 6:29 AM