none
MSI error while running pipeline in Azure ML RRS feed

  • Question

  • Hello,

    I have been attempting to convert my Python code from Jupyter Notebooks to run in Azure Machine Learning Services.  My first step is to create a pipeline with a Notebook runner step to run a Python Notebook.  I am now getting the following error in the 70_driver_log file:  Has anyone seen this error before?

    Executing:  11%|█         | 5/47 [00:09<01:47,  2.56s/cell]WARNING - MSI: Failed to retrieve a token from 'http://172.17.0.1:39089/MSI/token//?resource=https://management.core.windows.net/&api-version=2017-09-01' with an error of 'HTTPConnectionPool(host='172.17.0.1', port=39089): Max retries exceeded with url: /MSI/token//?resource=https://management.core.windows.net/&api-version=2017-09-01 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5c409e8470>: Failed to establish a new connection: [Errno 111] Connection refused',))'.



    At which point, after the time out period has exceeded, the pipeline fails.

    For your reference, here is my pipeline code:

    ## import all relevant libraries

    import os
    import azureml.core
    from azureml.core import Workspace, Experiment, Datastore, Environment, ScriptRunConfig, RunConfiguration
    from azureml.core.conda_dependencies import CondaDependencies
    from azureml.widgets import RunDetails

    from azureml.pipeline.core import PipelineParameter
    from azureml.pipeline.core import PipelineData
    from azureml.core.datastore import Datastore

    from azureml.contrib.notebook import NotebookRunConfig, AzureMLNotebookHandler
    from azureml.contrib.notebook import NotebookRunnerStep

    from azureml.pipeline.core import Pipeline
    from azureml.pipeline.steps import PythonScriptStep

    from azureml.core.compute import AmlCompute, ComputeTarget
    from azureml.core.compute_target import ComputeTargetException

    ## get workspace
    ws = Workspace.from_config()

    ## get the default data store
    def_blob_store = ws.get_default_datastore() 

    def_blob_store = Datastore(ws, "workspaceblobstore")
    print("Blobstore's name: {}".format(def_blob_store.name))


    aml_compute_target = "cluster-dev03"

    # Instantiate environment
    env = Environment(name="textanalyticsenv")

    conda_run_config = RunConfiguration(framework="python")

    conda_run_config.environment.docker.enabled = True
    conda_run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE
    conda_run_config.environment = env
    cd = CondaDependencies.create(pip_packages=['azureml-sdk','azureml-core==1.0.85'],conda_packages=['scikit-learn','nltk','pandas','numpy','spacy','textblob'])
    conda_run_config.environment.python.conda_dependencies = cd




    print('run config is ready')

    source_directory = "./"

    handler = AzureMLNotebookHandler(timeout=600, progress_bar=False, log_output=True)

    cfg = NotebookRunConfig(source_directory=source_directory, notebook="process_text.ipynb",
                            handler = handler,
                            parameters={"arg1": "Machine Learning"},
                            run_config=conda_run_config)

    my_pipeline_param = PipelineParameter(name="pipeline_param", default_value="my_param")

    print('Source directory for the step is {}.'.format(os.path.realpath(source_directory)))
    notebook_runner_step = NotebookRunnerStep(name="training_notebook_step",
                                              notebook_run_config=cfg,
                                              params={"my_pipeline_param": my_pipeline_param},
                                              #inputs=[output_data],
                                              #outputs=[output_from_notebook],
                                              allow_reuse=True,
                                              compute_target="cluster-dev03",
                                              output_notebook_pipeline_data_name="notebook_result")

    print("Notebook Runner Step is Created.")

    print("Notebook Run Config is created.")

    pipeline = Pipeline(workspace=ws, steps=[notebook_runner_step])

    pipeline.validate()
    print("Pipeline validation complete")

    pipeline_run1 = Experiment(ws, 'process_text_data').submit( pipeline, regenerate_outputs=False)
    print("Pipeline is submitted for execution")

    Thank-you

    Melinda

    Tuesday, February 11, 2020 12:21 AM

All replies

  • Hi Melinda,

    Is this error persistent i.e does this occur when you try to run again?

    Does this error occur before the notebook_runner_step timeout expires? 

    -Rohit

    Wednesday, February 12, 2020 8:45 AM
    Moderator