local compute target always results in creating new local environment RRS feed

  • Question

  • Hello,

    I am running the code example (pasted at bottom), it does use the local compute and completes successfully but instead of using existing local environment named "myenv" it creates a new environment, downloads and installs packages in it first.

    Is it not supposed to utilize the existing environment locally instead of preparing the new one? Am I missing anything here?

    Log File 60.control_log.txt show that conda_env_checker.bat does not find the environment (although different name) and then conda_env_builder.bat script kicks in to build new environment. Here are exact log detail:

    Streaming log file azureml-logs/60_control_log.txt
    Running: ['cmd.exe', '/c', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\azureml_runs\\train-on-local_1588961369_5651b1d3\\azureml-environment-setup/conda_env_checker.bat']
    Starting the daemon thread to refresh tokens in background for process with pid = 275792
    Materialized conda environment not found on target: C:\Users\Administrator/.azureml/envs/azureml_da3e97fcb51801118b8e80207f3e01ad

    Logging experiment preparation status in history service.
    Running: ['cmd.exe', '/c', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\azureml_runs\\train-on-local_1588961369_5651b1d3\\azureml-environment-setup/conda_env_builder.bat']
    Running: ['conda', '--version']
    conda 4.8.3

    from azureml.core import ScriptRunConfig, Experiment
    from azureml.core.environment import Environment

    exp = Experiment(name="myexp", workspace = ws)
    # Instantiate environment
    myenv = Environment(name="myenv")

    # Add training script to run config
    runconfig = ScriptRunConfig(source_directory=".", script="train.py")

    # Attach compute target to run config
    runconfig.run_config.target = "local"

    # Attach environment to run config
    runconfig.run_config.environment = myenv

    # Submit run 
    run = exp.submit(runconfig)


    Friday, May 8, 2020 6:35 PM

All replies

  • Hi,

    Please follow the below to configure a development environment for local computer.

    Create your own environment

    You can create an environment by instantiating Environment object and then setting its attributes: set of Python packages, environment variables and others.

    Add Python packages

    The recommended way is to specify Conda packages, as they typically come with complete set of pre-built binaries.

    In [ ]:
    from azureml.core.environment import CondaDependencies
    myenv = Environment(name="myenv")
    conda_dep = CondaDependencies()

    You can also add pip packages, and specify the version of package

    In [ ]:

    Specify environment variables

    You can add environment variables to your environment. These then become available using os.environ.get in your training script.

    In [ ]:
    myenv.environment_variables = {"MESSAGE":"Hello from Azure Machine Learning"}

    Submit run using environment

    When you submit a run, you can specify which environment to use.

    On the first run in given environment, Azure ML spends some time building the environment. On the subsequent runs, Azure ML keeps track of changes and uses the existing environment, resulting in faster run completion.

    In [ ]:
    from azureml.core import ScriptRunConfig, Experiment
    myexp = Experiment(workspace=ws, name = "environment-example")

    To submit a run, create a run configuration that combines the script file and environment, and pass it to Experiment.submit. In this example, the script is submitted to local computer, but you can specify other compute targets such as remote clusters as well.

    In [ ]:
    runconfig = ScriptRunConfig(source_directory=".", script="example.py")
    runconfig.run_config.target = "local"
    runconfig.run_config.environment = myenv
    run = myexp.submit(config=runconfig)

    To audit the environment used by for a run, you can use get_environement.

    In [ ]:

    List and get existing environments

    Your workspace contains a dictionary of registered environments. You can then use Environment.get to retrieve a specific environment with specific version.

    In [ ]:
    for name,env in ws.environments.items():
        print("Name {} \t version {}".format(name,env.version))
    restored_environment = Environment.get(workspace=ws,name="myenv",version="1")
    print("Attributes of restored environment")

    Other ways to create environments

    From existing Conda environment

    You can create an environment from existing conda environment. This make it easy to reuse your local interactive environment in Azure ML remote runs. For example, if you've created conda environment using

    conda create -n mycondaenv

    you can create Azure ML environment out of that conda environment using

    myenv = Environment.from_existing_conda_environment(name="myenv",conda_environment_name="mycondaenv")

    From conda or pip files

    You can create environments from conda specification or pip requirements files using

    myenv = Environment.from_conda_specification(name="myenv", file_path="path-to-conda-specification-file")
    myenv = Environment.from_pip_requirements(name="myenv", file_path="path-to-pip-requirements-file")

    Estimators and environments

    Estimators are backed by environments that define the base images, Python packages and other settings for the training environment.

    For example, to see the environment behind PyTorch Estimator, you can create a dummy instance of the Estimator, and look at the run_config.environment property.

    In [ ]:
    from azureml.train.dnn import PyTorch
    pt = PyTorch(source_directory=".", compute_target="local")

    Using environments for inferencing

    You can re-use the training environment when you deploy your model as a web service, by specifying inferencing stack version, and adding then environment to InferenceConfig.

    from azureml.core.model import InferenceConfig
    myenv.inferencing_stack_version = "latest"
    inference_config = InferenceConfig(entry_script="score.py", environment=myenv)

    When compute_target = 'local', then you will be using your local computer as target compute. There is no data stored in the cloud besides what you explicitly log. Can you please add more details about the issue.


    Tuesday, May 12, 2020 6:40 AM
  • Ram,

    The issue I am facing is that fresh local environment gets created instead of using the existing local environment even though it is using local compute target. Experiment is using my local compute, but I am unable to make it use the already existing local environment.

    With approach-1 (Create your own environment), run.get_environment(option returned 

    azureml_da3e97fcb51801118b8e80207f3e01ad (newly created local environment) whereas I don't want new local environment to be created on the fly, rather I want my existing local environment to be used.

    With approach-2 (List and get existing environments),

    Environment.get(workspace=ws,name="myenv",version="1") returns error that "myenv" doesn't exist (which is true as this does not exist in the AML workspace)

    With approach-3 (Other ways to create environment), new local environment gets created.

    Appreciate you looking into this.



    Wednesday, May 13, 2020 6:03 PM
  • Hello Nasir,

    the code above instructs to use the environment you specified in the run config. To run the script in your current environment please set user managed to true

    here are some links with docs and samples

    env.python.user_managed_dependencies = True


    Thursday, May 14, 2020 4:05 PM
  • Hello Vitaliy,

    I went through the github sample on using-environments. The closest I got (code sample below) ends with access denied error.

    Let me restate my scenario; on my laptop I have anaconda python environment setup. I created the environment locally on my laptop and installed azureml python sdk. When I create an experiment and execute that from my local (on laptop) it runs successfully for both azureml compute instance as well as local compute target (meaning my laptop as compute target). But whenever I run the experiment with compute target as local, it does not utilize already available python environment on my laptop (that already has azureml sdk and other dependencies), it rather creates a new environment on my local laptop and completes successfully.

    So the question is that, when we define compute target as local, is there no option to utilize the already available local python environment (on my laptop).

    The below code results in the access denied error (control log is copied at the end)

    from azureml.core.runconfig import RunConfiguration
    run_config_user_managed = RunConfiguration()
    run_config_user_managed.environment.python.user_managed_dependencies = True
    run_config_user_managed.environment.python.interpreter_path = 'C:\\ProgramData\\Anaconda3\\envs\\myenv'

    from azureml.core import ScriptRunConfig, RunConfiguration, Experiment
    experiment = Experiment(ws, "MyExperiment")
    config = ScriptRunConfig(source_directory='.', script='train.py', run_config=run_config_user_managed)
    run = experiment.submit(config)

    Streaming log file azureml-logs/60_control_log.txt
    Running: ['cmd.exe', '/c', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\azureml_runs\\MyExperiment_1589563494_cb36c5e1\\azureml-environment-setup/conda_env_checker.bat']
    Starting the daemon thread to refresh tokens in background for process with pid = 7216

    Control Log
    Running: ['C:\\ProgramData\\Anaconda3\\envs\\myenv', 'azureml-setup/run_script.py', 'C:\\ProgramData\\Anaconda3\\envs\\myenv', 'azureml-setup/context_manager_injector.py', '-i', 'ProjectPythonPath:context_managers.ProjectPythonPath', '-i', 'RunHistory:context_managers.RunHistory', '-i', 'TrackUserError:context_managers.TrackUserError', 'train.py']
    Logging experiment running status in history service.
    [WinError 5] Access is denied
    Logging error in history service: Exception while controlling run:
     [WinError 5] Access is denied
     Details can be found in azureml-logs/60_control_log.txt log file.

    Uploading control log...



    Friday, May 15, 2020 5:48 PM
  • Hi,

    We have forwarded this feedback to our product team will update you.


    Tuesday, May 19, 2020 3:28 AM
  • Thanks Ram.


    Tuesday, May 19, 2020 6:09 AM
  • okay, looks like you've passed the environments part:) This seems like a system issue. Can you try to run it from the elevated shell?

    Tuesday, May 19, 2020 11:53 AM
  • I did run with elevated shell and the results were same.


    Tuesday, May 19, 2020 11:58 AM