local compute target always results in creating new local environment

Question

Hello,

I am running the code example (pasted at bottom), it does use the local compute and completes successfully but instead of using existing local environment named "myenv" it creates a new environment, downloads and installs packages in it first.

Is it not supposed to utilize the existing environment locally instead of preparing the new one? Am I missing anything here?

Log File 60.control_log.txt show that conda_env_checker.bat does not find the environment (although different name) and then conda_env_builder.bat script kicks in to build new environment. Here are exact log detail:

Streaming log file azureml-logs/60_control_log.txt
Running: ['cmd.exe', '/c', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\azureml_runs\\train-on-local_1588961369_5651b1d3\\azureml-environment-setup/conda_env_checker.bat']
Starting the daemon thread to refresh tokens in background for process with pid = 275792
Materialized conda environment not found on target: C:\Users\Administrator/.azureml/envs/azureml_da3e97fcb51801118b8e80207f3e01ad

Logging experiment preparation status in history service.
Running: ['cmd.exe', '/c', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\azureml_runs\\train-on-local_1588961369_5651b1d3\\azureml-environment-setup/conda_env_builder.bat']
Running: ['conda', '--version']
conda 4.8.3

from azureml.core import ScriptRunConfig, Experiment
from azureml.core.environment import Environment

exp = Experiment(name="myexp", workspace = ws)
# Instantiate environment
myenv = Environment(name="myenv")

# Add training script to run config
runconfig = ScriptRunConfig(source_directory=".", script="train.py")

# Attach compute target to run config
runconfig.run_config.target = "local"

# Attach environment to run config
runconfig.run_config.environment = myenv

# Submit run 
run = exp.submit(runconfig)

---

Nasir

Answer 1

Hi,

Please follow the below to configure a development environment for local computer.

Create your own environment

You can create an environment by instantiating Environment object and then setting its attributes: set of Python packages, environment variables and others.

Add Python packages

The recommended way is to specify Conda packages, as they typically come with complete set of pre-built binaries.

In [ ]:

from azureml.core.environment import CondaDependencies

myenv = Environment(name="myenv")
conda_dep = CondaDependencies()
conda_dep.add_conda_package("scikit-learn")

You can also add pip packages, and specify the version of package

In [ ]:

conda_dep.add_pip_package("pillow==5.4.1")
myenv.python.conda_dependencies=conda_dep

Specify environment variables

You can add environment variables to your environment. These then become available using os.environ.get in your training script.

In [ ]:

myenv.environment_variables = {"MESSAGE":"Hello from Azure Machine Learning"}

Submit run using environment

When you submit a run, you can specify which environment to use.

On the first run in given environment, Azure ML spends some time building the environment. On the subsequent runs, Azure ML keeps track of changes and uses the existing environment, resulting in faster run completion.

In [ ]:

from azureml.core import ScriptRunConfig, Experiment

myexp = Experiment(workspace=ws, name = "environment-example")

To submit a run, create a run configuration that combines the script file and environment, and pass it to Experiment.submit. In this example, the script is submitted to local computer, but you can specify other compute targets such as remote clusters as well.

In [ ]:

runconfig = ScriptRunConfig(source_directory=".", script="example.py")
runconfig.run_config.target = "local"
runconfig.run_config.environment = myenv
run = myexp.submit(config=runconfig)

run.wait_for_completion(show_output=True)

To audit the environment used by for a run, you can use get_environement.

In [ ]:

run.get_environment()

List and get existing environments

Your workspace contains a dictionary of registered environments. You can then use Environment.get to retrieve a specific environment with specific version.

In [ ]:

for name,env in ws.environments.items():
    print("Name {} \t version {}".format(name,env.version))

restored_environment = Environment.get(workspace=ws,name="myenv",version="1")

print("Attributes of restored environment")
restored_environment

Other ways to create environments

From existing Conda environment

You can create an environment from existing conda environment. This make it easy to reuse your local interactive environment in Azure ML remote runs. For example, if you've created conda environment using

conda create -n mycondaenv

you can create Azure ML environment out of that conda environment using

myenv = Environment.from_existing_conda_environment(name="myenv",conda_environment_name="mycondaenv")

From conda or pip files

You can create environments from conda specification or pip requirements files using

myenv = Environment.from_conda_specification(name="myenv", file_path="path-to-conda-specification-file")

myenv = Environment.from_pip_requirements(name="myenv", file_path="path-to-pip-requirements-file")

Estimators and environments

Estimators are backed by environments that define the base images, Python packages and other settings for the training environment.

For example, to see the environment behind PyTorch Estimator, you can create a dummy instance of the Estimator, and look at the run_config.environment property.

In [ ]:

from azureml.train.dnn import PyTorch

pt = PyTorch(source_directory=".", compute_target="local")
pt.run_config.environment

Using environments for inferencing

You can re-use the training environment when you deploy your model as a web service, by specifying inferencing stack version, and adding then environment to InferenceConfig.

from azureml.core.model import InferenceConfig

myenv.inferencing_stack_version = "latest"

inference_config = InferenceConfig(entry_script="score.py", environment=myenv)

When compute_target = 'local', then you will be using your local computer as target compute. There is no data stored in the cloud besides what you explicitly log. Can you please add more details about the issue.

• Training the models locally using local data - https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-environment#local

Thanks

Answer 2

Ram,

The issue I am facing is that fresh local environment gets created instead of using the existing local environment even though it is using local compute target. Experiment is using my local compute, but I am unable to make it use the already existing local environment.

With approach-1 (Create your own environment), run.get_environment() option returned 

azureml_da3e97fcb51801118b8e80207f3e01ad (newly created local environment) whereas I don't want new local environment to be created on the fly, rather I want my existing local environment to be used.
With approach-2 (List and get existing environments), 
Environment.get(workspace=ws,name="myenv",version="1") returns error that "myenv" doesn't exist (which is true as this does not exist in the AML workspace)
With approach-3 (Other ways to create environment), new local environment gets created.
Appreciate you looking into this.
Thanks,
 

---

Nasir

Answer 3

Hello Nasir,

the code above instructs to use the environment you specified in the run config. To run the script in your current environment please set user managed to true

here are some links with docs and samples

env.python.user_managed_dependencies = True

https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.environment.pythonsection?view=azure-ml-py
https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training/using-environments/using-environments.ipynb
-Vitaliy

Answer 4

Hello Vitaliy,

I went through the github sample on using-environments. The closest I got (code sample below) ends with access denied error.

Let me restate my scenario; on my laptop I have anaconda python environment setup. I created the environment locally on my laptop and installed azureml python sdk. When I create an experiment and execute that from my local (on laptop) it runs successfully for both azureml compute instance as well as local compute target (meaning my laptop as compute target). But whenever I run the experiment with compute target as local, it does not utilize already available python environment on my laptop (that already has azureml sdk and other dependencies), it rather creates a new environment on my local laptop and completes successfully.

So the question is that, when we define compute target as local, is there no option to utilize the already available local python environment (on my laptop).

The below code results in the access denied error (control log is copied at the end)

from azureml.core.runconfig import RunConfiguration
run_config_user_managed = RunConfiguration()
run_config_user_managed.environment.python.user_managed_dependencies = True
run_config_user_managed.environment.python.interpreter_path = 'C:\\ProgramData\\Anaconda3\\envs\\myenv'

from azureml.core import ScriptRunConfig, RunConfiguration, Experiment
experiment = Experiment(ws, "MyExperiment")
config = ScriptRunConfig(source_directory='.', script='train.py', run_config=run_config_user_managed)
run = experiment.submit(config)

Streaming log file azureml-logs/60_control_log.txt
Running: ['cmd.exe', '/c', 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\azureml_runs\\MyExperiment_1589563494_cb36c5e1\\azureml-environment-setup/conda_env_checker.bat']
Starting the daemon thread to refresh tokens in background for process with pid = 7216

Control Log
Running: ['C:\\ProgramData\\Anaconda3\\envs\\myenv', 'azureml-setup/run_script.py', 'C:\\ProgramData\\Anaconda3\\envs\\myenv', 'azureml-setup/context_manager_injector.py', '-i', 'ProjectPythonPath:context_managers.ProjectPythonPath', '-i', 'RunHistory:context_managers.RunHistory', '-i', 'TrackUserError:context_managers.TrackUserError', 'train.py']
Logging experiment running status in history service.
[WinError 5] Access is denied
Logging error in history service: Exception while controlling run:
 [WinError 5] Access is denied
 Details can be found in azureml-logs/60_control_log.txt log file.

Uploading control log...

-Nasir

---

Nasir

Answer 5

Hi,

We have forwarded this feedback to our product team will update you.

Thanks

Answer 6

Thanks Ram.

---

Nasir

Answer 7

okay, looks like you've passed the environments part:) This seems like a system issue. Can you try to run it from the elevated shell?

Answer 8

I did run with elevated shell and the results were same.

---

Nasir