sticky
Installing Applications and staging data on Batch compute nodes RRS feed

  • General discussion

  • Installing Applications and staging data on Batch compute nodes

    This post describes the mechanisms which Azure Batch supports for moving data/applications onto compute nodes so that the data/application(s) are available for use by your tasks.

    Ways to get files onto Batch compute nodes

    1. Pool start task resource files: For applications or data which need to be installed on every node in the pool. Use this along with either an application package or the start task's resource file collection in order to perform an install command (i.e. run an MSI or script to set things up). For example: Every job that runs on the pool runs MyApplication.exe which must first be installed with MyApplication.msi. Note that if you use this mechanism, you need to set the start task's wait for success property to true.
    2. Application package references on the pool: For applications or data which need to be installed on every node in the pool. There is no install command associated with an application package but you can use a start task to run any install command. Use this if your application is copy-deployable (no install needed), or consists of a very large number of files. Application packages are well suited for large numbers of files because they lump a large number of file references into a small payload – including 100+ resource files in a request to add a task to the Batch service might cross internal system limitations on number of resource files on a single task. Also use application packages if you have rigorous versioning requirements where you might have many different versions of the same application and need to choose between them.
    3. Job preparation task resource files: For applications or data which must be installed for the job to run, but do not need to be installed on the entire pool. For example: if your pool is being shared between many different types of jobs, and only one job type needs MyApplication.msi run, then it makes sense to put the installation step into a job preparation task. For more information about job preparation tasks see: https://azure.microsoft.com/documentation/articles/batch-job-prep-release/
    4. Task resource files: For when application or data is only relevant to an individual task. For example: You have 5 tasks, each processes a different file and then writes their output to blob storage.  In that case, the input file should be specified on the tasks resource files collection because no two tasks ever share the same input file.

    A note on file scope
    Determining the scope at which a file is needed is critical to getting the best performance out of the Azure Batch file management solutions. Files which are scoped to the pool should use pool application packages, or a start task. Files scoped to the job should use a job preparation task. A good example of files generally scoped at the pool or job level are applications (i.e. Blender or MATLAB). Files scoped to the task should use task resource files. For example, if 1000 tasks share the same 1GB file, and you specify the file at the task level, you will download 1GB 1000 times, compare that with a job preparation task where you will download the file N times, where N is the number of nodes which the job runs on.

    Azure Storage
    Take a look at this article for more Azure Storage throughput information: https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/
    The most important thing to note for Azure Batch is the blob download scalability targets. As of June 2016 each blob can scale up to 60 MB per second. You can do some basic math to estimate your expected download wait times based on this. For example, 20 nodes, 60 MB per second, so each node gets 3 MB per second, or 200 nodes which means each node would get 300 KB per second. Also notice that the Azure Storage file share scalability targets are the same as for a single blob, so the same scalability problems apply for file shares too.

    Running installers (using a start task or job preparation task)
    Running installers can be done using a start task or a job preparation task. The command to invoke the installer in a silent non-interactive mode should be run using the command line parameter of the start task or job preparation task.  Note that some installers require you to run as an administrator -- so look for the run elevated property on the above tasks (note on Linux that if you specify the run elevated property there is no need to include sudo on the command line).  Tasks run in this fashion should be idempotent in order to avoid issues where a VM reboots or another job with the same job preparation task runs and triggers the setup task to run again. Note that for the start task, you must specify wait for success true in order to force the installation to happen before any work is scheduled to the node.

    A note on reboots (mostly for Windows)
    If after you run an install command you need to reboot the VM, then you must install with your installer via start task. If you're on Windows, you should follow this algorithm:

    1. Set wait for success to true
    2. Set run elevated to true
    3. Craft a command script which will install your application in an idempotent fashion (i.e. it needs to check if the application is already installed and exit gracefully if it is). Note that if your application doesn't support an easy way to look this up you can use a file sentinel.
    4. The command should do the following:
      If the application is installed, exit with exit code 0.
      If the application is NOT installed, install it, call "shutdown /r /t 0" to schedule a reboot, and then exit with a nonzero exit code.  This will move the VM to "StartTaskFailed" state which will forbid any scheduling on it.  But once the shutdown command has run, the VM will restart and re-run the start task, at which point the script will realize the application has been installed and the start task will complete successfully. Note: You need to be an administrator on Windows to run "shutdown /r /t 0" which is why you must have run elevated set to true.

    Note: You cannot use a job preparation task to manage an install which requires a reboot because it does not support temporarily moving to a "failed" state like the start task does.  Because of this, it cannot delay task scheduling until after the reboot.

    Docker Containers
    Another way to configure your Batch VM is to use a Docker container. The recommended way to use Docker containers is through the Batch Shipyard toolkit that can be found here: https://github.com/Azure/batch-shipyard. Code samples are also provided for those that wish to craft their own Docker deployment solution which can be found here: https://github.com/Azure/azure-batch-samples/tree/master/Python/Batch.

    Other ways to get data onto Batch compute nodes
    There are other ways to get data onto Batch compute nodes which are not officially integrated into the Batch REST API. Because you have total control over Azure Batch nodes, and can run custom executables, you can pull data from any number of custom sources as long as the Batch node has connectivity to the target and you can get the credentials to that source onto the Azure Batch node. A few common examples are:

    1. Downloading data from SQL
    2. Downloading data from other web services/custom locations
    3. Mapping a network share

    Note that for each of these data sources, the principles of data scope and managing throughput still apply.

    If you have any questions or feedback about the information presented in this post, feel free to ask it here.





    Thursday, June 16, 2016 12:24 AM
    Owner