locked
Sharing files between all tasks in a single job RRS feed

  • Question

  • Each job I submit to my pool may execute using some files that are common to all tasks in the job.

    At the moment I use this code to share the files between tasks, using a JobPreparationTask:

    CommandLine = "cmd /c (robocopy %AZ_BATCH_TASK_WORKING_DIR% %AZ_BATCH_NODE_SHARED_DIR% /S) ^& IF %ERRORLEVEL% LEQ 1 exit 0"

    However this copies the files to a directory shared between all jobs.  I notice in the docs (https://azure.microsoft.com/en-gb/documentation/articles/batch-api-basics/) it implies there is a folder "%AZ_BATCH_NODE_ROOT_DIR%\JOB_ID", which looks like a folder common to all tasks in a job that might be automatically cleaned up for me when a job is complete.  However this folder isn't mentioned directly, and there isn't (I think) an environment variable for it, although there is one for AZ_BATCH_JOB_ID.

    Is it therefore acceptable for me to do the following?

    CommandLine = "cmd /c (robocopy %AZ_BATCH_TASK_WORKING_DIR% \"%AZ_BATCH_NODE_ROOT_DIR%\%AZ_BATCH_JOB_ID%\" /S) ^& IF %ERRORLEVEL% LEQ 1 exit 0"

    Will that behave as I want? i.e. copying the files to a location shared between the tasks in a job, and cleaning them up when the job is complete?  Or should I leave them where they are and perhaps use variables like AZ_BATCH_JOB_PREP_WORKING_DIR to find them?




    • Edited by JamesThurley Thursday, May 26, 2016 8:36 AM Typo
    Thursday, May 26, 2016 8:35 AM

Answers

  • You should be able to use AZ_BATCH_JOB_PREP_WORKING_DIR to find them.  That directory should live for the duration of your job and be cleaned up afterwards. It also saves you having to do any file copies which means if you have any significant amount of data your JobPrepTask will run faster.

    So in short: Leave them where they are and use AZ_BATCH_JOB_PREP_WORKING_DIR.

    • Marked as answer by JamesThurley Friday, May 27, 2016 8:10 AM
    Thursday, May 26, 2016 10:26 PM