locked
Batch Apps : How to specify job local files to upload and files present in a blob container RRS feed

  • Question

  • Hi,

    I'm using the Batch Apps and I wonder how can I submit my job with some files to upload using JobInputs as well Uri or path of files stored in a blob container.

    My use case is :

    When job is submitted, some files specific to this job are uploaded to the TVM from my local machine.

    I have market data files that are stored in a blob container and downloaded on a TVM when a job is executed.

    An individual task will need all the market data files and the task files.

    The JobSubmission class expected  RequiredFiles that is a list of JobInputs.

    Thank you for your help

    Tuesday, June 16, 2015 2:16 AM

All replies

  • Hello Joug,

    The Batch Apps infrastructure will help you only with files being transferred via the JobInputs API.

    For the market data files, you have two options:

    Option 1. Upload a copy of the market data files to Batch Apps storage using the files API.  You can then reference those files as job inputs using the FileReference type (a FileReference is a reference to a file already in your account).  That is, you will not need to re-upload them with every job.  The advantage of this is that these files are associated with the job and Batch Apps will manage downloading them to the TVM.  The disadvantage is that you have to take a copy of them into your Batch Apps account, which could be annoying if the files changed frequently.

    Option 2. Write code in your task processor to download the market data files from their blob storage location, using HTTP GET or the Storage Client Library.  The disadvantage of this is that you have to do it yourself, you take additional dependencies in your task processor and you have no visibility of the association between the job and the market data files.

    My recommendation would be to use Option 1 if the market data files are reasonably static.  (Even if the files change regularly then you could run an automated job to copy them into your Batch account.)  Let me know if you need any more info about how to refer to existing files in a job submission.

    Tuesday, June 16, 2015 6:25 PM
  • Hi Ivan,

    Thank you for your clear explanation.

    I agree with you. Option one is the most appropriate.

    I created an Azure File Share and programmatically uploaded my files in a directory. My problem is that I don't know how to link the job file inputs and the Cloud File even with the FileReference.

    I noticed FileReference inherates from JobInput but I don't know how to instantiate it from my CouldFile. The constructor of the FileReference expects a IUserFiles and a FileIdentifier ).

    Do you have any idea how to do that?

    Thank you

    Wednesday, June 17, 2015 3:26 AM
  • A FileReference does not refer to an arbitrary blob or Azure file share -- it refers to a file in your Batch account.  Therefore, you must use the Batch Apps file management APIs to upload the files to your Batch account (as a one-off operation) and then get FileReferences to these copies of the files.

    To do the upload:

    await client.UserFiles.UploadAsync(new FileInfo(filePath));

    To use previously uploaded files in a job:

    foreach (var fileRef in await client.UserFiles.QueryByNameAsync(marketDataFileNameList)) {

      jobSubmission.RequiredFiles.Add(fileRef);

    }

    Hope that clarifies the approach!

    Wednesday, June 17, 2015 3:35 PM