File Limit on uploading files to batch nodes

Question

Hi,

I understand that there is a limit of 250 on the number of files that can be uploaded to a batch node when adding a task. Is it possible to increase this limit as I will sometimes need more than this.

Thanks

---

pdcrc

Answer 1

Please refer Batch service quotas and limits : https://docs.microsoft.com/en-us/azure/batch/batch-quota-limit

Open Support case if you would like to increase the limits : https://docs.microsoft.com/en-us/azure/batch/batch-quota-limit#increase-a-quota

---

Regards,
Manjunath Sullad

Answer 2

Thanks. I've tried using the "increase-a -quota" process but the only quota that it offers for increasing is the number of Low Priority vCPUs and this is not what I want. How do I go about requesting an increase in the number of resource files that I am allowed to upload to a task?

---

pdcrc

Answer 3

This is not a limit on the number of files, but rather a size of the request being sent to Batch. It cannot be increased. If you are approaching the limit you should look into using the storageContainerUrl option instead of individually listing files: https://docs.microsoft.com/en-us/rest/api/batchservice/task/add#resourcefile

Answer 4

I've been struggling with this and under https://docs.microsoft.com/en-us/azure/batch/resource-files I have found the line:

ResourceFile inputFile = ResourceFile.FromAutoStorageContainer(containerName);

This doesn't work. All I get is : 'ResourceFile' does not contain a definition for 'FromAutoStorageContainer'.

The only things allowed after 'ResourceFile' are 'Equals' and 'ReferenceEquals'.

What am I missing?

---

pdcrc

Answer 5

According to the release notes for the Azure Batch SDK, the method FromAutoStorageContainer was introduced in version 10.0 of the SDK. The method is still there in the current source code on GitHub.

Suggest that you double check that you're using the latest version of the Microsoft.Azure.Batch NuGet package (currently v12) and not another, older package.

Also, check that the ResourceFile class you're using is the one from the Batch SDK - it's a common enough concept that you may be using a different type without intending to. Try fully qualifying the class as Microsoft.Azure.Batch.ResourceFile.

Answer 6

Thanks for this. I have now got it working. I installed V9 instead of V12 when transferring the program to my current machine as it used a constructor for Resource Files that is not in V12. Using the FromAutoStorageContainer method means I no longer need that and can also simplify things by getting rid of all the code that used the single file approach. So once again, well done.

---

pdcrc