locked
Adding multiple tasks to batchclient - Max request length exceeded error RRS feed

  • Question

  • Hi,

    I am trying to add quite a lot of tasks to BatchClient and each task has their own set of resources with their shared access signatures. When i tried calling AddTask method by passing in all the tasks it get

    Microsoft.Azure.Batch.Common.BatchException: RequestBodyTooLarge: The request body is too large and exceeds the maximum permissible limit.

    Are there options in the API to tweak this?


    Please mark the response as answers if it solves your question or vote as helpful if you find it helpful. http://thoughtorientedarchitecture.blogspot.com/

    Wednesday, October 21, 2015 12:55 PM

Answers

  • Hi Haripraghash, sorry for the slow reply!

    In general for handling too many resource files, youll need to generate a zip containing the resource files and unzip it on the VM.  There are a few ways to to this:

    1) You can use a public tool such as 7Zip on the command line to unzip your zip archive, and include 7Zip as a resource file to your tasks (or install it via a StartTask onto your Pool VMs).  So your command line would look something like this: "cmd /c 7zip.exe e TaskResourceFiles.zip && <normal task command line here>"

    2) You can write some custom code to perform the unzipping and compile it to an executable, and use that instead of a public executable.  So your command line would look something like: "cmd /c MyUnzipper.exe TaskResourceFiles.zip && <normal task command line here>"  In C#, the code to perform this unzipping is available in the System.IO.Compression and System.IO.Compression.FileSystem DLLs.  Something like the following is all that you need:
    ZipFile.ExtractToDirectory(zipFilePath, destinationDirectoryPath);

    Hopefully this helps some.
    -Matt
    Friday, October 30, 2015 10:00 PM

All replies

  • Hi Haripraghash,

    If you add a lot of tasks, which have a large payload of resource files or environment variables it is possible to exceed this limit.  We generally recommend for environment variables to use JobCommonEnviornmentVariables if your variables are the same for all tasks, and for resource files if you exceed the limit the best solution is usually to zip the resource files up into a single ZIP and specify your tasks command line to first unzip the zip archive and then run the task executable. 

    If you like I can share a code snippet of the resource file zip based solution to give you a better idea.

    Thanks,

    -Matt
    Friday, October 23, 2015 9:07 PM
  • Hi Mathew,

    A code snippet would be really helpful.


    Please mark the response as answers if it solves your question or vote as helpful if you find it helpful. http://thoughtorientedarchitecture.blogspot.com/

    Saturday, October 24, 2015 1:52 PM
  • Hi Haripraghash, sorry for the slow reply!

    In general for handling too many resource files, youll need to generate a zip containing the resource files and unzip it on the VM.  There are a few ways to to this:

    1) You can use a public tool such as 7Zip on the command line to unzip your zip archive, and include 7Zip as a resource file to your tasks (or install it via a StartTask onto your Pool VMs).  So your command line would look something like this: "cmd /c 7zip.exe e TaskResourceFiles.zip && <normal task command line here>"

    2) You can write some custom code to perform the unzipping and compile it to an executable, and use that instead of a public executable.  So your command line would look something like: "cmd /c MyUnzipper.exe TaskResourceFiles.zip && <normal task command line here>"  In C#, the code to perform this unzipping is available in the System.IO.Compression and System.IO.Compression.FileSystem DLLs.  Something like the following is all that you need:
    ZipFile.ExtractToDirectory(zipFilePath, destinationDirectoryPath);

    Hopefully this helps some.
    -Matt
    Friday, October 30, 2015 10:00 PM