locked
Record by record vs Batch RRS feed

  • Question

  • Hi,

    Would like to clarify that if we use Logic App to integrate different systems, does it process record by record or we can make it process in a batch (say 1000 records at a time in parallel)?

    Thanks,


    Yuji

    Friday, May 12, 2017 3:48 AM

Answers

  • We can make the execution in parallel. If we are using foreach loop In your logic app.by default foreach does parallel execution. ForEach loop has 20 degrees of parallelism, so will run all messages in parallel.

    Sujith

    Friday, May 12, 2017 4:08 AM
  • Hi,

    I have seen example where we can change execution to Sequntial in ForEach Loop.But i havent seem any where we can change Parallelism.

    As far as i know, i dont think we can do that.

    Other can correct me if i am wrong.

    Thanks,

    Sujith.


    Sujith

    Friday, May 12, 2017 6:55 PM
  • Hi guys,

    Yep, Sujith is correct in this one. There have been lots of talks about allowing us to have more control on how to batch, but there is not much option right now.

    One thing you can do today, by introducing a Service Bus as a temporary storage is this:

    1) Publish all individual messages in Service Bus (you can do this from your source system or use Logic Apps for that)

    2) Use the when one or more messages arrive in a <queue/topic subscription>. Here you can batch how many messages you want to process at a time.

    3) If you use SplitOn with this (I can't remember from the top of my head if the trigger uses splitOn by default, maybe yes) you will be able to process all of them in parallel.

    3) From here is just a matter of adjusting the ideal size of the batch, the polling frequency, and you can have more control on how many items you want to execute at a time.

    One thing to be aware is that most of the Logic Apps Connectors have a maximum number of concurrent executions (there is a comprehensive list here) so you might want to make sure you are under those limits, otherwise you will see issues like messages rejected because of execution limits.

    I hope this helps, Wagner.

    Friday, May 12, 2017 9:39 PM

All replies

  • We can make the execution in parallel. If we are using foreach loop In your logic app.by default foreach does parallel execution. ForEach loop has 20 degrees of parallelism, so will run all messages in parallel.

    Sujith

    Friday, May 12, 2017 4:08 AM
  • Thanks Sujith,

    Is there any way to change the degrees of parallelism, for example, change to 40?


    Yuji

    Friday, May 12, 2017 11:43 AM
  • Hi,

    I have seen example where we can change execution to Sequntial in ForEach Loop.But i havent seem any where we can change Parallelism.

    As far as i know, i dont think we can do that.

    Other can correct me if i am wrong.

    Thanks,

    Sujith.


    Sujith

    Friday, May 12, 2017 6:55 PM
  • Hi guys,

    Yep, Sujith is correct in this one. There have been lots of talks about allowing us to have more control on how to batch, but there is not much option right now.

    One thing you can do today, by introducing a Service Bus as a temporary storage is this:

    1) Publish all individual messages in Service Bus (you can do this from your source system or use Logic Apps for that)

    2) Use the when one or more messages arrive in a <queue/topic subscription>. Here you can batch how many messages you want to process at a time.

    3) If you use SplitOn with this (I can't remember from the top of my head if the trigger uses splitOn by default, maybe yes) you will be able to process all of them in parallel.

    3) From here is just a matter of adjusting the ideal size of the batch, the polling frequency, and you can have more control on how many items you want to execute at a time.

    One thing to be aware is that most of the Logic Apps Connectors have a maximum number of concurrent executions (there is a comprehensive list here) so you might want to make sure you are under those limits, otherwise you will see issues like messages rejected because of execution limits.

    I hope this helps, Wagner.

    Friday, May 12, 2017 9:39 PM