locked
ADF v2 and Batch Service in parallel, howto? Need guidance RRS feed

  • Question

  • Hi. I would like to test loading a large data set into a CRM system using ADF v2. If everything goes well in this test, I will use the orchestration capabilities of ADF v2 coupled with execution of Batch Service to build a complex migration pipeline.

    TL;DR; How do I load and split data from a table to batch service with custom activity when I want lots of batch nodes?

    The source of the data would be something like an Azure SQL table. To generalize the target, when targeting a non-supported (or not fully supported) LOB system I understand that this requires custom code, pointing to Batch Service and a custom activity. Ideally, I would have multiple batch nodes in a pool working on the data loading in parallel. This means each node in the pool will work on parts of the data set, not the whole set. 

    I am finding it hard to figure out how to slice/split and process data from the documentation of ADF v2. If I were directly interacting with the Batch Service, I would create a job and tasks in that job.

    Am I misunderstanding something here? Can someone give me a hint on what I might be looking for?

    Tuesday, February 20, 2018 9:50 AM

All replies