locked
How to resume the failed activity without triggering the whole pipeline again in ADF V2 RRS feed

  • Question

  • In ADF V1 if any of the activities failed in a pipeline then by just re-running the failed input/output dataset we could have resumed the failed activities without re-running the whole pipeline again.

    In case of ADF V2, whenever you trigger the pipeline it always starts from the very first activity. Now suppose there is a scenario where there are 10 activities in the pipeline. Activity 1-9 takes takes for examples 10 hrs to complete, now the pipeline failed at the 10th activity, what I want to do is, after rectifying the error I want to execute only 10th activity without re-running the other 9 activities as it will take another 10 hrs plus data would be doubled. Is there any way to this achieve this in ADF V2?

    Wednesday, November 21, 2018 9:00 AM

All replies

  • Hi Deepanjan,

    I'm afraid this hasn't been supported in ADF, you could submit your feedback here

    Thanks.

    Wednesday, November 21, 2018 2:00 PM
  • Hi Deepanjan,  although there is no out of box support from ADF  for this, i think its a classic case of designing workflows with fallback Support. I have had recently designed couple of workflows where activities are being chained in ADFv2 some being in form of nested pipelines  and i usually maintain a  activity history table which contains all information around previous 2 days  activity failures and design most activities in a rerunnable fashion so that if the data transformation /loading  has already happened then the flag for that activity is set to a binary 1/0 and the activity is either completely skipped  or rerun with no change .  I would suggest to think in those lines, instead of piggybacking on tooling support as that will provide higher grain of restartability controls.

    Viswa

    Sunday, November 25, 2018 8:11 PM