locked
Handle 404 Bad Request for Azure Data factory pipeline not to fail during iteration RRS feed

  • Question

  • Hi All,

    I have a requirement to pull data from REST API source, while iterating over the API calls the pipeline fails when it encounters a 404 bad request. I have set the fault tolerance to SKIP INCOMPATIBLE ROWS in Copy Activity inside for Each Loop. I would like to continue the iteration and ignore these bad requests if it happens during the execution. Is there a way to achieve this?

    Any suggestion is highly appreciated.

    Tuesday, March 24, 2020 6:30 AM

Answers

  • Hello CoolDbGuy and thenk you for your question.  Could you please tell me if I understand your situation correctly?

    You have a ForEach activity iterating over a list of endpoints.  The inner activities consist of a copy activity using REST dataset.  There are no issues with pagination in the copy activity.  The Copy activity either succeeds completely, or fails immediately with 404.  There are no partial copies.  The failures due to 404 cause the pipeline to return failed status.  You want it to return success status.

    To cause the ForEach to ignore the inner activity failures, please add a second activity connected to the inner copy activity by a red 'on failure' dependency.  Do not add an activity following the inner copy activity connected by a green 'on success' dependency.  The second activity can be a wait activity or anything else.

    If you already have activities following the inner copy activity, let me know and I can come up with some logic to help.

    Tuesday, March 24, 2020 6:10 PM

All replies

  • Hello CoolDbGuy and thenk you for your question.  Could you please tell me if I understand your situation correctly?

    You have a ForEach activity iterating over a list of endpoints.  The inner activities consist of a copy activity using REST dataset.  There are no issues with pagination in the copy activity.  The Copy activity either succeeds completely, or fails immediately with 404.  There are no partial copies.  The failures due to 404 cause the pipeline to return failed status.  You want it to return success status.

    To cause the ForEach to ignore the inner activity failures, please add a second activity connected to the inner copy activity by a red 'on failure' dependency.  Do not add an activity following the inner copy activity connected by a green 'on success' dependency.  The second activity can be a wait activity or anything else.

    If you already have activities following the inner copy activity, let me know and I can come up with some logic to help.

    Tuesday, March 24, 2020 6:10 PM
  • We have not received a response from you.  Are you still facing the issue?  If you found a solution, would you please share it here with the community?  Otherwise, let us know and we will continue to engage with you on the issue.
    Thursday, March 26, 2020 7:59 PM