locked
Error handling and Logging RRS feed

  • Question

  • Hello all,

    Whats the best way for error handling and logging in adf. I have many activities in pipeline and want to capture/notify whenever any one of the activity gets failed. The only solution i see for now is deriving failures from all components and logging them to a SP component. Please suggest if you implement anything different.

    Thanks in advance.


    Porus

    Wednesday, February 19, 2020 2:27 PM

All replies

  • Hi Aneeshporus,

    Thanks for your query. 

    1.You could use Azure Monitor in ADF and enable diagnostic log and store the logs into Azure Blob Storage.

    2.Another choice,you could use ADF monitor SDK or REST API to retrieve the activities run details by Pipeline Run Id:

    List<ActivityRun> activityRuns = client.ActivityRuns.ListByPipelineRun(
    resourceGroup, dataFactoryName, runResponse.RunId, DateTime.UtcNow.AddMinutes(-10), DateTime.UtcNow.AddMinutes(10)).ToList(); 
    if (pipelineRun.Status == "Succeeded")
        Console.WriteLine(activityRuns.First().Output);
    else
        Console.WriteLine(activityRuns.First().Error);

    For more info please refer to this doc: Alert and monitor data factories by using Azure Monitor
    Video: Monitor Data factory pipelines using Azure Monitor and Log Analytics

    Hope this helps. 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, February 19, 2020 9:10 PM
  • Hi Aneeshporus,

    Just checking in to see if the above suggestion was helpful. If this answers your query, please do click “Mark as Answer” and Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, February 20, 2020 9:28 PM
  • Hi Aneeshporus,

    Just checking to see if the above information was helpful. If you still need assistance do let us know.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, February 25, 2020 3:48 AM