Azure ADF Pipeline Logs RRS feed

  • Question

  • Hi,

    We created pipelines and able to see the pipeline run statistics in Azure Audit Logs.

    Is there any way to read these azure audit logs and store in our custom database? For example, we maintain a log database for each ETL/pipeline run statistics success/failure, loaded row count etc. This db is our centralized log repository.

    We would like to read azure pipeline run logs and store in custom db.

    Please clarify.


    Peer Md.

    Monday, December 12, 2016 4:51 AM

All replies

  • How about creating another pipeline for that?
    Monday, December 12, 2016 6:26 PM
  • Hi Alexandre,

    Thanks for your response.

    We can create another pipeline or application for loading the logs from Azure to our custom db.

    But the question, where azure logs or pipeline logs stored? Is it a db or web service?

    How we can access this to retrieve azure logs?

    For example,

    In Pipeline activity run, we will see activity details as below

    Data Read: 2.25kb

    Written: 2.25 kb

    Data Volume:

    Rows: 1000



    And other details. We would like to read this and store in our db.

    Please advice.


    Peer Md.

    Tuesday, December 13, 2016 3:22 AM
  • You could use an Azure Batch Azure Automation (https://azure.microsoft.com/en-us/services/automation/) to run a PowerShell script that runs the commands Get-AzureRmDataFactoryRun and/or Save-AzureRmDataFactoryLog (https://docs.microsoft.com/en-us/azure/data-factory/data-factory-monitor-manage-pipelines). Or the equivalent using Azure .NET SDK in a custom ADF job.

    Tuesday, December 13, 2016 5:27 PM
  • Hi Alexandre,

    When I use Save-AzureRmDataFactoryLog, it gives me an error as below, which states no logs are available

    Save-AzureRmDataFactoryLog : HTTP Status Code: NotFound
    Error Code: RunLogsNotAvailable

    is there any other way to get the data read and data write details.

    Thanks & Regards,

    Wednesday, June 13, 2018 4:04 PM
  • Hi,
    Has anyone  found an answer to the above?
    We are getting the same info:
    Save-AzureRmDataFactoryLog : HTTP Status Code: NotFound
    Error Code: RunLogsNotAvailable
    Tuesday, February 11, 2020 12:18 PM
  • Hi Alexandra86

    Since the Original poster query is older, I would suggest you to please open a new thread, that way your issue will have more visibility and helps to find a resolution faster. 

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, February 12, 2020 12:29 AM