none
Azure ADF Pipeline Logs RRS feed

  • Question

  • Hi,

    We created pipelines and able to see the pipeline run statistics in Azure Audit Logs.

    Is there any way to read these azure audit logs and store in our custom database? For example, we maintain a log database for each ETL/pipeline run statistics success/failure, loaded row count etc. This db is our centralized log repository.

    We would like to read azure pipeline run logs and store in custom db.

    Please clarify.

    Thanks,

    Peer Md.

    Monday, December 12, 2016 4:51 AM

All replies

  • How about creating another pipeline for that?
    Monday, December 12, 2016 6:26 PM
  • Hi Alexandre,

    Thanks for your response.

    We can create another pipeline or application for loading the logs from Azure to our custom db.

    But the question, where azure logs or pipeline logs stored? Is it a db or web service?

    How we can access this to retrieve azure logs?

    For example,

    In Pipeline activity run, we will see activity details as below

    Data Read: 2.25kb

    Written: 2.25 kb

    Data Volume:

    Rows: 1000

    Throughput:

    Duration:

    And other details. We would like to read this and store in our db.

    Please advice.

    Thanks,

    Peer Md.


    Tuesday, December 13, 2016 3:22 AM
  • You could use an Azure Batch Azure Automation (https://azure.microsoft.com/en-us/services/automation/) to run a PowerShell script that runs the commands Get-AzureRmDataFactoryRun and/or Save-AzureRmDataFactoryLog (https://docs.microsoft.com/en-us/azure/data-factory/data-factory-monitor-manage-pipelines). Or the equivalent using Azure .NET SDK in a custom ADF job.

    Tuesday, December 13, 2016 5:27 PM
  • Hi Alexandre,

    When I use Save-AzureRmDataFactoryLog, it gives me an error as below, which states no logs are available

    Save-AzureRmDataFactoryLog : HTTP Status Code: NotFound
    Error Code: RunLogsNotAvailable

    is there any other way to get the data read and data write details.

    Thanks & Regards,
    Praveen.

    Wednesday, June 13, 2018 4:04 PM