locked
Copy Activity - How to check data read / data written by pipeline? RRS feed

  • Question

  • Hello,

    In "Pipeline runs" there is the amount of data read / written per table. Is it possible to view this information by pipeline?

    I thought this information was in the Log Analytics ADFActivityRun / ADFPipelineRun tables, but I couldn't find any information, either by pipeline or by table.

    Best regards,
    Cristina

    Thursday, January 9, 2020 6:47 PM

Answers

  • Hi Cristina,

    I haven't heard back about any direct way to get the total data read/written by a pipeline, but one possible way you could try is to get the data read and data written information from each copy activity and the sum all together to get the total data read/written. 




    I would recommend you to please post your suggestion/idea in ADF user voice forum. All the feedback shared in this forum are monitored and reviewed by ADF engineering team and will take appropriate action.

    ADF feedback forum: https://feedback.azure.com/forums/270578-azure-data-factory

    Once feedback thread is created, please do post it here as it might help other users with similar idea to up-vote and comment on you feature request. This way, the priority of the feature request can be increased.

    Hope this info helps.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.



    Monday, January 13, 2020 5:28 PM

All replies

  • Hi Cristina,

    It's a great question. I'm not sure if this is possible at the pipeline level but I have reached out to internal team to get more info about your query.

    I will keep you posted once I have a response from the team.



    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, January 9, 2020 10:40 PM
  • Thank you!

    Best regards,
    Cristina


    Thursday, January 9, 2020 10:48 PM
  • Hi Cristina,

    I haven't heard back about any direct way to get the total data read/written by a pipeline, but one possible way you could try is to get the data read and data written information from each copy activity and the sum all together to get the total data read/written. 




    I would recommend you to please post your suggestion/idea in ADF user voice forum. All the feedback shared in this forum are monitored and reviewed by ADF engineering team and will take appropriate action.

    ADF feedback forum: https://feedback.azure.com/forums/270578-azure-data-factory

    Once feedback thread is created, please do post it here as it might help other users with similar idea to up-vote and comment on you feature request. This way, the priority of the feature request can be increased.

    Hope this info helps.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.



    Monday, January 13, 2020 5:28 PM
  • Hi KranthiPakala-MSFT,

    Thanks for the reply, I'll suggest creating this feature in the forum. 

    I would like to see the values ​​in the Log Analytics "ADFActivityRun" table, but I didn't find it. Do you know why this information does not exist in the table, but when we create an alert it appears?



    Best regards,
    Cristina
    Tuesday, January 14, 2020 12:14 PM
  • Hi KranthiPakala-MSFT,

    Please disregard my question, I was unable to view the information because of the notebook used. By exporting the fields from table "ADFActivityRun" in Log Analytics I could see. 

    Best regards,
    Cristina
    Tuesday, January 14, 2020 1:43 PM
  • Thanks for sharing your findings, Cristina!  

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, January 14, 2020 7:31 PM