locked
Pipeline to skip files when there is no permission while copies data from on-premise to Azure Data Lake RRS feed

  • Question

  • We have a data factory pipeline that copies from on-premise to Azure Data Lake  
    Some files/folders in on-premise have restricted access, let’s say it has rwx------  access (Only the owner who created has full access and no one else has access).
    In this case the Azure Data Factory copy activity fails and none of the other files in the folder are copied to Azure Data Lake.
    it is a huge amount of file within allot of folders (and many are nested)
     
    is there a work around/option to skip those files/folders and continue with the copy activity?
    Wednesday, March 25, 2020 1:43 PM

All replies

  • Hi SJari

    Thanks you for reaching out. I have reached out to internal team to see if there a work-around to avoid the pipeline failure. Will keep you posted as soon as I have a response from the team. 

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, March 25, 2020 11:34 PM
  • Hi SJari,

    After having conversation with internal team, for permission errors, we don't skip them by default. ADF engineering team is having a skip file feature (work item), and it is planned to be ready in Prod before end of April (Note: This is tentative ETA). Once this feature is released, you can specify if you would like to skip such errors.

     I don't see any possible work around other than having permissions. I would recommend you to please follow/subscribe to Azure Data Factory updates to know about the latest updates, roadmap, announcements related to Azure Data Factory.

    Hope this helps.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, March 27, 2020 7:17 PM
  • Hi SJari,

    Just checking to see if the above answer was helpful. If this answers your query, please do consider to click “Mark as Answer” and "Up-Vote" as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Monday, March 30, 2020 6:57 PM