none
Delete Activity RRS feed

  • Question

  • Hi

    I am trying to use a Delete Activity to delete all the files that were just Copied to a master file in another folder. I want to be able to say "delete all files that were just Copied"; I have tried Last Modified but it doesn't seem to do the trick. Any suggestions? I can't use time and say delete all files that were in the folder 1 sec ago because some new files might have been added to the folder.


    Thanks!
    Mike


    Mike Kiser

    Tuesday, November 26, 2019 4:45 PM

Answers

  • You can try using the TriggerTime as your starting point and say delete all files lower than trigger time, so any files that come after Pipeline starts wont be affected.

    https://azure.microsoft.com/en-us/blog/clean-up-files-by-built-in-delete-activity-in-azure-data-factory/


    Asad Khan

    • Marked as answer by EMKISER Wednesday, November 27, 2019 3:47 AM
    Tuesday, November 26, 2019 5:59 PM
  • Can you tell me what the sink type is?  I may come up with a few other ideas.

    If it is blob storage, you could create a blob event trigger on the sink, which gets the name/path in the sink, and deletes it in the source.

    If you use Data Flow, there is an option which lets you delete the file after copying.

    • Marked as answer by EMKISER Friday, November 29, 2019 10:57 PM
    Tuesday, November 26, 2019 11:37 PM
    Moderator
  • Hey Martin,

    Thanks for your reply; its a Copy Activity from Blob to a different Blob Folder. When I create a merged Blob file in the other folder, I want to delete all of the files that were used in the Copy of the individual files to the Merged file so that next time when the Tumbling Window comes around there will not be any duplicates when it fires off the Pipeline which creates the Merged Blob File. 

    I really appreciate your help with this.....

    Thanks!
    Mike Kiser


    Mike Kiser

    • Marked as answer by EMKISER Friday, November 29, 2019 10:57 PM
    Wednesday, November 27, 2019 3:50 AM
  • Since you are using a Tumbling Window trigger, Asad Khan's idea sounds better than anything else I have come up with.  If I understand correctly, the files are being copied based upon a window of modified time, so using the same window for deleting makes sense.  Since you marked his as answer, can I assume it solved your problem?
    • Marked as answer by EMKISER Friday, November 29, 2019 10:57 PM
    Wednesday, November 27, 2019 8:01 PM
    Moderator

All replies

  • You can try using the TriggerTime as your starting point and say delete all files lower than trigger time, so any files that come after Pipeline starts wont be affected.

    https://azure.microsoft.com/en-us/blog/clean-up-files-by-built-in-delete-activity-in-azure-data-factory/


    Asad Khan

    • Marked as answer by EMKISER Wednesday, November 27, 2019 3:47 AM
    Tuesday, November 26, 2019 5:59 PM
  • Can you tell me what the sink type is?  I may come up with a few other ideas.

    If it is blob storage, you could create a blob event trigger on the sink, which gets the name/path in the sink, and deletes it in the source.

    If you use Data Flow, there is an option which lets you delete the file after copying.

    • Marked as answer by EMKISER Friday, November 29, 2019 10:57 PM
    Tuesday, November 26, 2019 11:37 PM
    Moderator
  • Hey Martin,

    Thanks for your reply; its a Copy Activity from Blob to a different Blob Folder. When I create a merged Blob file in the other folder, I want to delete all of the files that were used in the Copy of the individual files to the Merged file so that next time when the Tumbling Window comes around there will not be any duplicates when it fires off the Pipeline which creates the Merged Blob File. 

    I really appreciate your help with this.....

    Thanks!
    Mike Kiser


    Mike Kiser

    • Marked as answer by EMKISER Friday, November 29, 2019 10:57 PM
    Wednesday, November 27, 2019 3:50 AM
  • Since you are using a Tumbling Window trigger, Asad Khan's idea sounds better than anything else I have come up with.  If I understand correctly, the files are being copied based upon a window of modified time, so using the same window for deleting makes sense.  Since you marked his as answer, can I assume it solved your problem?
    • Marked as answer by EMKISER Friday, November 29, 2019 10:57 PM
    Wednesday, November 27, 2019 8:01 PM
    Moderator
  • Yes, this ADF forum is awesome!! thanks for your reply Martin!!! I have it working and it is very cool....

    Mike


    Mike Kiser

    Friday, November 29, 2019 10:58 PM