none
Delete Blob Activity Needed in Azure Data Factory UI

    Question

  • The Data Factory UI in Azure Portal includes multiple activities that one can chain up when creating a Data Factory pipeline, like the Copy activity for instance. I use this activity to read a CSV text file from a blob, parse its contents, and copy them to a SQL Server database. This works like a charm. However, after the Copy activity completes its work successfully, I would like to archive this blob by moving it to another location (either to a different storage container or to another folder within the same container). There does not seem to be an activity available that can help me with this. 

    I would like to suggest a new "Delete Blob" activity that would delete a blob from a storage container. I can then achieve the goal of copying to a SQL Server database and then archive the CSV file. To do this, I would chain one Copy activity (to copy the data to SQL Server) to another Copy activity (to move the blob to the archive) and then to a Delete Blob activity (to remove the blob that has been archived). From looking around online forums, this seems to be a basic scenario for many users of Azure Data Factory. I am certainly not the only one looking for a way to delete a blob from within a pipeline.

    Current documentation on MSDN suggests creating a new Azure Batch to run a custom-build .NET Console Application that will perform the deleting of the blob. This seems like overkill. Why should I have to program an entire console application and create an Azure Batch for a task as mundane as deleting a blob? Especially, when the Azure Data Factory UI already includes a Copy activity. The Delete Blob or Move Blob activities seem like they would naturally fit alongside the Copy activity. The promise of setting up pipelines with the UI feels broken when something as simple as deleting a blob is not one of the activities included out of the box. 

    I hope the Azure team considers adding a Delete Blob or Move Blob activity to the Data Factory UI. Thanks!


    Friday, March 30, 2018 3:37 PM

All replies