none
Deleting Old Blobs in a C# Console Application

    Question

  • Hello,

    I am trying to write an application that I can run which will go into a specific folder in Blob storage and delete any files that were last modified more than 14 days ago. I found this article that helped me get started. It had some incorrect syntax which I have been able to fix, but now I am stuck.

    I have no syntax errors, but when I run my application, it sits on the 'Running...' line without ever doing anything. Does it take a really long time to create an IEnumerable list of Blobs? I think something is wrong with my configuration for that list creation. Here is my code: 

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using Microsoft.WindowsAzure.Storage;
    using Microsoft.WindowsAzure.Storage.Blob;
    
    namespace Last24Hours_Blob_Clean
    {
        class Program
        {
            static void Main(string[] args)
            {
                int DaysToKeep = 14;
                // Name of storage account that will do the processing
                const string StorageAccountName = "mlxdevopsmlpoc";
                // Secondary storage account key
                const string StorageAccountKey = "...";
                // Container name where you want to store the results of the job
                const string StorageContainerName = "hddefaulttest";
                string storageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", StorageAccountName, StorageAccountKey);
                // Retrieve storage account from connection string.
                CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
                // Create the blob client.
                CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
                // Next select the container which you are looking for your old blobs.
                CloudBlobContainer container = blobClient.GetContainerReference(StorageContainerName);
                string pathPrefix = "ASA/";
    
                Console.WriteLine("Storage Name: {0}, \ncontainer: {1}, \nkeeping last {2} days, \nstarting from today's date: {3}\n\nRunning...", StorageAccountName, StorageContainerName, DaysToKeep, DateTime.Now.ToString());
                Console.ReadLine();
    
                // Now we will set up Blob access condition option which will filter all the blobs which are not modified for X (this.DaysToKeep) amount of days
                IEnumerable<IListBlobItem> blobs = container.ListBlobs(prefix: pathPrefix);
                Console.WriteLine("blobs ready: looking in {0}" + "ASA/", container.Uri);
                foreach (IListBlobItem blob in blobs)
                {
                    Console.WriteLine("Checking {0}", blob.Uri.ToString());
                    CloudBlockBlob cloudBlob = container.GetBlockBlobReference(blob.Uri.ToString());
                    Console.WriteLine(cloudBlob.Properties);
                    cloudBlob.DeleteIfExists(DeleteSnapshotsOption.None, AccessCondition.GenerateIfNotModifiedSinceCondition(DateTime.Now.AddDays(-1 * DaysToKeep)), null, null);
                }
                Console.WriteLine("Done!");
                Console.ReadLine();
    
            }
        }
    }

    And here is the output: 

    Is this the correct way to point to the contents of the folder "hddefault/ASA/"? That is my goal. 

    Any help is greatly appreciated! Thank you,

    -Danielle


    • Edited by Danielle F Monday, June 8, 2015 11:03 PM
    Monday, June 8, 2015 11:03 PM

Answers

  • Hi Danielle,

    You could use the below snippet to list the blobs which are older than 14 days. The ListBlobs method takes prefix parameter where you could pass "ASA" as the first parameter so that all the blobs that are listed within the container hddefaulttest/ASA are listed.

     static void GetOldBlobs()
        {
            CloudStorageAccount acc = new CloudStorageAccount(new StorageCredentials("account name", "account key"), false);
            var client = acc.CreateCloudBlobClient();
            var container = client.GetContainerReference("container name");
            var blobs = container.ListBlobs("", true).OfType<CloudBlockBlob>().Where(b => (DateTime.UtcNow.AddDays(-14) > b.Properties.LastModified.Value.DateTime)).ToList();
        }

    You could refer the following link for details:
    http://stackoverflow.com/questions/23935673/azure-delete-old-blobs-based-on-last-modified

    Regards,
    Malar.

    Monday, June 15, 2015 10:14 AM

All replies

  • Hi Danielle,

    Thank you for posting in here.
    We are checking on this and will get back at earliest.

    Regards,
    Manu Rekhar

    Tuesday, June 9, 2015 5:50 PM
    Moderator
  • Hi Nanielle,

    There is a statement Console.ReadLine() there, do you see it? :)

    Best Regards,

    Zhaoxing Lu

    Thursday, June 11, 2015 2:38 PM
  • Haha oops! Sometimes it takes another pair of eyes for silly mistakes like that. I tried running that without the ReadLine and now it runs! But my follow-up question is that it doesn't actually delete any blobs. Here is the console output: 

    And here are the contents of the blob storage before AND after (no change): 

    Any idea why it's not deleting the blobs? By my calculations, 14 days ago would be 5/28/2015

    Thursday, June 11, 2015 3:16 PM
  • Hi Danielle,

    Every time you enumerated a blobItem, you were trying to create a CloudBlockBlob object by passing the whole Uri as "blobName". Therefore the service will regard the blob not exists since the combined Uri of the cloudBlob is completely wrong. Please cast the listed blob object to CloudBlockBlob directly.

     foreach (IListBlobItem blob in blobs)
                {
                    Console.WriteLine("Checking {0}", blob.Uri.ToString());
                    CloudBlockBlob cloudBlob = blob as CloudBlockBlob;
                    Console.WriteLine(cloudBlob.Properties);
                    cloudBlob.DeleteIfExists(DeleteSnapshotsOption.None, AccessCondition.GenerateIfNotModifiedSinceCondition(DateTime.Now.AddDays(-1 * DaysToKeep)), null, null);
                }

    Best Regards,

    Zhaoxing Lu


    • Edited by Zhaoxing Lu Friday, June 12, 2015 6:08 AM
    • Proposed as answer by Zhaoxing Lu Wednesday, June 17, 2015 1:56 AM
    Friday, June 12, 2015 6:05 AM
  • Ok, I see your point. So I tried that, and I get this error after it runs through ~3 blobs. 

    Exception:Thrown: "The remote server returned an error: (412) The condition specified using HTTP conditional header(s) is not met.." (System.Net.WebException)
    A System.Net.WebException was thrown: "The remote server returned an error: (412) The condition specified using HTTP conditional header(s) is not met.."
    Time: 6/12/2015 9:24:39 AM
    Thread:<No Name>[19712]

    The error occurs when it hits this line of code:

    cloudBlob.DeleteIfExists(DeleteSnapshotsOption.None, AccessCondition.GenerateIfNotModifiedSinceCondition(DateTime.Now.AddDays(-1 * DaysToKeep)), null, null);

    Does this have something to do with my AccessCondition? Does that mean that if the condition is not met, then the access permissions for blob storage go away? Or does it just mean Delete the blob if the condition is met?

    Friday, June 12, 2015 4:59 PM
  • Hi Danielle,

    You could use the below snippet to list the blobs which are older than 14 days. The ListBlobs method takes prefix parameter where you could pass "ASA" as the first parameter so that all the blobs that are listed within the container hddefaulttest/ASA are listed.

     static void GetOldBlobs()
        {
            CloudStorageAccount acc = new CloudStorageAccount(new StorageCredentials("account name", "account key"), false);
            var client = acc.CreateCloudBlobClient();
            var container = client.GetContainerReference("container name");
            var blobs = container.ListBlobs("", true).OfType<CloudBlockBlob>().Where(b => (DateTime.UtcNow.AddDays(-14) > b.Properties.LastModified.Value.DateTime)).ToList();
        }

    You could refer the following link for details:
    http://stackoverflow.com/questions/23935673/azure-delete-old-blobs-based-on-last-modified

    Regards,
    Malar.

    Monday, June 15, 2015 10:14 AM