locked
Azure blobs RRS feed

  • Question

  • Hi 

    I am using blobs to load and store data periodically. I am loading data in to same folder. First time I am processing few blobs, next time when I access the folder, I need to process only the blobs that are added newly. Is there any way to identify the blobs that are newly added?(Like setting flags in the blobs that are processed already?

    Regards,

    Sridevi.

    Tuesday, December 23, 2014 9:36 AM

Answers

  • You can have two containers, one for storing new blobs, the other for processed blobs. Just process the blobs in the new container and after you are done, copy the blob to the processed container.

    Frank

    • Marked as answer by Manu Rekhar Monday, December 29, 2014 6:26 AM
    Wednesday, December 24, 2014 4:59 PM

All replies

  • Hi Sridevi,

    You could use List Containers to List all of the containers in a storage account and List Blobs to List all of the blobs in a container.
    We would require some more time to check if there is any provision to list only new Blobs and will revert to you when we have an answer.

    Meanwhile you could refer the following link for a list of REST APIs for the Blob Service.

    http://msdn.microsoft.com/en-us/library/azure/dd135733.aspx

    Regards,
    Malar.

    Tuesday, December 23, 2014 3:37 PM
  • Hi Malar,

    Thank you for your suggestions.

    My idea was to maintain a list of already processed blobs and omit them when a new blob is added. but this will be time consuming as number of blobs grows day by day. 

    Please let me know if you have any idea to do this on large number of blobs.

    Regards,

    Sridevi.

    Wednesday, December 24, 2014 4:32 AM
  • Hi Sridevi_K,

    I suggest that you could  try to set Metadata  for your blob  as a Flag to indicate that the blob has been processed already.

    First, when  you upload the Blob, you could set Metadata for your  blob like below:

    CloudBlockBlob blob = sourceContainer.GetBlockBlobReference("test");
                  using (var fileStream = System.IO.File.OpenRead(@"path/file"))
                  {
                      blob.UploadFromStream(fileStream);
                  } 
                       blob.Metadata["Flag"]="New";
                      blob.SetMetadata();
    

    Then  when you load it into your folder, you can  change the Metadata "Flag" Value  as "Updated" like below:           

       blob.Metadata["Flag"]="Updated";
        blob.SetMetadata();
    

    So Before you  process the blobs ,You can list the blobs in your container, and just query the  blob whose Flag is "New".

    For more information about Metadata in Blob ,please refer to the link below:

    http://msdn.microsoft.com/en-us/library/azure/hh225342.aspx

    Best Regards,

    Kevin Shen.

    Wednesday, December 24, 2014 7:26 AM
  • You can have two containers, one for storing new blobs, the other for processed blobs. Just process the blobs in the new container and after you are done, copy the blob to the processed container.

    Frank

    • Marked as answer by Manu Rekhar Monday, December 29, 2014 6:26 AM
    Wednesday, December 24, 2014 4:59 PM