Sync Blob Container with Local Folder

Question

I am trying to develop the azure blobstorage sync service.

1. Is there any service which which we can sync our blob container with our local folder.

2. Could we achieve it via programming. 

3. Could anyone suggest the algorithm to sync the blob storage with local folder.

Regards 

Munish Bhargav  

Answer 1

Hi Munish,

I think you can certainly achieve it with programming, I would certainly be interested in comparing the following:

DateTime lastWriteTimeUtc = File.GetLastWriteTimeUtc(@"c:\file.txt");

And the last modified entry in BlobProperties (the blob metadata).

Beyond that you may be interested in hashing the contents of the local file and seeing if you can compare it with the ETag given by BlobStore.

BWC;

Answer 2

Thanks bwc for quick reply,

bwc would you elaborate your views. If you achieve this then how you did it.

I have some problem about.

1. Bigest problem is how to remember that particular container is sync with which local folder.

2. If blob name has been modified that how to resolve it.

3. There is also one conflict in delete and add. Suppose we have sync folder both have abc folder at azure storage

and local folder then we delete the folder from local folder. but how the service will come to know that whether the

bolder has been deleted from local storage or added to the azure blob storage. Think of it.

Renaming is also big issue  

Munish

Answer 3

Hi Munish,

Thing that you need to consider first is what do you want to synchronize and what's the purpose of the synchronization. Do you want to make use of cloud as a back up device? Do you want 2 way synchronization (i.e. the contents are the same in both your local & cloud storage) or 1 way synchronization (local storage is always backed up in the cloud)? Do you need versioning support (for example you may want to keep last 5 copies of the same document in the cloud). I think these questions will determine your overall application architecture. 

Regarding the code, I think there is a CloudSync project in Azure samples that you can take a look. Also there are some utilities available on CodePlex as well which you can refer so as to have a better understanding on how you'll actually implement the synchronization.

Just my 2 cents.

HTH

Gaurav

Answer 4

Hello, you were very interested in Live Framework. So I'm guessing you want to create something similar to Live Framework. First of all, while the CTP version of Live Framework has been terminated, the project is not. We will release new versions of Live Framework in the future.

Then you must be aware of the difference between Live Mesh/Framework and blob storage. Live Mesh works for per user. That is, one Windows Live account has its own cloud storage. Blob storage, on the other hand, is usually shared, unless the application is written for yourself only. If that's the case, you'd better use the free Live Mesh rather than the paid blob storage. If you want to distribute the application to other people, you must create a container for each user, and find a way to authenticate the user. You must also be aware that if you ship your storage account's authentication key with the client application, you take the risk that one user may use the information to hack files stored for the other user. So I'm afraid it is not a good idea to use blob storage for this kind of application.

Well, one workaround to the security problem is to create a intermediate service that will work with blob storage, and your client application will work with this intermediate service instead of shipping the storage account key with it.

Now for your questions:

1. You need a client application. You can use user settings to remember which folder should be synced or not. If you want something similar to Live Mesh, you need to write some Windows Shell extensions, which can be very difficult. But if you just want to provide a custom UI for the user to select the synced folders, you can simply use user settings.

If you need to store the information in the cloud, you can take advantage of the container's metadata.

2. Please be aware that blob storage doesn't support rename directly. What you can do is to copy the original blob to a new blob with the new name, and then delete the original blob. If you only need one way sync, you can simply delete any files/blobs that do not exist in the source, and add the new ones. Otherwise, you will need to keep a trace of user operations, to decide whether a file/blob is newly created, or it is no longer needed. Assume only your application can access the blob, you can easily trace every operation. On the client side, you can PInvoke into the ReadDirectoryChangesW function to monitor the directory changes.

3. Same as above.

---

Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 5

Thanks Gaurav and Yi-Lun Luo for reply,

First of all i want to make sure to both of you that i am very much clear about the azure services.

I know the difference between all the services and hoe to use then. I do create few apps.

To be exact I more specific i want to make the Live mesh like application on azure blob storage.

Which i think 70% possible and i just need the sync on blob storage same as provided by live mesh.

Thanks

Munish   

Answer 6

Gaurav could you provide the link of the CloudSync Project you are taking about and other projects available on CodePlex.

Munish    

Answer 7

Hi Munish,

Actually it was CloudDrive project. You can find it under Samples folder in Windows Azure SDK installation (C:\Program Files\Windows Azure SDK\v1.0\samples\CloudDrive).

Others you can search on CodePlex (www.codeplex.com). Just search for "Azure"

Thanks

Gaurav

Answer 8

Cloudberry provide very nice service for syncing the Amazon S3 bucket or folders with the 

local folder with the help of Powershell, Cloudberry Snap-In and Windows task scheduler see the link

http://blog.cloudberrylab.com/2009/03/how-to-share-files-in-amazon-s3-bucket.html

Would there any such kind of service or command line tool or API available in azure with which we

can achieve Azure Blobs storage sync with Local Folders.

Regards

Munish Bhargav 

 

 

Answer 9

I got an email from Cloudberry just the other day saying they were being to support Azure.

You could maybe look into that.

BWC;

Answer 10

Thanks bws for reply. I need not the support from cloudberry . I need same kind of functionality

from microsoft.

I am just that explained cloudberry develop the service for azure product does microsoft

look and work toward this. If yes then just let me know.

Thanks