Preloading data for use in the cloud

Question

Hi guys,

This might seem like a silly question, but I have a scenario where I'd like a few million records to be stored in TableStorage before my app is actually live. Consider it "lookup" data.

It's pretty much all textual data that I'm going to parse into a bunch of properties.

Any advice on the best way to achieve this efficiently?

Is it possible to 'upload' the contents of my local dev based TableStorage to teh cloud during a deployment?

Answer 1

Hello, you can load the data to table storage from a local desktop application, before publishing your cloud service to the portal. However, currently there's no tool for you to import data from local storage to cloud storage. You'll have to upload the data manually in your code.

---

Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 2

Hi Yi-Lun,

Let me see if I get this straight. So if I have a bunch of data in my local table storage sitting on my machine, when I click "Deploy To Cloud" in VS, will the deployment include the pre-existing data in my local TableStorage along with it or do I have to go through the process of populating the data in the cloud myself?

Answer 3

You have to populate it yourself.

I would also recommend making looking at using SQL Data Services for hosting that many records. It will provide you with greater functionality. There's also been some initial exploratory work done with it and the MS Sync Framework for data replication. This might be able to address your data load issue.

Answer 4

@Senkwe: I think what Yi-Lun means, which is the approach I plan to take right now is to write a small console app that does this migration for you ... before or after publishing your web or worker roles but definitely before starting your app on the cloud.

Conceptually, the console app on your developer machine would open two connections: SOURCE to your dev local storage and TARGET to the cloud storage. And then loop through your dev store code tables, harvesting entities as you go and writing those entities into the cloud store.

The resulting client app will be something similar to your SSIS migration task scripts.

---

Pita.O: http://www.arizentax.com/

Answer 5

Thanks Pita,

That looks like what I'm gonna have to do.

Would be a bit of a mission for my scenario tho. I'll likely have quite a few records to upload one by one.

Thanks.

Answer 6

On the contrary, it will actually be a simple task to write the app, since it's just a few lines of code for each entity set (table). It will just take a long time to run for the kind of data volume you mention

// using StorageClient

var aa = from a in localDataServiceContext.CreateQuery<MyTable>("MyTables") select a;

foreach(var a in aa)
{
     // am not sure if you need to detach a from localService Context at this point but that's just detail, right?
   cloudDataServiceContext.AddObject("MyTables", a);
}
cloudDataServiceContext.SaveChanges();  // you may want to call this line within the loop after every x objects are added.

---

Pita.O: http://www.arizentax.com/