Constructive feedback regarding the lack of local structured storage and sync RRS feed

  • General discussion

  • I noticed there are some threads related to local structured storage already in this forum, but I decided to start a new one because I have a slightly different viewpoint. While I have a million positive things to say about WinRT and Metro style apps, I am frankly very concerned with the lack of a story for offline structured storage and sync in Metro style apps.

    I feel that the lack of support for local structured storage is an extremely unfortunate, and very limiting, omission from WinRT and the "Windows Tailored" profile of the .NET Framework.

    I'm am getting the impression that Microsoft is assuming that:

    • All Metro style apps will target "always connected" scenarios
    • Windows.Storage will suffice for those apps that don't

    Please do not make that mistake. Please reconsider. We absolutely need to be able to create Metro style apps that deal with data and continue to work well when the device is offline.

    It's awesome that there is good support for building OData clients and other REST clients - but this only addresses the online scenario.

    The "structured" part of Windows.Storage is a very limited model, essentially limited to name/value pairs, insufficient for all but the most basic scenarios. Yes there is local file storage, which is great of course. But forcing every app developer out there to build her own DBMS on top of local file storage will simply not cut it, especially with all of System.Data having been removed from the profile. If local file storage was sufficient for most device apps, then things like SQLCE would have no purpose today already. And SQLCE clearly has a purpose, and has played a very important role for occasionally connected device apps for a very long time.

    There is also a tremendous need for synchronization with a server-side database such as SQL Azure, mostly to be able to roam data between devices. Yes there is the roaming storage model in WinRT, but it shares the same limitations of local storage mentioned above, and on top of this is very limited in capacity (currently 30KB if memory serves). It is simply insufficient for all but the simplest roaming data needs. Again, forcing every app developer to design and implement her own synchronization solution is very bad. You can do much better to enable developers.

    In summary, I strongly urge you to consider adding a story for local structured (SQL-like) storage - coupled with a story for synchronization with a server-side data store - to Metro style apps.

    There are many potential ways to do this.

    I believe the best way would be to extend Windows.Storage with (LINQ queryable) structured table storage capabilities, and to allow using roaming storage as the underlying store while also dramatically increasing the roaming capacity.

    Another possible way would be to include the System.Data namespace in the profile and allow people to use SQLCE for local structured storage (on top of local file storage). This would preferably be accompanied with releasing an updated version of Sync Framework with an easy out-of-box way to quickly enable sync between a local SQLCE and SQL Azure.

    Daniel Stolt, Perceptible, http://www.perceptible.net
    • Edited by Daniel Stolt Saturday, September 24, 2011 4:23 PM
    Saturday, September 24, 2011 4:17 PM

All replies

  • I can't agree more.

    Finally we got a SQLCE with a (limited) Entity Framework on Windows Phone 7.1 Now the same missing feature ist there again. Sure - on a Win8 device we have far more horsepower to implement a simple local storage by ourself - but its a pain to do it again...

    Hopefully MSFT has learned from WP7 :)

    Saturday, September 24, 2011 6:19 PM
  • Yes, exactly.

    When people at BUILD asked speakers about this, the answer that came back was almost invariably "you need to use file storage". And while there is nothing to stop us (except of course the omission of System.Data) from all going off and implementing our own storage solutions on top of file storage, it doesn't make sense to have third parties reinvent all those wheels considering all the necessary pieces (ADO.NET, EF, SQLCE, Sync Framework) exist already in other parts of the .NET developer offering.

    Daniel Stolt, Perceptible, http://www.perceptible.net
    Sunday, September 25, 2011 8:46 PM
  • I couldn't agree more. This appears to be a serious limitation. I urge you to consider Mr Stolt's suggestions above.
    Monday, October 3, 2011 9:36 PM
  • This is a very serious concern for large scale nationwide/worldwide applications where connectivity can not be guaranteed and structured storage is an absolute requirement to support "Disconnected User" use cases.  This is not an extreme or outlier use case but a pervasive mandatory one by thousands of organization.

    Microsoft needs to address this before release of Windows 8. There needs to be a solution or it will seriously diminish the Metro Style Application adoption. 

    Friday, November 18, 2011 12:02 PM
  • Yes, indeed. Please don't repeat the same mistake with Windows Metro. Others platforms have already some sort of structured local storage (sqllite is one of the more popular) out of the box.
    Sunday, November 27, 2011 8:28 PM
  • Windows 8 is going to run on full computers. And yet we developers have no access to proper local data, storable with System.Data.DataSets? I understand the imperative for the new UI, but I can't understand why Microsoft thinks it has to turn computers into a limited device that can't do much with the local file system!!

    Good lord, even Blackberry devices give me access to local SQL style storage using sqllite!! The absence of System.Data, even one without SQL providers, is appalling to me - and horrifying. Good thing we never got WinFS, I suppose!

    The idea that I can more easily program in a more traditional "powerful client" paradigm for a cheapie LG Android phone than I can for some monster gaming desktop running Metro... I can't even fathom it!

    Friday, December 16, 2011 2:12 PM
  • We often need to be able to provision large gigabytes of data onto the device before the user goes into the field. This is often achieved with a USB key or an SD card slot with the specific data on it, and the app will pull data from it and save any edits back to it. Having to have to provision the data into the local app storage instead is a major pain in the ***.
    Friday, December 16, 2011 6:16 PM
  • This truly makes no sense.  The need for an application to move smoothly between connected and disconnected states is critical. And, even Microsoft knows this because they just built IndexedDB into IE10 to support these same kinds of scenarios.  I don't want my app to crash because the network drops.  I don't even want my users to be forced to wait for data from the web every tome they open my app. I want to cache the user's data locally and sync their data to the cloud using an async thread.

    And, we know there will be some kind of storage engine under IndexedDB … just give us access to it!

    Sunday, March 11, 2012 8:16 PM
  • Hear, hear!

    Suggest either implementing API similar to LINQ to SQL for WP 7.5, or extend the use of IndexedDB from only JS to also C# and XAL code.

    Please mark as answer, if this was it. Visit my SQL Server Compact blog

    Wednesday, May 2, 2012 8:22 AM
  • I think it is safe for us to assume there'll be some structured store in the future, looking at the random JET exceptions that are happening in the OS. <wink>

    In the meantime sqlite port to WinRT https://github.com/timheuer/Callisto looks promising.

    Wednesday, May 2, 2012 11:34 AM
  • How about the Sqlite implementation in Callisto? It fully supports linq.


    Monday, May 7, 2012 2:40 PM
  • I totally agree, we need a local db and good sync tools. You already have all of this, why let us reinventing the wheel?

    Thursday, May 10, 2012 12:22 PM
  • I totally agree, we need a local db and good sync tools. You already have all of this, why let us reinventing the wheel?

    I agree.  Reinvent the wheel is not so good.
    Thursday, May 10, 2012 12:30 PM
  • I agree!!!

    We need metro-apps SQL Client or metro-apps with SQL Compact!!!

    Thursday, May 10, 2012 12:46 PM
  • Does Callisto project suffer the same problem of SQLWinRT ? Can I use Callisto's SQLite on my Win 8 Metro project ?

    I'm asking because of this:

    In http://social.msdn.microsoft.com/Forums/en-BZ/winappswithcsharp/thread/1e03b3e7-7659-4407-ac07-a5a3db7a178c

    Tim Heuer [MSFT] wrote: [...] please note that while awesome, the SQLWinRT project on codeplex is a wrapper to communicate with the classic SQLite engine...which uses APIs that would not pass store validation currently.

    Thursday, May 10, 2012 1:52 PM
  • Nicolo: As far as I can tell, the SQLite guys are working towards a version the passes the certification (you will see several recent check-ins tagged WinRT). So my guess it that hopefully it will be ready around when Windows8 ships.
    Friday, May 11, 2012 6:13 PM
  • You can use now Siaqodb- NoSQL solution as local database for Metro style apps:http://siaqodb.com/?p=818 
    Thursday, May 31, 2012 11:51 AM
  • Friday, August 31, 2012 2:54 AM