Answered by:
Batching in Sync Framework 2.0 RTM

Question
-
Hello,
I did not dive into the new 2.0 RTM until now because I application runs fine.
Now I am running into some memory problems and thinking to implement batching.
I´ve tried it before but could not solve it, because I am using DateTime for change tracking instead of timestamps in my solution.
Therefore my question now...
AFAIK the new 2.0 RTM provides some new features for batching.
Do I still need timestamps or is there another stable way to implement batching?
And are there any other memory improvements in the new release?
For example the posted surrogates:
http://blogs.msdn.com/mahjayar/archive/2008/10/01/dbsyncprovider-improving-memory-performance-in-wcf-based-synchronization.aspx
Regards,
Martin- Moved by Max Wang_1983 Tuesday, April 19, 2011 11:04 PM Forum consolidation (From:SyncFx - Microsoft Sync Framework Database Providers [ReadOnly])
Friday, December 11, 2009 3:11 PM
Answers
-
I have seen a couple interesting posts around batching wtih datetime these days. as Jin said, this approach is possible techinically. however I think it is worth to emphasize that timestamp or min_active_rowversion is the highly recommended anchor type to accuracy and data convergence's puspose. there are a lot of sample code using Datetime as the anchor type as this is the easy and most straitfoward way to show the idea of sync services and for demo purposes.
Also, the new db sync providers shipped with MSF V2 (http://msdn.microsoft.com/en-us/sync/default.aspx ) has much robust batching support so I also suggest to evaluate that to see if this can be a better fit to your sceanrio and business needs.
BTW, I have played with the batching with anchor type a bit and proved ( as the concept level ) that it can be used with batching, there are a few issues ( and some a bit difficult to resolve ) around it and I believe there will be other issues too if we play with it further -- that is also a reason I wont' recommend this.
below is the stord proc I used.
/* * Important Note: this script is merely used to verify that datetime can be used as anchor type for Sync Service V1 batching. * Datetime as anchor is hightly NOT recommended for sync services, regardless batching is used or not * This proc assumes 1 second for batch size as default and '1901-01-01 00:00:000' is the earliest datetime used in your tracking layer * */ create procedure dbo.sp_new_batch_anchor ( @sync_last_received_anchor datetime, @sync_batch_size Int, @sync_max_received_anchor datetime output, @sync_new_received_anchor datetime output, @sync_batch_count Int output) as print @sync_last_received_anchor print @sync_batch_size print @sync_max_received_anchor print @sync_new_received_anchor print @sync_batch_count -- convert the datetime value to timestamp for easy comuputing declare @sync_last_received_anchor_dbts bigint declare @sync_max_received_anchor_dbts bigint declare @sync_new_received_anchor_dbts bigint declare @minDateTimeVal bigint declare @minDateTime datetime declare @maxDateTimeVal bigint declare @maxDateTime datetime declare @tempBatchCount bigint print 'start proc' select @minDateTime = '1901-01-01 00:00:000' select @minDateTimeVal = convert ( bigint, convert( timestamp, convert ( binary(8), @minDateTime ) ) ) select @maxDateTime = getdate() select @maxDateTimeVal = convert ( bigint, convert( timestamp, convert ( binary(8), @maxDateTime ) ) ) select @tempBatchCount = @sync_batch_count print 'min and max datetime values' print @minDateTime print @minDateTimeVal print @maxDateTime print @maxDateTimeVal if ( @sync_last_received_anchor is null OR @sync_last_received_anchor <= @minDateTime ) select @sync_last_received_anchor_dbts = @minDateTimeVal else select @sync_last_received_anchor_dbts = convert ( bigint, convert ( timestamp, convert ( binary(8), @sync_last_received_anchor) ) ) print 'last rec anchor value' print @sync_last_received_anchor_dbts if ( @sync_max_received_anchor is null OR @sync_max_received_anchor <= @minDateTime ) select @sync_max_received_anchor_dbts = @maxDateTimeVal else select @sync_max_received_anchor_dbts = convert ( bigint, convert (timestamp, convert ( binary(8), @sync_max_received_anchor ) ) ) if @sync_batch_size is null or @sync_batch_size <= 0 set @sync_batch_size = 300 -- 1 SECOND print 'batch size : ' print @sync_batch_size -- simplest form of batching if @sync_last_received_anchor_dbts is null or @sync_last_received_anchor_dbts <= @minDateTimeVal begin print 'simple batching for inital sync' set @sync_new_received_anchor_dbts = @maxDateTimeVal print @sync_new_received_anchor_dbts print 'get batch count' if @sync_batch_count is null or @sync_batch_count <= 0 set @tempBatchCount = (@sync_max_received_anchor_dbts / @sync_batch_size) + 1 end else begin print 'simple batching for subsequent sync' set @sync_new_received_anchor_dbts = @sync_last_received_anchor_dbts + @sync_batch_size if @sync_batch_count is null or @sync_batch_count <= 0 begin print 'get batch count' set @tempBatchCount = (@sync_max_received_anchor_dbts / @sync_batch_size) - (@sync_new_received_anchor_dbts / @sync_batch_size) + 1 end end --- ensure batch count is Int32 value --- better logic will be needed print 'temp batch count:' print @tempBatchCount if ( @tempBatchCount > 1000 ) set @sync_batch_count = 100; else set @sync_batch_count = @tempBatchCount print @sync_batch_count -- check if this is the last batch if @sync_new_received_anchor_dbts >= @sync_max_received_anchor_dbts begin set @sync_new_received_anchor_dbts = @sync_max_received_anchor_dbts if @sync_batch_count <= 0 set @sync_batch_count = 1 end print @sync_batch_count print 'max rec and new rec anchor dbts values' print @sync_max_received_anchor_dbts print @sync_new_received_anchor_dbts print 'convert back to datetime for return values' -- convert the values back to datetime before proc returns select @sync_max_received_anchor = convert ( datetime, convert (timestamp, convert ( binary(8), @sync_max_received_anchor_dbts ) ) ) select @sync_new_received_anchor = convert ( datetime, convert (timestamp,convert ( binary(8), @sync_new_received_anchor_dbts ) ) ) print 'max rec and new rec anchor date values' print @sync_max_received_anchor print @sync_new_received_anchor -- end of proc
This posting is provided "AS IS" with no warranties, and confers no rights.- Marked as answer by Sean_KelleyModerator Friday, December 18, 2009 11:18 PM
Friday, December 18, 2009 5:59 PMModerator
All replies
-
Are you using offline scenario with DbServerSyncProvider & SqlCeClientSyncProvider instead of collaboration scenario with SqlSyncProvider or DbSyncProvider?
The Sync framework v2.0 batching support is for collaboration scenario.
Please check whether that applies to your situation.
If you are using Sync V2 collaboration providers, then enabling batching should be simple, just need to specify pooling directory and max memory sizes on client and server sides etc.
There is a code sample with batching support at place like C:\Program Files\Microsoft SDKs\Microsoft Sync Framework\2.0\Samples\SharingAppDemo-CEProviderEndToEnd if you have installed Sync Framework V2. You can take a look.Friday, December 11, 2009 6:38 PMAnswerer -
Hi Jin, yes I am using an offline Szenario. Windows 2003 Server with SQL 2005 Server on the one side, WM 2005 and WM 6.1 with SqlCE on the other side. So batching still does not really work in this situation without using timestamps? Regards, MartinMonday, December 14, 2009 8:23 AM
-
It's possible to use Datetime with batching in offline scenario.
The batching document is at http://msdn.microsoft.com/en-us/library/bb902828.aspx
The anchor calculation needs to changed to work on Datetime as the scenario needed.Monday, December 14, 2009 6:46 PMAnswerer -
Sorry! I only believe that if you show me a working example :-)
I tried to change the anchor calculation of that example to DateTime and it did not work
as well as many others here in the board tried to make Batching working with DateTime.
Until now I did not heard about anybody who made the current offline-scenario-batching working with DateTime columns :-(
Regards,
MartinTuesday, December 15, 2009 8:41 AM -
Bump.
Just curious if a batching example using 'DateTime' does exist.
DamianThursday, December 17, 2009 10:35 PM -
I have seen a couple interesting posts around batching wtih datetime these days. as Jin said, this approach is possible techinically. however I think it is worth to emphasize that timestamp or min_active_rowversion is the highly recommended anchor type to accuracy and data convergence's puspose. there are a lot of sample code using Datetime as the anchor type as this is the easy and most straitfoward way to show the idea of sync services and for demo purposes.
Also, the new db sync providers shipped with MSF V2 (http://msdn.microsoft.com/en-us/sync/default.aspx ) has much robust batching support so I also suggest to evaluate that to see if this can be a better fit to your sceanrio and business needs.
BTW, I have played with the batching with anchor type a bit and proved ( as the concept level ) that it can be used with batching, there are a few issues ( and some a bit difficult to resolve ) around it and I believe there will be other issues too if we play with it further -- that is also a reason I wont' recommend this.
below is the stord proc I used.
/* * Important Note: this script is merely used to verify that datetime can be used as anchor type for Sync Service V1 batching. * Datetime as anchor is hightly NOT recommended for sync services, regardless batching is used or not * This proc assumes 1 second for batch size as default and '1901-01-01 00:00:000' is the earliest datetime used in your tracking layer * */ create procedure dbo.sp_new_batch_anchor ( @sync_last_received_anchor datetime, @sync_batch_size Int, @sync_max_received_anchor datetime output, @sync_new_received_anchor datetime output, @sync_batch_count Int output) as print @sync_last_received_anchor print @sync_batch_size print @sync_max_received_anchor print @sync_new_received_anchor print @sync_batch_count -- convert the datetime value to timestamp for easy comuputing declare @sync_last_received_anchor_dbts bigint declare @sync_max_received_anchor_dbts bigint declare @sync_new_received_anchor_dbts bigint declare @minDateTimeVal bigint declare @minDateTime datetime declare @maxDateTimeVal bigint declare @maxDateTime datetime declare @tempBatchCount bigint print 'start proc' select @minDateTime = '1901-01-01 00:00:000' select @minDateTimeVal = convert ( bigint, convert( timestamp, convert ( binary(8), @minDateTime ) ) ) select @maxDateTime = getdate() select @maxDateTimeVal = convert ( bigint, convert( timestamp, convert ( binary(8), @maxDateTime ) ) ) select @tempBatchCount = @sync_batch_count print 'min and max datetime values' print @minDateTime print @minDateTimeVal print @maxDateTime print @maxDateTimeVal if ( @sync_last_received_anchor is null OR @sync_last_received_anchor <= @minDateTime ) select @sync_last_received_anchor_dbts = @minDateTimeVal else select @sync_last_received_anchor_dbts = convert ( bigint, convert ( timestamp, convert ( binary(8), @sync_last_received_anchor) ) ) print 'last rec anchor value' print @sync_last_received_anchor_dbts if ( @sync_max_received_anchor is null OR @sync_max_received_anchor <= @minDateTime ) select @sync_max_received_anchor_dbts = @maxDateTimeVal else select @sync_max_received_anchor_dbts = convert ( bigint, convert (timestamp, convert ( binary(8), @sync_max_received_anchor ) ) ) if @sync_batch_size is null or @sync_batch_size <= 0 set @sync_batch_size = 300 -- 1 SECOND print 'batch size : ' print @sync_batch_size -- simplest form of batching if @sync_last_received_anchor_dbts is null or @sync_last_received_anchor_dbts <= @minDateTimeVal begin print 'simple batching for inital sync' set @sync_new_received_anchor_dbts = @maxDateTimeVal print @sync_new_received_anchor_dbts print 'get batch count' if @sync_batch_count is null or @sync_batch_count <= 0 set @tempBatchCount = (@sync_max_received_anchor_dbts / @sync_batch_size) + 1 end else begin print 'simple batching for subsequent sync' set @sync_new_received_anchor_dbts = @sync_last_received_anchor_dbts + @sync_batch_size if @sync_batch_count is null or @sync_batch_count <= 0 begin print 'get batch count' set @tempBatchCount = (@sync_max_received_anchor_dbts / @sync_batch_size) - (@sync_new_received_anchor_dbts / @sync_batch_size) + 1 end end --- ensure batch count is Int32 value --- better logic will be needed print 'temp batch count:' print @tempBatchCount if ( @tempBatchCount > 1000 ) set @sync_batch_count = 100; else set @sync_batch_count = @tempBatchCount print @sync_batch_count -- check if this is the last batch if @sync_new_received_anchor_dbts >= @sync_max_received_anchor_dbts begin set @sync_new_received_anchor_dbts = @sync_max_received_anchor_dbts if @sync_batch_count <= 0 set @sync_batch_count = 1 end print @sync_batch_count print 'max rec and new rec anchor dbts values' print @sync_max_received_anchor_dbts print @sync_new_received_anchor_dbts print 'convert back to datetime for return values' -- convert the values back to datetime before proc returns select @sync_max_received_anchor = convert ( datetime, convert (timestamp, convert ( binary(8), @sync_max_received_anchor_dbts ) ) ) select @sync_new_received_anchor = convert ( datetime, convert (timestamp,convert ( binary(8), @sync_new_received_anchor_dbts ) ) ) print 'max rec and new rec anchor date values' print @sync_max_received_anchor print @sync_new_received_anchor -- end of proc
This posting is provided "AS IS" with no warranties, and confers no rights.- Marked as answer by Sean_KelleyModerator Friday, December 18, 2009 11:18 PM
Friday, December 18, 2009 5:59 PMModerator -
Hi,
thanks for the answer. A last question...
is there a blog post or anything else about the pros/cons of using timestamps instead of DateTime objects?
I am thinking about adding 2 new timestamp columns to my tables and leave the DateTime columns as it is for the rest of my app.
Do I understand right that timestamps fix the problem of e.g. mobile devices with wrong time settings (because of flashing the memory or something else etc.).
Regards,
MartinTuesday, December 22, 2009 11:06 AM -
Yunwen,
I think I speak for alot of people that 'simple examples' of how sync technology works is appreciated, but if I had of known that 'timestamp / min_active_rowversion' was the more preferred method of Sync tracking (and for batching) I would have spent the extra time to understand it and implement it. Now I have a Sync application with 18 months of development into the 'DateTime' method that follows a 'Highly Not Recommended' technique.
I dont want to distract from this original threads intention, but if MSDN examples could be more up front with which different techniques are available, and at least provide examples of all techniques instead of 'what is easy to explain' it would go a long way. Developers dont mind steep learning curb as long as its the correct method.
On that note, thank you very much for your source code!!! It was very helpful.
DamianWednesday, December 23, 2009 4:33 AM