none
Performance penalty with 1000s of calls to Observable.Interval?

    Question

  • Is there any performance considerations to having thousands of running Timer objects (spawned by calls to Reactive RX's Observable.Interval extension method)? This would be for a server application. The number of potentially concurrently running timers would be thousands to tens of thousands.
    Thursday, May 09, 2013 1:38 AM

Answers

  • From memory there is a single thread that processes the timings and there is a single priority heap that keeps track of all the scheduled observables. It scales incredibly well. So even with tens of thousands of `Observable.Interval` queries you only have one actual timer and one actual thread managing it all.

    James C-S

    • Marked as answer by Arash Emami Friday, May 10, 2013 8:27 PM
    Thursday, May 09, 2013 10:37 AM

All replies

  • Observable.Interval will call you back from a Thread Pool thread so there is no issue per se with creating thousands of these. You will want to make sure that you don't execute any code in the callback that could potentially starve the threadpool. You might also consider setting the ThreadPool.SetMinThreads to avoid any delay in adding new threads to the pool if you have upfront knowledge that this will be highly concurrent.

    That said are you sure you need to create thousands of these since that in itself would be an unusual design but perhaps you have a niche requirement.

    Thursday, May 09, 2013 8:36 AM
  • From memory there is a single thread that processes the timings and there is a single priority heap that keeps track of all the scheduled observables. It scales incredibly well. So even with tens of thousands of `Observable.Interval` queries you only have one actual timer and one actual thread managing it all.

    James C-S

    • Marked as answer by Arash Emami Friday, May 10, 2013 8:27 PM
    Thursday, May 09, 2013 10:37 AM
  • Observable.Interval will call you back from a Thread Pool thread so there is no issue per se with creating thousands of these. You will want to make sure that you don't execute any code in the callback that could potentially starve the threadpool. You might also consider setting the ThreadPool.SetMinThreads to avoid any delay in adding new threads to the pool if you have upfront knowledge that this will be highly concurrent.

    That said are you sure you need to create thousands of these since that in itself would be an unusual design but perhaps you have a niche requirement.

    I'm creating thousands of auto-save objects - I would rather they operate independently and not have to hook into some sort of global "timer" object.
    Friday, May 10, 2013 2:48 AM
  • From memory there is a single thread that processes the timings and there is a single priority heap that keeps track of all the scheduled observables. It scales incredibly well. So even with tens of thousands of `Observable.Interval` queries you only have one actual timer and one actual thread managing it all.

    James C-S

    That is great news!  Thank you James, marking your post as the answer, though any additional feedback/elaboration/confirmation from anyone on the RX team is greatly appreciated
    Friday, May 10, 2013 2:50 AM
  • Hi,

    > there is a single thread that processes the timings and there is a single priority heap

    Unless I'm missing something in the source code, this doesn't appear to be true except for EventLoopScheduler.  At least it's not anymore, if it were ever true.

    Calling an Interval or Timer operator,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.Linq/Reactive/Linq/QueryLanguage.Time.cs

    without passing in a scheduler on platforms that support optimizations causes an instance of the internal Timer class to be used,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.Linq/Reactive/Linq/Observable/Timer.cs

    which uses the DefaultScheduler,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.Core/Reactive/Concurrency/DefaultScheduler.cs

    to create a periodic timer from Rx's concurrency abstraction layer,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.PlatformServices/Reactive/Concurrency/ConcurrencyAbstractionLayerImpl.cs

    which creates an instance of the native threading Timer class.  The CAL for Windows Store apps does something similar.

    There appears to be no coalescing of timers, at least not in Rx's CAL.  I don't know whether this is true for native timers, though I suspect that it's not.  I believe that multiple timers will execute their callbacks concurrently on different pooled threads, and not just with respect to each other but also with respect to subsequent intervals of the same timer. It all appears to be highly concurrent.  Though of course Rx uses an AsyncLock to ensure serialized notifications in the operator layer, as part of the Rx contract.

    Note that ThreadPoolScheduler,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.PlatformServices/Reactive/Concurrency/ThreadPoolScheduler.cs

    shares a similar implementation of periodic scheduling using the native threading timer.  No particular advantages there.

    Similar for TaskPoolScheduler,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.PlatformServices/Reactive/Concurrency/TaskPoolScheduler.cs 

    though it uses Task instead of the native threading timer.

    NewThreadScheduler,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.PlatformServices/Reactive/Concurrency/NewThreadScheduler.cs

    is perhaps a bit more optimized since it loops on a single thread.  Though there's no coalescing of timers here either, and of course that wouldn't make much sense if you're explicitly passing in a scheduler that represents the requirement of using a new thread per subscription.

    EventLoopScheduler,

    https://rx.codeplex.com/SourceControl/latest#Rx/NET/Source/System.Reactive.PlatformServices/Reactive/Concurrency/EventLoopScheduler.cs

    does in fact use a single thread with a priority queue.  It also uses a single native threading Timer across the entire queue (it relies on the CAL's CreateTimer method).

    I admit, before examining the source code myself, I was actually under the impression that there was some kind of timer coalescing going on for all schedulers.  So either I've missed something during my analysis or perhaps I've simply misunderstood comments from the Rx team, IIRC particularly in the release notes for Rx 2.0 Beta/RC on the Rx Team blog.

    - Dave


    http://davesexton.com/blog

    • Edited by Dave Sexton Wednesday, May 29, 2013 3:26 PM Small clarifiaction.
    • Proposed as answer by Dave Sexton Thursday, August 14, 2014 2:36 PM
    Wednesday, May 29, 2013 3:24 PM
  • I think you are missing something in your analysis... [Which for you Dave, is pretty unusual I have to say :)]

    I'll miss out a few steps for expediency... when specifying the DefaultScheduler Rx ultimately uses a System.Reactive.Linq.Observαble.Timer._ and it's Run method enqueues the work on one of two PriorityQueues - a short term and a long term queue depending on the due time (short term = within 10 seconds).

    If the short term queue is selected, it will, as you say, create a System.Threading.Timer for each work item - as each timer expires it will dequeue the next work item off the short term queue (no need to associate the work item with it's timer because the queue is a priority queue so the top item is always the next one to go...)

    HOWEVER...

    If the long term queue is selected, it enqueues the work item and then checks to see if the next due work item has changed as a result - and *only* if it has then it creates a new Timer. Even then this is assigned to a static SerialDisposable that holds the previous long term queue timer which is therefore disposed. >There's your time coalescing in action<.

    This logic is mostly in the abstract System.Reactive.Concurrency.LocalScheduler class that the DefaultScheduler derives from.

    To prove this, run the following through a memory profiler:

    static void Main(string[] args)
    {
        Console.WriteLine("Ready");
        Console.ReadKey(true);
        var subscriptions = new IDisposable[10000];
        var lockTime = DateTime.Now;
        for (int i = 0; i < 10000; i++)
        {
            var dueTime = lockTime.AddDays(1).Add(TimeSpan.FromSeconds(i));
            //var dueTime = DateTime.Now.AddSeconds(5);
            subscriptions[i] = Observable.Timer(dueTime).Subscribe(_ => { });
        }
        Console.WriteLine("Scheduled");
        Console.ReadKey(true);
    }

    If you go with the code as is, you'll see just two System.Threading.Timer instances created.

    If you switch the commented line in, and snap the memory as soon as "Scheduled" appears (got to get it within 5 seconds!) you'll see 10000 System.Threading.Timers.

    I think this is pretty reasonable behaviour. Easy to miss though.


    James World / http://www.zerobugbuild.com


    Thursday, May 30, 2013 3:12 PM
  • Hi James,

    Thanks for double-checking.  I think you're probably correct for single-shot timers, so that's really nice to know.  Unfortunately I don't have time at the moment to repeat the full analysis for my own personal understanding, but I'll find the time soon :)

    However, the OP specifically asked about Interval, so my analysis leaned toward periodic scheduling.  Please correct if I'm wrong, but doesn't the internal Timer class use its π sink when a period is specified?  That's the code path to which I was referring.  As mentioned previously, it relies on the IScheduler.SchedulePeriodic method.  Perhaps coalescing is going on somewhere in there too, but I didn't see any.

    - Dave


    http://davesexton.com/blog

    Thursday, May 30, 2013 3:53 PM
  • Yes, SchedulerPeriodic  uses π sinks which will always use a Timer per invocation, except for EventLoopScheduler, as you said.

    Reading the notes in the source, SchedulerPeriodic puts a lot of emphasis on avoiding timer drift in favour of accuracy, the timer keeps firing allowing invocations to queue up if the work duration goes longer than the interval - I guess if executing precisely on the beat is not important to you, and your intervals are > 10 seconds, then you might be better off with recursive scheduling (which I discuss here: http://www.zerobugbuild.com/?p=259 - Bart discusses timing in *a lot* more detail in the article Dave referred to here: http://blogs.msdn.com/b/rxteam/archive/2012/06/20/reactive-extensions-v2-0-release-candidate-available-now.aspx).


    James World / http://www.zerobugbuild.com


    • Edited by James World Friday, May 31, 2013 7:40 AM Corrected the behaviour of how overlapped invocations work on the periodic timer.
    Thursday, May 30, 2013 6:23 PM