Subject observed on thread pool sometimes hangs

Question

Original goal: execute several actions asynchronously in serial order using Subject to avoid overlapping. It's some kind of notifications.

The following simplified code uses Subject<T> wrapped into ObserveOnObserver with ThreadPoolScheduler and sometimes it hangs so that new items are queued into subject, but nothing is processing. Unfortunately we cannot reproduce it in our testing or debugging environment, the issue happens from time to time only in production for very few customers. Probably, it depends on hardware or software environment.

internal sealed SomeSingletonClass
{
    private readonly IObserver<Action> eventsObserver;

    public SomeSingletonClass()
    {
        var subject = new Subject<Action>();
        var subscription = subject.ObserveOn(Scheduler.ThreadPool).Subscribe(action => action()); // sometimes hangs
        // var subscription = subject.ObserveOn(new EventLoopScheduler()).Subscribe(action => action()); // seems to work
 
        eventsObserver = subject;
    }

    private void Foo()
    {
        // ...
        eventsObserver.OnNext(() => { /* some foo code */ });
    }

    private void Bar()
    {
        // ...
        eventsObserver.OnNext(() => { /* some bar code */ });
    }
}

As a workaround we replaced  ObserveOn(Scheduler.ThreadPool) with ObserveOn(new EventLoopScheduler()) and it seems to be a quick fix, at least there no more support requests about this issue. Since the root cause wasn't determined we cannot be sure that issue is completely solved, plus having mostly sleeping dedicated thread isn't nice from performance point of view.

Recently we got full memory dump with hanged subject (that customer used buggy version with thread pool), it should be enough for investigation. ObserveOnObserver's queue contains dozens of jammed actions (so Subject itself works well), isAcquired flag is true (so new items doesn't start processing), observer wasn't faulted or disposed. At the same time there are no deadlocks, no one background thread executes some previously added action and thread pool's queue is empty (ThreadPoolRequestQueue's head and tail references to null).

It looks like some action was executed but didn't recursively re-add another action into the scheduler. How this may happen without thread aborting?

Rx version: 1.0.10621.0. Code which could affect this issue was partially rewritten in new Rx versions for refactoring needs, but probably this issue was also fixed. Could someone explain where is the bug in observing subject on thread pool?

Details: desktop WPF-application written with .Net 3.5, compiled for any cpu, executed on x86. Should I provide additional information?

Similar unanswered question in another topic: http://social.msdn.microsoft.com/Forums/mr-IN/rx/thread/92402f6d-8c3a-4cb5-8c37-fabfda987a0a (scroll down to msuser-ngm's comment from Wednesday, October 27, 2010 8:34 PM).

Answer 1

Hi,

Unfortunately, I never found the cause of this hang.

We're currently using version 1.1.11111.0. I haven't noticed this happen with this version (although I haven't tried to do what I did before, which was build an IScheduler around a subject). We're hoping to upgrade to version 2 soon (the last time I tried, I ran into this foreground thread problem).

The Rx code has gone through so many rewrites since then... it's definitely possible that this works correctly with the latest code. Sorry I can't provide more help.