RX2 exception in EventLoopScheduler

Question

We have immigrated from RX1.1.11111.1 to RX 2.0.20823.2. And now we have encountered a rare exception from the EventLoopScheduler:

an Unhandled Exception occured in non UI thread.
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Reactive.Concurrency.EventLoopScheduler.Run()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Does anyone have a guess what is the problem? Is it because we didn't use onError delegate and one of our methods failed?

Here's the gist of the code:

EventLoopScheduler m_scheduler = new EventLoopScheduler();

. . .

m_receivedMessageHandler.StatusReceived.ObserveOn(m_scheduler) .Subscribe(p_unit => sendAll(m_retransmitManager, m_endPoint));

Can an exception inside sendAll cause this behavior? Or is it something inside RX?

Answer 1

Hi,

Is this a .NET 4.0 or 4.5 app?

I don't think that omitting the OnError handler is related at all.  It's also unlikely that the error is the result of sendAll throwing, given that the NullReferenceException is being thrown by the scheduler's Run method, assuming that's the complete stack trace.  Or is there an inner exception that you haven't posted?

Is it possible that StatusReceived pushes notifications concurrently?  The scheduler should be thread-safe anyway, but perhaps it's worth asking.

Can you reproduce the error in a short-but-complete program?

- Dave

---

http://davesexton.com/blog

Answer 2

Hi,

Also, if you could attach a debugger and enable first-chance breaking on exceptions, then you can eliminate the possibility that your code or some other internal Rx code is throwing an exception and the NullReferenceException is the result of a bug in the fault path.

- Dave

---

http://davesexton.com/blog

Answer 3

We encountered this problem few weeks ago on our production environment (Win2008R2 64bit, .NET 4.0, Rx 2.0.20823.2, compiled 64-bit application)
And these exceptions ended with unhandled exceptions in app that crashed it.

I assume in your case this is 4.0 environment, because swithing to 4.5 seems to fix the problem.

We reviewed and rereviewed our code and even EventLoopScheduler code in reflector but didn't find any places that could throw NRE even in multithreaded environment.
We have some crashing dumps but couldn't get useful info from them...
As for the stack trace I think it's unlikely that some inlining was occured, because throwing exception in user code give you much more deep stack trace even in release configuration.

If you can't upgrade to 4.5, you can try to disable JIT optimizations, adding System.Reactive.PlatformServices.ini and writing this in it:

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

It is fixed our problem before we upgraded to 4.5. It is not a nice solution, but this can help you temporarily.

Seems like a bug in 64-bit JIT 4.0, but still not sure.
We tried to reproduce this, but failed. As usual, it is only reproducible on production environment according to Murphy's law. =)

Answer 4

I work with Lena.

The last description looks exactly as our problem,

we work in 64 bit windows 7,

the NRE is shown only in production station (even so our application is compiled in Debug and not Release)

and we do work in 4.0.

We can't move into .Net4.5 :-(.

I have tried to put the ini file in the executable folder but it still crashed.

1) how do I know the JIT is reading and using the ini file? and that the dll is not optimized?

2) is there another solution of disabling the JIT optimization?

3) is our application needs to be compiled in release?

thanks

ittay

Answer 5

Hi,

The StatusReceived is raising notifications concurrently.

We have tried to add lock on the OnNext but it didn't help.

It never happens on our development stations and it didn't happen with rx 1.11111.1.1.

We tried ylemsoul idea of using AllowOptimize=0 but didn't help also.

Trying to repreduce with a simple program,

mean while any ideas?

thanks

ittay

Answer 6

Hi Ittay,

If locking OnNext didn't help then perhaps it's not a concurrency issue.

I'd still like to know if the failure is due to a bug in an error path; i.e., if the NRE is hiding the real error.

Can you take a memory dump of the failing app in production?  For example, you can configure adplus to automatically save a memory dump when your app crashes.  Then you can use Visual Studio or SOS in WinDebug to analyze the dump and find out more about the state of the program when it crashed.

I know this is a bit advanced, but if you can't repro locally then I see no other way.

- Dave

---

http://davesexton.com/blog

Answer 7

Hi,

No inner exception exist during thiscrash (we log the crash exception + inner exceptions but this case there were none).

I will take a snapshot of the memory when the exception happens and send it to you.

thanks

Ittay