Modify System.Exception in SetILFunctionBody
- I am wondering if I can get some help on this.
We have working "Proxy" that is invoked via insertion into methods and gathers all the calling arguments, etc. and passes to a handler.
We are using the SetILFunctionBody during JIT compilation. It works fine for 1.1, but in 2.0 ONLY works when using LoaderOptimization other than SingleHost when hooking corlib methods such as the System.Exception .ctor. We are hooking many methods - all NON core methods work well.
I have come to the conclusion that it has to do with Domain Neutrality of mscorlib (only) being in the shared domain.
So, my questions are this: Other than the LoaderOptimizationAttribute OR the args to the CorRuntimeBindEx to load the CLR - how can one override the LoaderOptimization via registry, .config file, etc.?
Second, if we immediately "Sandboxed" the Proxy via CreateInstance into another domain and marshalled the parameters - do you think that would get around the domain neutral issues - we just want the call and the args.
It seems like something really changed in 2.0 as 1.1 lets us do as we please.
And of course, any other suggestions would be appreciated...
Answers
>>
Ok, assuming we use the ModuleLoadFinished (and do you have any links to any good examples here) and put our proxy in there...
how do we ever get out of mscorlib?
<<There is an old IL rewriting sample on MSDN at
http://msdn.microsoft.com/msdnmag/issues/03/09/NETProfilingAPI/,
and one of my blog entries talks a bit about IL rewriting (with a note on the dangers of having mscorlib reference outside itself):http://blogs.msdn.com/davbr/archive/2006/02/27/540280.aspx
but there isn't really a whole lot out there to guide you through what you want to do. If your instrumented code absolutely must escape mscorlib.dll somehow, you could try writing IL that does a P/Invoke outside of the instrumented mscorlib function into native code you write. That native code would then call back into managed code (presumably a non-mscorlib assembly) via reverse p/invoke.
That's a bit intrusive to the app if you're doing this to every mscorlib function. But if you're hand-picking a few that aren't called every microsecond, this is worth exploring. Any mscorlib instrumentation is pretty dangerous stuff, though, and if you try this with mscorlib functions that are called very early on startup, all sorts of bad things can happen.
The crux is that the CLR and mscorlib have insider knowledge about each other. The loader saw pretty substantial revs from 1.x to 2.0, and it has all sorts of assumptions about what mscorlib looks like (mscorlib is loaded first, it references no one, etc.). I understand how frustrating this is, particularly since things worked fine for you in 1.1, and they appear to work fine in 2.0 as long as mscorlib is not loaded domain-neutral (though I'd be wary of any conclusions you draw from observed behavior, as there might be circumstances other than domain-neutral that could spell trouble when you instrument mscorlib).
One of the perils of writing profilers is that, since they're so low to the ground, major revs of the CLR often require changes to the profilers as well just to keep up. Still, you are not the only one who really needs to instrument mscorlib and is running into problems doing so. So this is something we need to do some more thinking about.
Regarding lack of documentation--yes, I agree, more would be better! We are at least taking steps in the right direction. Now we at least have the API documented on MSDN, as opposed to 1.x which had a Word doc and comments in the corprof.idl file. :-)
All Replies
Hi! Let me ask a couple questions so I understand better what you're seeing. What exactly is going wrong with SetILFunctionBody when mscorlib is loaded domain neutral? Are there problems with the SetILFunctionBody call itself, or do you see problems shortly afterward when the function you instrumented is JITted (or executed)? Also, do you modify the metadata of mscorlib at runtime to add any references from mscorlib to any other assemblies? If so, you will definitely see problems as the loader assumes mscorlib references nothing else. Finally, have you looked into the Enter2 function hook? If all you want is a notification of a call and its arguments, that might be all you need (rather than having to do any IL rewriting at all).
- David,
What is wrong is that the newly Jitted code crashes as soon as it is executed by throwing a Exception...well, since it is the Exception class that has the problem - it goes into an infinite JIT loop and crashes with stack overflow...we crash during execution as soon as it does the new xxx() on the proxy, not modification.
We have put NO reference in the mscorlib. However, we do put in all the tokens into the method. We are hooking the .ctor methods. Our proxy is in the GAC, we use the opcode for new to load the class (and assume it'll find in the GAC). This code works well in 1.1. Also it works fine (with no AppDomains in 2.x). If more than 1 AppDomain and set SingleDomain - it crashes like this. If the Exception is first thrown in the default domain - then it's fine in SingleDomain - but then the first call to CreateDomain generates an EngineExecutionException.
As for the Enter/Leave - we tried that, we had a Com Proxy and it worked fine for calling. However, we had hell trying to get the parameters using the FunctionID to work into PCCOR_SIGNATURE and then parsing the signature and formulating the marshal from a native type to managedtype. Not to mention, everybody told us - do not call managed code from the unmanaged code in Function Enter/Leave...well, as long as you realize that you could have a recursive callback problem and manage it - we didn't have any problems. But we abandoned it because we got tired of playing around with VARIANTS, SAFEARRAYS, and other very unfriendly code types.
Any way we could talk on the phone or does that violate rules of this blog... Phone won't work well for me, but I think we can make good progress on this forum. As I understand it (and please correct me if I'm wrong), you prefer all your inspection code (which sits in what you're calling your "proxy") to be managed. And yes, I agree, if you go the FunctionEnter2 route, you must not be running managed code from there!
Your managed proxy resides in its own assembly (that you created) in the GAC, and you rewrite IL of ctors from mscorlib so that they call your GAC'd proxy's ctor so that your proxy's ctor may do something with the hooked mscorlib object's ctor parameters. The part I'm still trying to understand is how you make the rewritten IL of the mscorlib object ctor call your proxy ctor. It sounds like you're using the "newobj" IL instruction, which means you need to include in that instruction a metadata token representing the target of the call (which will be your proxy's ctor). This token would probably be a MemberRef, which must have been composed in part by using an AssemblyRef that references your proxy's assembly. That right there constitutes a reference from the assembly you're instrumenting (i.e., mscorlib) to your proxy. (I was a little vague when I used the term "reference" in my earlier post, so I apologize if I was unclear.)
If that's indeed what's going on, you may simply be lucky that you're only hitting problems when mscorlib is loaded domain neutral. In general, such a scenario can't be assumed to work--everyone's allowed to reference mscorlib, but mscorlib isn't allowed to reference anyone. The standard fix we recommend for this is not to create your own assembly to house the proxy. Instead, dynamically add your proxy to mscorlib directly using the metadata API at runtime. Literally add your proxy's class and methods inside mscorlib at runtime (say from ModuleLoadFinished() for mscorlib), so that any tokens you use to reference your proxy are Defs instead of Refs.
Then again, I could easily be misintepreting how your proxy fits into the equation. :-) If that's so, please let me know how your proxy actually gets called.
- David,
Everything you said is correct, we add the reference, we use the newobj, and we put calls on entry into the method and any exit...
Ok, assuming we use the ModuleLoadFinished (and do you have any links to any good examples here) and put our proxy in there...
how do we ever get out of mscorlib?
The Proxy eventually reads an XML file, which contains "handlers" that use the arguments and use LoadFrom to invoke special code which is designed to understand and do things with the method.
Right now, we hook all of the sqlclient classes...and calculate response time for executeQuery, etc. - we use ThreadLocal storage to track all the way back to the user identity and track the queries.
On ASP.NET we have hooked the HttpRuntime class (globals.asax) and we intercept all of the callbacks for Begin/End Session/Application/Request and errors.
Now, both of those work fine in 2.0 as they must not be in corelib...it's only the core classes...
So, if we stick our proxy in corelib - how do we ever get out of there...we cannot put our entire instrumentation in it...and we actually start TCP servers (so we can control the process) and all kinds of other stuff.
I wrote a transparent "proxy" that allows me to move between AppDomains seamlessly as we do not know what AppDomain we will be in, but we have to be able to get back to our SubSystem (which lives in the Default Domain so it won't be unloaded).
So, I guess you confirm what we are seeing - although we've never had a problem on 1.1...do you have a good example for loading into mscorlib and if we do how do we every get out of it (can we CreateInstanceAndUnwrap and marshal our way out?)
Thanks for you help...as you know, there's almost no documentation on this kind of stuff (get maybe 1 page on google and 1/2 of it is Japaneze - but we still try to read it :)) Hi,
Could you specify what kind of error do you get? There are a number of scenarios when the CLR-inside errors raises that you can't event catch but in your case it is a managed Exception (you said it leads to the overflow) so I think it can be just a bad IL code inserted. You can find the type of exception in Exception callback of the profiler.
Or I misunderstand smth?
- Thanks for your response...
We are hooking the .ctor in System.Exception...
Nothing wrong with the IL because it works fine in MultiDomain or MultiDomainHost mode, it works fine with SingleDomain as long as you don't specify a new Domain...
The hook "fixes" up tokens, and instantiates a .DLL in the GAC...that .DLL dispatches to "substitutes" that do the handling for Exceptions, ASP.NET events (hooking the HttpApplication class), SQL client calls, etc....
The crash occurs immediately on the new xxx() call (newobj) that is injected...it causes "rejits" of the Exception which in turn causes our code to be invoked...which in turn another rejit...eventually, stack overflow.
It is only for 2.0, apps work fine in 1.1. For some reason, something changed in 2.0. I am sure it has to do with the LoaderOptimization and domain neutrality. I believe that mscorlib in SingleDomain mode absolutely will not call outside of itself...
Any clues? >>
Ok, assuming we use the ModuleLoadFinished (and do you have any links to any good examples here) and put our proxy in there...
how do we ever get out of mscorlib?
<<There is an old IL rewriting sample on MSDN at
http://msdn.microsoft.com/msdnmag/issues/03/09/NETProfilingAPI/,
and one of my blog entries talks a bit about IL rewriting (with a note on the dangers of having mscorlib reference outside itself):http://blogs.msdn.com/davbr/archive/2006/02/27/540280.aspx
but there isn't really a whole lot out there to guide you through what you want to do. If your instrumented code absolutely must escape mscorlib.dll somehow, you could try writing IL that does a P/Invoke outside of the instrumented mscorlib function into native code you write. That native code would then call back into managed code (presumably a non-mscorlib assembly) via reverse p/invoke.
That's a bit intrusive to the app if you're doing this to every mscorlib function. But if you're hand-picking a few that aren't called every microsecond, this is worth exploring. Any mscorlib instrumentation is pretty dangerous stuff, though, and if you try this with mscorlib functions that are called very early on startup, all sorts of bad things can happen.
The crux is that the CLR and mscorlib have insider knowledge about each other. The loader saw pretty substantial revs from 1.x to 2.0, and it has all sorts of assumptions about what mscorlib looks like (mscorlib is loaded first, it references no one, etc.). I understand how frustrating this is, particularly since things worked fine for you in 1.1, and they appear to work fine in 2.0 as long as mscorlib is not loaded domain-neutral (though I'd be wary of any conclusions you draw from observed behavior, as there might be circumstances other than domain-neutral that could spell trouble when you instrument mscorlib).
One of the perils of writing profilers is that, since they're so low to the ground, major revs of the CLR often require changes to the profilers as well just to keep up. Still, you are not the only one who really needs to instrument mscorlib and is running into problems doing so. So this is something we need to do some more thinking about.
Regarding lack of documentation--yes, I agree, more would be better! We are at least taking steps in the right direction. Now we at least have the API documented on MSDN, as opposed to 1.x which had a Word doc and comments in the corprof.idl file. :-)
Hi,
I think it will be interesting. We have the same problem with the instrumenting on the early loading phase. I know the method to solve this problem, I've tested it but we didn't implemented it yet =(.
If you don't know there is the class System.AppDomainManager in the FW 2.0. You can set your own appdomain manager. After that your managed assembly - your AppDomainManager will be loaded before any call of the mscorlib methods.
You can read about AppDomainManager here: http://blogs.msdn.com/shawnfa/archive/2004/11/12/256550.aspx
AppDomainManager in its turn can load any assembly you want and you will not have any exceptions, occurs because the incorrect tokens resolution - your assembly is already loaded.
Please notify about your expirience.
2 David: Thanks a lot for your blog. I like it very much.
- Sergey,
I am familar with the AppDomainManager - I actually attempted to use it to change the LoaderOptimization on the app (since I do not have access to the app) - even though I interception the init and create of the Domain, it was too late to change the global domain...and although I changed the Setup, SetData("LOADER_OPTIMIZATION"), etc. - it didn't matter to the runtime.
So, you are saying - Load the assembly up front early on in AppDomain creation, and the Domain Manager is given some kind of "special pass", and that the corlib would somehow be "ok" with the assembly because it was already loaded? Hi,
Our problem was that we are instrumenting method during the AppDomain loading between AppDomainCreationStarted and AppDomainCreationFinished callbacks. During these period we've got correct tokens, but these tokens couldn't be resolved. I actually do not know the real reason of this, but this behavior can be fixed using AppDomainManager.
I set environment variables as it said in the documentation of AppDomainManager. My AppDomainManager in it's turn loads the assembly I want to call. So the tokens to that assembly resolves correctly.
- well, I'm going to try some things and get back to you and let you know how it goes...
- Nope, even though I combined everything into 1 dll which also contained the AppManager classes AND I successfully loaded the .DLL in the initialize every single time - mscorlib throws exceptions when executing newobj. I can set the app to MultiHost load...and it works fine...
If it was an instruction issue - then wouldn't it allways fail...not just for SingleDomain (mscorlib domain neutral).
Hell, I'd just merge everything into mscorlib, but I don't have the key and I'd have to merge every single runtime version...
Any suggestions? It's possible to merge everything into mscorlib.dll without having to resign the assembly (and thus without needing a key), so long as you're flexible on your definition of "merge". :-) You can do this dynamically at runtime, inside the ModuleLoadFinished callback from mscorlib.dll. At that point, you can use GetModuleMetadata() to get pointers to metadata interfaces on mscorlib.dll, including those that can WRITE to the metadata like IMetaDataEmit. This allows you to add whatever classes, properties, functions, etc. you like to mscorlib.dll.
Of course, none of this is persisted to disk, it must be done each time the app runs, you're forced to use the jitted (not ngen'd) version of mscorlib, and all your additions will be jitted as well. But at least you don't need a key! :-)
I'm not sure-- you might already have known about this. I wanted to mention it here explicitly just in case. In spite of the disadvantages of this approach, it is generally considered the safest to use.
- David, Thanks we are considering this...
We modified to inject into a different class this time so we did not get the infinite loop, we picked the IndexOutOfRangeException...it tells me:
Injected into System.IndexOutOfRangeException::.ctor 18a5f98
EXCEPTION: multidomain.exe ERROR: System.IO.FileLoadException: Could not load f
ile or assembly 'CLRGatewayProxyWrapper, Version=0.0.0.0, Culture=neutral, Publi
cKeyToken=229f8702f35954dd' or one of its dependencies. The parameter is incorre
ct. (Exception from HRESULT: 0x80070057 (E_INVALIDARG))
File name: 'CLRGatewayProxyWrapper, Version=0.0.0.0, Culture=neutral, PublicKeyT
oken=229f8702f35954dd'
at TestIt.Called.doit(Object TheObject)
No doubt it is in the GAC...and I even copied it into the app directory - works fine on anything BUT stuff out of corelib...
Would be nice if the guys at Microsoft would give a more descriptive message or at least say in plain english WHY an Assembly would not load. I would have to say Assembly loading is one of the top 5 error spots for any .NET application and there's not alot of information when it goes bad. Have you tried looking at the fusion log? Info about enabling and reading the log is here:
http://blogs.msdn.com/suzcook/archive/2003/05/29/57120.aspx
You can search through this blog http://blogs.msdn.com/junfeng (and others) for fuslogvw.exe for more info.
- David, that's a great idea - I'll let you know how it goes...
Ted - Well,
I did everything in the blog...no errors were reported and I even set force and saw successful binds...perhaps mscorlib does not write it's errors? No InnerExceptions to check either...
I ran ildasm and checked the manifest, everything checks out - it works fine on anything but mscorlib...
I wish it would just simply say "Can't load this file...checked here, here, here, etc...." like it does for ASP.NET.
Debugger is not gonna work since this is injected code, it will probably be confused...
Any other ideas? Try using the registry directly to confirm that logging is configured properly:
HKLM\Software\Microsoft\Fusion\ForceLog <DWORD> 1
HKLM\Software\Microsoft\Fusion\LogPath <REG_SZ> “Path_to_directory_that_must_exist”Note that if the download cache size is exceeded, no log will be generated.
Also, you said "it works fine on anything but mscorlib". Do you mean you only see this failure when the IL you rewrite (to call into your assembly) comes from mscorlib (while IL rewritten from other assemblies cause a successful load)? If so, I wonder if mscorlib and the AppDomain are fully initialized by the time the CLR tries to load your assembly. Does this mean you're still trying to add a reference from mscorlib to another assembly (CLRGatewayProxyWrapper)? If so, you're still venturing into dangerous territory...
- David, Thanks for your help...not sure if I thanked you yet...
The registry values I confirmed when I ran my tests...
Yes, the same IL works fine in all cases on 1.1, and fine for anything but a class loaded from mscorlib.
AppDomains are initialized, because I have my own AppDomainManager...the mscorlib hook is coming after the Initialization...
Yes, I am still adding reference from mscorlib, I've also got a version that does a full blown dynamic invoke (no references) - does not matter, something about mscorlib in V2 that it does not like to call outside.... - I've also got exception "File or assembly not found" for the "System.Web" when modifying the "early-called" methods (called during appdomain loading). Fusionlog also has shown nothing wrong. I do not know how to solve this problem.
- I am mostly a Java programmer, in 1.5 they added a package called java.lang.instrument, in that they allowed for "callbacks" to java code where you can supply the bytes back...nice thing about Java is that there are numerous packages that allow for bytecode modification.
Does not help me now, but sure would be nice if .NET built a managed code proxy system for wrapping methods. Yes, I know that you can do this via the message attribute - but you need source code for that.
I suppose that Microsoft really doesn't want people doing this kind of stuff...
It's all trial and error, mostly error...it's painful...
Any suggestions from anyone? Something I'd like to understand better. A few posts above, you said:
"Yes, I am still adding reference from mscorlib, I've also got a version that does a full blown dynamic invoke (no references) - does not matter, something about mscorlib in V2 that it does not like to call outside...."
This version that does a full blown dynamic invoke with no references-- does that mean that all the code you rewrite and add to mscorlib has no references outside of mscorlib (even to other MS assemblies like System.Web, etc.)? Specifically, the code you rewrite contains a newobj instruction to create an instance of a type defined in mscorlib, and that type does not reference (directly or indirectly) any other types outside of mscorlib (either your own types or other MS types from other assemblies)?
If that's accurate, it's interesting you're still hitting a file load exception in such a case, as that implies the CLR is still trying to load a file even though there are no references from mscorlib to the outside world. In other words, it sounds like the execution of the newobj instruction (or some other instruction you insert) is still trying to force a load of some other assembly (either yours or MS). Is this accurate, or have I gone astray somewhere? Can you provide more info on exactly what is causing the attempted load?
- Hi,
I work with Ted and wanted to clarrify the above statement. We are still creating an instance of an object that is defined in another dll.
What we did was we tried it a bunch of different ways. 1st, we put the dll in the GAC and did a newobj instruction. Then we tried a couple of different ways to create that object using dynamic invocation where we specifically point to the dll and also where we let the clr automatically find the dll. None of these methods worked.
--abel - Thanks for the clarification. Given that, I suspect you are hitting the issue where the AppDomain isn't initialized enough that the CLR will be ready to try loading another assembly just yet. If there's some way to stuff everything into mscorlib so you avoid another module load, I'd expect that to fare better.
- Hi David,
The AppDomain(s) that we the problem in has long since been
created and initialized...in fact, I attached a AppDomainManager to it, and clearly see the creation/inialization long before we throw the first exception which causes the issue...evidently, the Exception class is not Jitted early on - but only when it's invoked...
I'd like to try a test with ILMerge, but how can I resign it with the key it is looking for since I do not have the .snk file?
Thanks, Ted Hi,
Have you ever tries to look at exception with windbg? It's interesting to see where that exception have been raised inside the mscorwks? I have some dumps of our cases and think it may be helpfull.
If your domain is well past initialization, you could easily be running into other profiler-broken assumptions by the CLR that are exacerbated when in SingleDomain mode as you've been suspecting all along.
I think you're best off merging all of your stuff into mscorlib.dll dynamically at runtime, using the metadata interfaces from within your ModuleLoadFinished callback.
If you want to experiment with merging on disk, you certainly will have security issues (and for good reason). You might want to look into delay-signing and test-signing:
http://msdn.microsoft.com/msdnmag/issues/06/07/CLRInsideOut/default.aspx
http://blogs.msdn.com/shawnfa/archive/2006/06/23/644648.aspx
http://blogs.msdn.com/shawnfa/archive/2005/10/24/484170.aspxbut realize these are DEVELOPER-MACHINE-ONLY solutions. It is not recommended that you ship a profiler to external customers that creates skip-verification entries on the customers' machines. This leaves customers vulnerable to spoofing attacks now that their CLR has ceased to perform complete strong name validation of those assemblies.
David,
No doubt our customers would have a heart attack (of course most of them don't even know their apps have
signed keys...ha ha...
I would only do this to test and see if it would work...if it did, it would confirm some things...and then I'd consider
the moduleloadfinished...
Thanks for all your help...
Ted
David,
To what method of dynamically merging metadata at runtime are you referring to - IMetaDataEmit::Merge or manually loading the assembly and creating the types one by one ? If you're talking about Merge/MergeEnd can you point me to some resources on how to do it because I've tried it without any luck in the past.
Thanks,
Paul
Hi, Paul! I wasn't referring to Merge. Just the various methods on IMetaDataEmit(2) to manually create types and methods (e.g., the Define* methods). Note that the point of this is to avoid loading other assemblies from mscorlib functions or from too early in the initialization process (ModuleLoadFinished of mscorlib counts as "too early"). So I don't recommend you load one assembly, read its types, and then pump them into mscorlib. I recommend you just have all the types and methods available to your profiler in whatever format you like such that you can read them somehow without requiring the runtime to load anything--simplest (and least maintainable :-)) would be to hard code everything in arrays of your own format in your profiler Then pump them into mscorlib one by one via IMetaDataEmit(2)::Define* and whatever other IMetaDataEmit(2) methods you need.
BTW, if you want to use Merge for some other purpose, the only docs I know of are the MSDN reference docs (e.g., http://msdn2.microsoft.com/en-us/library/ms231399.aspx and http://msdn2.microsoft.com/en-us/library/ms231399.aspx).
