locked
.NET 2.0 : profiler adding static fields causes an assert in GetOffsetForStaticData

    Question

  • I'm implementing a profiler which dynamically adds types to assemblies. Althought the injection code works in .NET 1.1 in .NET 2.0 for any type that has static fields I get an assert in Module::GetOffsetForStaticData (partial call stack) :

     

    ntdll.dll!_DbgBreakPoint@0()  
      mscorwks.dll!EEPolicy::LogFatalError()  + 0x140 bytes 
      mscorwks.dll!EEPolicy::HandleFatalError()  + 0x36 bytes 
      mscorwks.dll!CLRVectoredExceptionHandlerPhase3()  + 0xc6f1c bytes 
      mscorwks.dll!CLRVectoredExceptionHandlerPhase2()  + 0x20 bytes 
      mscorwks.dll!CLRVectoredExceptionHandler()  + 0x9c bytes 
      mscorwks.dll!_CLRVectoredExceptionHandlerShimX86()  + 0x26 bytes 
      ntdll.dll!ExecuteHandler2@20()  + 0x26 bytes 
      ntdll.dll!ExecuteHandler@20()  + 0x24 bytes 
      ntdll.dll!_KiUserExceptionDispatcher@8()  + 0xe bytes 
      mscorwks.dll!Module::GetOffsetsForStaticData()  + 0x55622 bytes 
      mscorwks.dll!MethodTableBuilder:: PlaceStaticFields()  + 0xaa bytes 
      mscorwks.dll!MethodTableBuilder::BuildMethodTableThrowing()  + 0x4d0 bytes 
      mscorwks.dll!ClassLoader::CreateTypeHandleForTypeDefThrowing()  + 0x3a4 bytes 
      mscorwks.dll!ClassLoader::CreateTypeHandleForTypeKey()  + 0x47 bytes 
      mscorwks.dll!ClassLoader:: DoIncrementalLoad()  + 0x5c bytes 
      mscorwks.dll!ClassLoader::LoadTypeHandleForTypeKey_Inner()  + 0x103 bytes 
      mscorwks.dll!ClassLoader::LoadTypeHandleForTypeKey_Body()  + 0xdb bytes 
      mscorwks.dll!ClassLoader::LoadTypeHandleForTypeKey()  + 0xa4 bytes 
      mscorwks.dll!ClassLoader::LoadTypeDefThrowing()  + 0x5a0c5 bytes 
      mscorwks.dll!ClassLoader::LoadTypeDefOrRefThrowing()  + 0x65e6b bytes 
      mscorwks.dll!ClassLoader::LoadTypeDefOrRefNoThrow()  + 0x86 bytes 
      mscorwks.dll!COMModule::GetHINST()  + 0x2e2 bytes 
      mscorwks.dll!COMModule::GetClasses()  + 0xca bytes 

     

    I'm adding the types in ModuleAttachedToAssembly using IMetaDataEmit:: DefineType/DefineField. Any ideas how I can fix this?

     

    Thanks

    Paul

    Tuesday, May 01, 2007 2:34 AM

Answers

  • I've talked with a couple people about this issue.  First off, it does appear you've run into an unfortunate limitation of the system (adding a type with statics via profiler il rewriting).  This is something we will look at for future versions of the CLR.

     

    For now, here are a couple things you can try:

     

    1) If the only reason you need a static is to implement the locking (so you enforce that only a single instance is created), you might try locking on the type instead of the static.  I know you indicated you don't want to do this, but it's worth a try... unless there's a specific reason why this wouldn't work in your situation?  As you discovered, you can't just index into the collection with your type name, but you can manually iterate through the types until you find the one with your name.

     

    2) If you absolutely need to put a static on your type, one thing you can try (no guarantees!) is to make your new type a generic type, that takes at least one generic parameter (e.g., MySingleton<T>).  Later on, you would use an instantiated version of your type (e.g., MySingleton<Object>).  The type system purposely delays laying out generic types until they're instantiated, and this impacts the timing of when we cache the statics for that type.  It doesn't matter so much what the type parameter on your new generic type is used for or what it's instantiated with, so long as there is a type parameter.  So you may have better luck creating a generic type that contains a static than a non-generic type that contains a static.  Again, this is obviously a gross hack, and relies on unguaranteed implementation details inside the CLR.

     

    Hopefully this will get you moving.  Please let us know how it goes!

    Wednesday, May 02, 2007 7:45 PM
    Owner

All replies

  • Hi, Paul.  Just to clariy... are you adding brand new classes, or are you modifying types that already exist in the module?  If the latter, then you're still venturing into dangerous territory as discussed here:

     

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1230140&SiteID=1

     

    Note that there have been significant changes to the loader between 1.1 & 2.0, so it's not surprising that type modification code could work in 1.1 but fail in 2.0.

    Tuesday, May 01, 2007 3:59 PM
    Owner
  • This is a brand new class added to mscorlib. I don't know how similar is Rotor code to the comercial version but I've looked at SSCLI 2.0 source and the module's static offsets array is initialized before the profiler "module load" and "module attached" callback methods are called; this array doesn't seem to be updated afterwards.

    I'm also running into another problem when trying to use the injected types : I can see the types via Assembly.GetTypes() but Type.GetType() fails. The following test code will  output "error" for all injected types (although they're visible via Assembly.GetTypes()) :

     

    using System;
    using System.Reflection;

     

    namespace Test {
        public class Test  {
            public static void Main(string[] args)  {
                Assembly corlib = typeof(string).Assembly;

                foreach (Type t in corlib.GetTypes()) {
                    if (t.FullName.StartsWith("MyNamespace.")) {
                        Console.WriteLine("Looking up type {0} ... ",
                            t.FullName, Type.GetType(t.FullName) == null ? "ERROR" : "OK");
                    }
                }
            }
        }
    }

     

    Example output : Looking up type MyNamespace.MyClass ... ERROR

    (MyNamespace.MyClass doesn't have any static fields and is injected into mscorlib)

     

    Debugging SSCLI 2.0 shows the types are added but RuntimeTypeHandle::GetTypeByName (runtimehandles.cpp) is not finding them.

     

    Thanks

    Paul

    Tuesday, May 01, 2007 5:38 PM
  • What kind of static are you adding?  I'm wondering if you'll see different results depending on whether they're RVA-statics, context-statics, or the regular old AppDomain statics.

     

    The GetType issue is a known one.  We've also seen cases where finding a newly-added type by name via Type.GetType(t.FullName) won't work, though you can still iterate through the array yourself (foreach (Type t in corlib.GetTypes())) and find your types that way.  Thanks for corroborating that one with us, as it always helps to know when other customers are hitting issues.

    Tuesday, May 01, 2007 7:48 PM
    Owner
  • It's an AppDomain static, the class is a singleton implementation:

     

    public class MySingleton {

       private static MySingleton   _instance;

       private static Object             _instanceLock = new object();

     

       /* Some instance state here */

      

       public static MySingleton Instance {

           get {

               lock (_instanceLock) {

                  if (_instance ==  null) {

                     _instance = new MySingleton();            

                  }

                  return _instance;

               }

           } 

       }

     

        /* Some instance methods here */

    }

    I'm using this singleton as storage backend for object specific data - please see the thread link you've posted in your first reply for what I'm trying to do. In the above code I can get rid of the _instance field and use AppDomain.Get(Set)Data to store the instance but the problem is the instance lock,  without static fields the only lock I can think of is on the type ( "lock(typeof(MySingleton))")  and I really don't want to do that.

     

    Thanks

    Paul

    Tuesday, May 01, 2007 9:47 PM
  • I've talked with a couple people about this issue.  First off, it does appear you've run into an unfortunate limitation of the system (adding a type with statics via profiler il rewriting).  This is something we will look at for future versions of the CLR.

     

    For now, here are a couple things you can try:

     

    1) If the only reason you need a static is to implement the locking (so you enforce that only a single instance is created), you might try locking on the type instead of the static.  I know you indicated you don't want to do this, but it's worth a try... unless there's a specific reason why this wouldn't work in your situation?  As you discovered, you can't just index into the collection with your type name, but you can manually iterate through the types until you find the one with your name.

     

    2) If you absolutely need to put a static on your type, one thing you can try (no guarantees!) is to make your new type a generic type, that takes at least one generic parameter (e.g., MySingleton<T>).  Later on, you would use an instantiated version of your type (e.g., MySingleton<Object>).  The type system purposely delays laying out generic types until they're instantiated, and this impacts the timing of when we cache the statics for that type.  It doesn't matter so much what the type parameter on your new generic type is used for or what it's instantiated with, so long as there is a type parameter.  So you may have better luck creating a generic type that contains a static than a non-generic type that contains a static.  Again, this is obviously a gross hack, and relies on unguaranteed implementation details inside the CLR.

     

    Hopefully this will get you moving.  Please let us know how it goes!

    Wednesday, May 02, 2007 7:45 PM
    Owner
  • Thanks for clarifying the situation David.  I've changed the implementation in the mean time to lock on type and use AppDomain Set(Get)Data and it works just fine. I did not want to lock on type because of performance reasons and because of all the bad things that can happen if you lock on a public object (like the type object is).

     

    Paul

    Wednesday, May 02, 2007 8:05 PM
  • David, do you know *ANY* way to make the Type.GetType and Assembly.GetType work for dynamically added types? This bug proves to be the biggest pain the neck; short of accessing the dynamic types via reflection they are totally unusable.

     

    This is what I'm doing:  

     

    I have one assembly (A) that defines a set of common types, types which are used to augment mscorlib defined types with some specific data. Because these types need to be referenced from mscorlib the whole assembly gets injected into mscorlib. The same assembly (A) is also referenced by other assemblies from the product so ideally I would like to redirect the assembly A reference to mscorlib because that's where the types are at runtime. I've tried redirecting the references using IMetaDataAssemblyEmit:Tongue TiedetAssemblyRefProps but that doesn't work because of the type lookup issue, so what I'm forced to do now is let the runtime load the assembly A and make the types defined in A forward the calls to the actual "runtime" types from mscorlib using reflection.

    Although ugly the above solution works but greatly limits what you can do with the injected types - since the "runtime" vs. the assembly A types are different you cannot cast, inherit from them, etc.

     

    Do you have any suggestions how can I actually redirect the assembly A types to the mscorlib dynamically added types?

     

    Thanks

    Paul

     

    Friday, June 01, 2007 9:23 PM
  • Hi, Paul, I need to ask a couple questions to make sure I'm understanding.  You ask how to make Type.GetType and Assembly.GetType work for dynamically added types, but above I thought you said that Assembly.GetTypes (plural) does work for you.  Is that workaround insufficient?

    Could you clarify for me why the fact that Type.GetType and Assembly.GetType fail for you causes SetAssemblyRefProps to fail as well?  Is it that SetAssemblyRefProps explicitly fails because you believe it is unable to find the type you pass to it, or is it that you're unable to call SetAssemblyRefProps in the first place because you're unable to determine the token to pass to SetAssemblyRefProps?

    Presumably the reason that other assemblies from your product are referencing assembly A instead of mscorlib is because those assemblies are compiled to disk before your profiler runs, correct?  In other words, those other assemblies are not generated dynamically at runtime by your profiler (via metadata interfaces), right?  If they were, I'd expect you could make those types directly reference your additions to mscorlib instead of assembly A in the first place.  How hard would it be to just dynamically generate any code that needs to reference A and have it reference mscorlib instead, rather than compiling it down to an on-disk assembly?

    Your workaround of having the real implementation in mscorlib and having A just forward calls into mscorlib sounds reasonable to me.  Although casting between A-types and mscorlib-types wouldn't work without explicit derivation chains of references, you should still be able to cast among A-types and cast among mscorlib-types separately (where the types are in the same derivation chain).  I take it that's insufficient?  Maybe if you gave an example of where this fails you?

    Monday, June 04, 2007 8:21 PM
    Owner
  • Hi David,

     

    What I have right now works but because I have to use reflection the performance suffers and the implementation is cumbersome. My main problem is with the communication between the mscorlib added types and the rest of assemblies, basically the only way to pass my data between mscorlib and the rest of the world is by using bare System.Objects and reflection to access the object's fields. I was looking for a better solution...

     

    My answers:

    - I apologize, I was not clear enough: SetAssemblyRefProps succeeds but it does not help, I get the same "failed to find type" exceptions.

    - Yes, the assemblies are compiled to disk.

    - I'm not sure how dynamically generating assemblies is going to help because I will have to add the same type refs to mscorlib added types, type refs which are failing now.

    - I have no problems using the types either only inside mscorlib or only outside of it; the problem comes when I want to communicate between mscorlib and the rest of assemblies.

     

    Thanks

    Paul

     

     

    Tuesday, June 05, 2007 5:19 PM
  • Hi, Paul, sorry for the long silence.  Just got back from vacation today.  Thanks for the clarifications.  Sounds like the following assemblies are in play:

     

    (1) mscorlib

    (2) Assembly A: Shipped with your profiler, containing core types used by your profiler

    (3) Other on-disk assemblies shipped with your profiler (that reference A)

    (4) other assemblies used by the app being profiled--I think these are irrelevant for this discussion.

     

    Since you want (1) to reference types in (2), you're moving the types from (2) to (1) (so (1) just references itself).  This means references from (3) to (2) must be modified to become references from (3) to (1).

     

    The reason I recommended building the contents of (3) dynamically from scratch using metadata interfaces at runtime is because that would eliminate the variable of the CLR first having to load (3) with the "wrong" references (i.e., referencing (2)) first, only to have you patch those up later (to point to (1)).  This scheme involves the CLR building its own internal metadata structures with "wrong" references, with the hope that they can be patched up shortly after (without the wrong references having been cached in other places that might not know to be updated when you patch them).

     

    As for the particular exception you mentioned, sounds like you're using SetAssemblyRefProps to transform (3)->(2) references to become (3)->(1) references, and it claims to succeed, but you still get a "failed to find type" exception later on.  What's the call stack at the time when you see the "failed to find type" exception?  I'm assuming this occurs after you've done your IMetaDatEmit operations and at some point while the managed code is running (e.g., while a type from (3) that references one of your new types in (1) is first used).  It sounds like you're attributing the cause of the exception to the same issue as Type.GetType and Assembly.GetType failing on your newly added types, but I'm not certain those are necessarily related.

     

    I spoke with Karel on the metadata team, who believes that, in theory, using SetAssemblyRefProps could actually work.  However, there are a few things going on in your scenario that could gum up the works:

     

    First, you're using SetAssemblyRefProps to redirect references to mscorlib--the nastiest of the nasty.  Because there are special assumptions in the CLR about mscorlib, SetAssemblyRefProps may simply be failing to do its job just because you're redirecting to mscorlib.  You can test this theory by making yet another assembly (call this (5)) containing the same types you added to mscorlib, and just testing to see whether you can redirect assembly references (3)->(2) to go (3)->(5) and see if that works.  If it does, that's evidence that SetAssemblyRefProps will simply not at all work when you redirect to mscorlib.  If it fails, that means something else is going on, and you have a more isolated scenario that could, hopefully, make debugging the problem easier.

     

    Second, (3) is ALREADY referencing (1), because all assemblies reference mscorlib anyway.  So your scenario of using SetAssemblyRefProps to change (3)->(2) to (3)->(1) is effectively forcing a second assembly reference of (3)->(1).  You could add this as part of your test above by having an existing (3)->(5) reference on disk for (3), and then your call to SetAssemblyRefProps would thus create a second (3)->(5) reference.  Again, this removes mscorlib from the picture while testing whether having two assembly refs that target the same assembly is a problem.

     

    Finally, if all experiments go bad, the issue could simply be related to assembly reference caching done in the bowels of the fusion loader.  If fusion caches all cross-assembly references, perhaps your addition of another cross-assembly reference isn't finding its way into that cache.  Neither Karel nor I are experts in fusion, so this is just speculation.

     

    So if everything fails, it may be that your only alternatives are to stick with the solution you already have, or create (3) from scratch dynamically at runtime via IMetaDataEmit (rather than loading a disk version of (3) and patching it via IMetaDataEmit).

    Tuesday, June 12, 2007 10:56 PM
    Owner
  • David,

     

    To rule out all these scenarios I've tried the following test using IL roundtripping:

    - disassembled using ildasm one of my assemblies that references assembly "A"

    - removed the reference to assembly "A" and redirected all imports to mscorlib (search & replace "[MyAssembly]" with "[mscorlib]")

    - compiled the IL code using ilasm; running ildasm on the output image shows that everything is referenced from mscorlib

    - loaded the assembly in a simple test app and sure enough I've got the same "TypeLoadException : could not load type" exception

     

    I think you will agree with me this test pretty much avoids all the possible problems you've enumerated : there is no redirection at runtime to mscorlib, there is only one reference to mscorlib and the fusion caching is out from the picture because assembly A is not referenced anymore. It looks like I'm stuck with what I have now.

     

    Thanks for all your help !

    Paul

     

    Thursday, June 14, 2007 2:48 AM
  • Paul, I like your approach to narrowing this down, and the result is definitely interesting!  I agree that you've eliminated the most problematic variables, and yet still you are reproducing the problem!  At this point I'd be very interested in stepping through some of this code to try to diagnose further on my end.  Might you have a fairly trimmed-down test case that exercises this, including the profiler code that adds a type into mscorlib, and the assembly you ilasm'd which references this new type in mscorlib?

     

    If you could post this somewhere I can access, I'd like to step through it.  If you're willing to post this but would rather not broadcast its location on the public forum, you can reach me directly by my blog's contact page here:

     

    http://blogs.msdn.com/davbr/contact.aspx

    Tuesday, June 26, 2007 10:20 PM
    Owner
  • David, sorry for the belated reply but I've been away for a while. I'll send you a a slimmed down example of what I'm doing to help you reproduce the issue.

    Monday, July 09, 2007 4:40 PM