none
ECMA-335 forbids null as a managed pointer, but ECMA-372 allows nullptr as an interior pointer RRS feed

  • General discussion

  • Related to the how can a function return an lvalue thread, there seems to be an inconsistency between ECMA-335 and ECMA-372:

    • In ECMA-335 (Common Language Infrastructure (CLI)), 6th Edition, §II.14.4.2 (Managed Pointers) says: "Managed pointers cannot be null, and they shall be reported to the garbage collector even if they do not point to managed memory."
    • In ECMA-335, §III.1.1 (Data types) says: "Pointer types (native unsigned int and &) […] Note that object references and pointer types can be assigned the value null."
    • In ECMA-335, §III.1.1.5.2 (Managed pointers (type &)) says: "Managed pointers cannot be null."
    • In ECMA-372 (C++/CLI Language Specification), 1st Edition, §12.3.6.1 (Definitions) says: "The default initial value for an interior pointer shall be nullptr."
    • In ECMA-372, §34.2.2 (Interior pointers) says: "An interior pointer to a type shall be emitted into metadata as a managed pointer to that type with the modopt IsExplicitlyDereferenced (§33.1.5.4)."

    The value of an interior pointer defaults to nullptr, but it is emitted as a managed pointer, which cannot be null – or can it?

    Does this just mean that an implementation of ECMA-335 does not have to support null managed pointers, but if it implements ECMA-372 as well, then it has to support them?

    Is there a list of errata for these standards?


    • Edited by ranta Monday, December 14, 2015 7:03 PM mention ECMA-335 §III.1.1 and §III.1.1.5.2
    Monday, December 14, 2015 6:47 PM

All replies

  • The cannot be null thing is likely an oversight. You can end up with null managed pointers even in C#:

    fixed (int *p = new int[0])
        Console.WriteLine(p == null);
    

    Monday, December 14, 2015 8:06 PM
    Moderator
  • mono-dmcs 2.10.8.1 actually compiled your "fixed (int *p = new int[0])" as a pinned unmanaged pointer, in violation of ECMA-335 6th ed. §II.23.2.9. This looks like Bug 319052 and may have been fixed on 24 September 2015.

    I got the following from Microsoft (R) Visual C# Compiler version 4.0.30319.34209:

    .method private hidebysig static void  Main() cil managed
    {
      .entrypoint
      // Code size       43 (0x2b)
      .maxstack  2
      .locals init ([0] int32& pinned p,
               [1] int32[] CS$0$0000)
      IL_0000:  ldc.i4.0
      IL_0001:  newarr     [mscorlib]System.Int32
      IL_0006:  dup
      IL_0007:  stloc.1
      IL_0008:  brfalse.s  IL_000f
      IL_000a:  ldloc.1
      IL_000b:  ldlen
      IL_000c:  conv.i4
      IL_000d:  brtrue.s   IL_0014
      IL_000f:  ldc.i4.0
      IL_0010:  conv.u
      IL_0011:  stloc.0
      IL_0012:  br.s       IL_001c
      IL_0014:  ldloc.1
      IL_0015:  ldc.i4.0
      IL_0016:  ldelema    [mscorlib]System.Int32
      IL_001b:  stloc.0
      IL_001c:  ldloc.0
      IL_001d:  conv.i
      IL_001e:  ldc.i4.0
      IL_001f:  conv.u
      IL_0020:  ceq
      IL_0022:  call       void [mscorlib]System.Console::WriteLine(bool)
      IL_0027:  ldc.i4.0
      IL_0028:  conv.u
      IL_0029:  stloc.0
      IL_002a:  ret
    } // end of method Program::Main

    so this one indeed declares a pinned managed pointer and IL_0011 writes a zero as unsigned native int to it; not verifiable but it was in unsafe context.

    Anyway, it seems to me that a C# compiler could work around the null restriction by declaring an additional local unmanaged pointer, then initializing it with either zero or the value of the pinned managed pointer. The code would then be portable to hypothetical stricter implementations of ECMA-335. I don't see any commercial reason for such a change though.

    Tuesday, December 15, 2015 9:58 AM
  • "I don't see any commercial reason for such a change though."

    More importantly, I don't see why a null managed pointer would pose any problems to a runtime. It can deal with null references, it can deal with managed pointers pointing outside the managed heap but it can't deal with managed null pointers. Weird I'd say.

    I'd guess that managed pointers were originally modeled after C++ references and that's how they got the & symbol and the null restriction.

    Tuesday, December 15, 2015 6:24 PM
    Moderator
  • For the garbage collector, a null managed pointer seems equivalent to a managed pointer pointing to unmanaged memory, or to an invalid pointer (e.g. a noncanonical pointer on x64). However, I imagine supporting null managed pointers requires some extra effort in other parts of the runtime:

    • If a method gets a byref parameter and reads or writes the referenced location, and the hardware does not raise exceptions for null pointer references, then the runtime may have to generate code to explicitly check for null pointers. It could omit these checks if it does not support null managed pointers.
    • If a call across a remoting boundary uses a copy-in/copy-out mechanism (ECMA-335 §I.8.2.1.1), a programmer might expect it to pass null managed pointers through unchanged but throw exceptions for invalid pointers. ECMA-335 doesn't seem to require this though.

    Perhaps then, the intention was that null managed pointers would be supported in unverifiable code only.

    • Can verifiable code call a method and pass a null managed pointer as a byref parameter?

      If you first load a null reference with "ldnull" and then call a method that takes an "int32&" parameter, peverify 4.0.30319.1 rejects the conversion. Likewise if you load a zero native unsigned int with "ldc.i4.0; conv.u".

      However, if you declare a local with ".locals init ([0] int32& 'null')" and load that with "ldloc.0", then peverify is happy, and I haven't found anything in ECMA-335 that forbids this.

    • Can verifiable code take a byref parameter and check whether it is a null managed pointer?

      "ldarg.0; ldnull; ceq" is not verifiable according to peverify.

      "ldarg.0; ldc.i4.0; conv.u; ceq" is not verifiable according to peverify.

      "ldarg.0; brtrue.s 'NotNull'" is verifiable according to peverify, but not according to ECMA-335 §III.3.18.

      ".locals init (int32& 'null'); ldarg.0; ldloc.0; ceq" is verifiable according to peverify, and I haven't found anything in ECMA-335 that forbids this.

    It looks like verifiable code cannot be prevented from constructing and using null managed pointers. It would be interesting to know whether .NET Compact Framework and .NET Micro Framework support them, and what happens in remoting.

    Saturday, December 19, 2015 12:00 PM
  • "If a method gets a byref parameter and reads or writes the referenced location, and the hardware does not raise exceptions ..."

    Yes but such checks are already required for references so requiring them for managed pointers as well shouldn't be much of a problem. And even on hardware that raises exceptions the current implementation emits some explicit null checks for references.

    But this is an interesting observation anyway. The following code bombs as expected but the exception stack trace indicates that it does at the wrong place (I think, I'd need to double check the C# spec for this, assuming that it says anything at all about this case):

    unsafe class Program {
        struct foo {
            int x;
            [MethodImpl(MethodImplOptions.NoInlining)]
            public int kboom() {
                // the null ref exception is thrown from here
                // and not from the kboom's callsite
                return x + 42;
            }
        }
        static int Main() {
            fixed (foo* p = new foo[0])
               return p->kboom();
        }
    }

    "If a call across a remoting boundary uses a copy-in/copy-out mechanism (ECMA-335 §I.8.2.1.1), ..."

    I'm not familiar enough with remoting to comment on this. But IMO it would be peculiar to restrict managed pointers to non-null values due to a rather exotic (and more or less obsolete these days) feature.

    "If you first load a null reference with "ldnull" and then call a method that takes an "int32&" ..."

    ldnull produces a reference, it seems normal to me that you get an error if you try to pass that reference as a managed pointer.

    "Likewise if you load a zero native unsigned int with "ldc.i4.0; conv.u". ..."

    That's one of the reasons why I suspect that the whole "managed pointers cannot be null" is an oversight. .locals init creates null managed pointers no matter what.

    ""ldarg.0; ldnull; ceq" is not verifiable according to peverify."

    "ldarg.0; ldc.i4.0; conv.u; ceq" is not verifiable according to peverify.

    This is a duplicate of the above, it's not the comparison itself that is the issue but the attempt to produce a null managed pointer.

    ""ldarg.0; brtrue.s 'NotNull'" is verifiable according to peverify, but not according to ECMA-335 §III.3.18."

    It's not clear what makes you say that. Is it the fact that the spec doesn't mention the possibility of value being a managed pointer? Then check brfalse :)

    "".locals init (int32& 'null'); ldarg.0; ldloc.0; ceq" is verifiable according to peverify, and I haven't found anything in ECMA-335 that forbids this."

    Yep, the previously noted problem, .locals init creates null managed pointers.

    "It looks like verifiable code cannot be prevented from constructing and using null managed pointers"

    Yep, that's my impression as well. I'm more convinced than before that "managed pointers cannot be null" is a flaw in the spec. It might make sense to ban null managed pointers in verifiable code but doing so would likely severely restrict the use of managed pointers. Either locals of managed pointer type would have to be unverifiable or definite assignment analysis would have to be used to ensure that such locals have been assigned non null values. The later option would make a method like "void foo(ref int x) { x = 42; }" unverifiable because there would be no way to check that x isn't null, that would depend on foo's callers being verifiable.


    Saturday, December 19, 2015 2:29 PM
    Moderator
  • "I think, I'd need to double check the C# spec for this, assuming that it says anything at all about this case"

    The C# spec says in 18.5.2:

    "A pointer member access of the form P->I is evaluated exactly as (*P).I. For a description of the pointer indirection operator (*), see §‎18.5.1"

    And in 18.5.1:

    "The effect of applying the unary * operator to a null pointer is implementation-defined. In particular, there is no guarantee that this operation throws a System.NullReferenceException."

    So the example I posted above works correctly, there is no requirement in C# for an exception to be thrown at the call site.

    Saturday, December 19, 2015 2:43 PM
    Moderator
  • Yes but such checks are already required for references so requiring them for managed pointers as well shouldn't be much of a problem.

    Emitting the checks doesn't seem hard to implement, but they cost some time and memory at run time. "no.nullcheck" can be used in unverifiable code though.

    The following code bombs as expected but the exception stack trace indicates that it does at the wrong place (I think, I'd need to double check the C# spec for this, assuming that it says anything at all about this case):

    ECMA-335 §II.13.3 says "The callvirt instruction shall not be used with unboxed value types", so the C# compiler has to use the call instruction; and §I.8.4.2 says the this pointer can then be null. If the C# language required a null check there, I suppose the C# compiler could use ldobj and pop.

    I'm not familiar enough with remoting to comment on this. But IMO it would be peculiar to restrict managed pointers to non-null values due to a rather exotic (and more or less obsolete these days) feature.

    I tried the following verifiable C++/CLI code in .NET Framework 4.5.2:

    public ref class Handler : MarshalByRefObject
    {
    public:
       void Swap(int% a, int% b)
       {
          int tmp = a;
          a = b;
          b = tmp;
       }
    };
    
    int main()
    {
       AppDomain^ appDomain = AppDomain::CreateDomain("child");
       Object^ obj = appDomain->CreateInstanceAndUnwrap(
          Handler::typeid->Assembly->FullName,
          Handler::typeid->FullName);
       Handler^ handler = cli::safe_cast<Handler^>(obj);
       int a = 42;
       handler->Swap(a, *cli::interior_ptr<int>());
       Console::WriteLine(a);
       return 0;
    }
    

    and got a fatal null pointer dereference with this native call stack:

    00 clr!CopyValueClassUnchecked+0xc2
    01 clr!MethodTable::FastBox+0x3e
    02 clr!MethodTable::Box+0x42
    03 clr!CrossDomainChannel::MarshalAndCall+0x2f780d
    04 clr!CrossDomainChannel::ExecuteCrossDomainCall+0x54
    05 clr!CrossDomainChannel::CheckCrossDomainCall+0xa4
    06 clr!CTPMethodTable::OnCall+0x69
    07 clr!TransparentProxyStub_CrossContextPatchLabel+0xa
    08 0x000007fe`997001cb
    09 clr!CallDescrWorkerInternal+0x83
    0a clr!CallDescrWorkerWithHandler+0x4a
    0b clr!MethodDescCallSite::CallTargetWorker+0x251
    0c clr!RunMain+0x1ee
    0d clr!Assembly::ExecuteMainMethod+0xb6
    0e clr!SystemDomain::ExecuteMainMethod+0x506
    0f clr!ExecuteEXE+0x3f
    10 clr!_CorExeMainInternal+0xae
    11 clr!CorExeMain+0x14
    12 mscoreei!CorExeMain+0xe0
    13 MSCOREE!CorExeMain_Exported+0x57
    14 KERNEL32!BaseThreadInitThunk+0xd
    15 ntdll!RtlUserThreadStart+0x1d

    It appears that cross-appdomain calls within a process do not tolerate null managed pointers any better than invalid pointers.

    I then set the ComPlus_UseNewCrossDomainRemoting=0 environment variable and that changed the exception to a System.AccessViolationException that I think could have been caught by managed code. The native call stack was different too:

    00 MSVCR120_CLR0400!memcpy+0x147
    01 clr!CMessage::GetObjectFromStack+0x8c
    02 clr!CMessage::GetArgs+0x201
    03 mscorlib_ni+0x447f32
    04 mscorlib_ni+0x447b6d
    05 mscorlib_ni+0x4479ec
    06 mscorlib_ni+0x4478cc
    07 mscorlib_ni+0x447464
    08 clr!CTPMethodTable__CallTargetHelper3+0x12
    09 clr!CallTargetWorker2+0x74
    0a clr!CTPMethodTable::OnCall+0x1fb
    0b clr!TransparentProxyStub_CrossContextPatchLabel+0xa
    0c 0x000007fe`997001cb
    0d clr!CallDescrWorkerInternal+0x83
    0e clr!CallDescrWorkerWithHandler+0x4a
    0f clr!MethodDescCallSite::CallTargetWorker+0x251
    10 clr!RunMain+0x1ee
    11 clr!Assembly::ExecuteMainMethod+0xb6
    12 clr!SystemDomain::ExecuteMainMethod+0x506
    13 clr!ExecuteEXE+0x3f
    14 clr!_CorExeMainInternal+0xae
    15 clr!CorExeMain+0x14
    16 mscoreei!CorExeMain+0xe0
    17 MSCOREE!CorExeMain_Exported+0x57
    18 KERNEL32!BaseThreadInitThunk+0xd
    19 ntdll!RtlUserThreadStart+0x1d

    There was some managed-code remoting going on:

    0:000> !clrstack
    OS Thread Id: 0x5b74 (0)
            Child SP               IP Call Site
    000000000040e3a8 000007fef8c51307 [HelperMethodFrame_PROTECTOBJ: 000000000040e3a8] System.Runtime.Remoting.Messaging.Message.InternalGetArgs()
    000000000040e4b0 000007fef7ae7f32 System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage..ctor(System.Runtime.Remoting.Messaging.IMethodCallMessage)
    000000000040e540 000007fef7ae7b6d System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage)
    000000000040e600 000007fef7ae79ec System.Runtime.Remoting.Proxies.RemotingProxy.CallProcessMessage(System.Runtime.Remoting.Messaging.IMessageSink, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Contexts.ArrayWithSize, System.Threading.Thread, System.Runtime.Remoting.Contexts.Context, Boolean)
    000000000040e670 000007fef7ae78cc System.Runtime.Remoting.Proxies.RemotingProxy.InternalInvoke(System.Runtime.Remoting.Messaging.IMethodCallMessage, Boolean, Int32)
    000000000040e720 000007fef7ae7464 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
    000000000040ec28 000007fef8d57292 [TPMethodFrame: 000000000040ec28] Handler.Swap(Int32 ByRef, Int32 ByRef)
    000000000040ec90 000007fe997001cb .main()
    000000000040f010 000007fef8d4a7f3 [GCFrame: 000000000040f010] 
    Saturday, December 19, 2015 4:28 PM