locked
Custom structure into byte arrays - union style

    Question

  •  

    Hi,

    In C, I could use the following mechanism to access the bytes of a structure:

     

    union
    {
      unsigned char bytes[10];
      struct
      {
        int x;
        int y;
        unsigned char p;
        unsigned char q;
      }
    };

    How could I achieve the same in C#?

    Any help in this regard will be highly appreciated.

    Regards,
    Radha.

    Sunday, November 07, 2010 3:56 PM

Answers

  • Thanks for all the replies.

    If this way of getting bytes of a struct should be avoided, is there a suggested way of accessing the bytes of a struct?

    I would like to access the byte (stream) of a struct to be able to send it to a device - am looking for the best possible way to do this.

    Please let me know.

     

    Regards,
    Radha.


    Use Marshal and GCHandle methods to get/set the bytes from a struct. Here's an example that uses them to implement stream extension methods for reading/writing structs from/to a stream. You should be able to modify this to suit (note that you need to add these methods to a static class, like all extension methods).

    /// <summary>Read a struct from a <see cref="Stream"/>.</summary>
    /// <typeparam name="T">The type of the struct to read.
    /// The packing used must match that used to write the struct.</typeparam>
    /// <param name="input">The stream from which to read the struct.</param>
    /// <returns>The struct read in.</returns>
    
    public static T ReadStruct<T>(this Stream input) where T: struct
    {
      byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
    
      if (input.Read(buffer, 0, buffer.Length) != buffer.Length)
      {
        throw new IOException("Premature end of input stream while reading a struct.");
      }
    
      GCHandle gcHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    
      try
      {
        return (T)Marshal.PtrToStructure(gcHandle.AddrOfPinnedObject(), typeof(T));
      }
    
      finally
      {
        gcHandle.Free();
      }
    }
    
    /// <summary>Write a struct to a <see cref="Stream"/>.</summary>
    /// <typeparam name="T">The type of the struct to read.
    /// The packing used must match that used to write the struct.</typeparam>
    /// <param name="output">The stream to which to write the struct.</param>
    /// <param name="data">The struct to write.</param>
    
    public static void WriteStruct<T>(this Stream output, T data) where T: struct
    {
      byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
      GCHandle gcHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    
      try
      {
        Marshal.StructureToPtr(data, gcHandle.AddrOfPinnedObject(), false);
        output.Write(buffer, 0, buffer.Length);
      }
    
      finally
      {
        gcHandle.Free();
      }
    }
    
    

    Monday, November 08, 2010 1:17 PM

All replies

  • Hello,

     

    Please take a look at this thread . The question of unions and how to translate them has been answered there.

     

    Hope this helps, if you have any other questions or comments, please let me know,

    Best Regards,
    Emanuel Varga


    If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful".
    Sunday, November 07, 2010 4:21 PM
  • I would strongly advise against this practice in managed code.  I assume that you are doing Interop with C++.

    http://msdn.microsoft.com/en-us/library/acxa5b99(VS.80).aspx

    Untested.

     

        // unions do not exist in C#,

        // though they can be simulated, I discourage it

        [StructLayout(LayoutKind.Explicit)]

        struct UnionEmulater

        {

            [MarshalAs(UnmanagedType.ByValArray, SizeConst=40)]

            [FieldOffset(0)]

            public char[] bytes;

            [FieldOffset(0)]

            public int x;

            [FieldOffset(0)]

            public int y;

            [FieldOffset(0)]

            char p;

            [FieldOffset(0)]

            char q;

        }

     

    Rudy   =8^D


    Mark the best replies as answers. "Fooling computers since 1971."

    http://rudedog2.spaces.live.com/default.aspx

     

     

    EDIT:  This type fails because it tries to apply the overlapping layout attribute to a reference to an object on the managed heap.  If you delete the array declaration, then it will work properly.

     

     

        // unions do not exist in C#,

        // though they can be simulated, I discourage it

        [StructLayout(LayoutKind.Explicit)]

        struct UnionEmulater

        {

            [FieldOffset(0)]

            public byte b0;

            [FieldOffset(1)]

            public byte b1;

            [FieldOffset(2)]

            public byte b2;

            [FieldOffset(3)]

            public byte b3;

            [FieldOffset(4)]

            public byte b4;

            [FieldOffset(5)]

            public byte b5;

            [FieldOffset(6)]

            public byte b6;

            [FieldOffset(7)]

            public byte b7;

            [FieldOffset(8)]

            public byte b8;

            [FieldOffset(9)]

            public byte b9;

            //

            [FieldOffset(0)]

            public short x;

            [FieldOffset(0)]

            public short y;

            [FieldOffset(0)]

            public char p;

            [FieldOffset(0)]

            public char q;       

        }

     

     

    Sunday, November 07, 2010 4:27 PM
  • More like this I'd say:

        [StructLayout(LayoutKind.Explicit)]
        unsafe struct UnionEmulater
        {
          [FieldOffset(0)]
          public fixed byte bytes[10];
          [FieldOffset(0)]
          public int x;      
          [FieldOffset(4)]
          public int y;
          [FieldOffset(8)]
          byte p;
          [FieldOffset(9)]
          byte q;
        }
    
    
    • Not all offsets are 0 because some fields are in a struct.
    • C#'s char has 2 bytes instead of 1 like in C.
    • MarshalAs only works in interop scenarios. If you really want the bytes of the struct then you need a fixed array.

    Of course, this way of getting the bytes of a struct should be avoided if possible.

     

     

    Sunday, November 07, 2010 4:46 PM
  • Thanks for all the replies.

    If this way of getting bytes of a struct should be avoided, is there a suggested way of accessing the bytes of a struct?

    I would like to access the byte (stream) of a struct to be able to send it to a device - am looking for the best possible way to do this.

    Please let me know.

     

    Regards,
    Radha.

    Monday, November 08, 2010 4:48 AM
  • It should be avoided in normal scenarios. If you need to interop with native code or with some device, it's fine. There's also the manual alternative. Assuming you send the bytes one by one to the device you can do something like:

          foreach (byte b in BitConverter.GetBytes(foo.x))
            device.Write(b);
          foreach (byte b in BitConverter.GetBytes(foo.y))
            device.Write(b);
          device.Write(foo.p);
          device.Write(foo.q);
    
    
    Monday, November 08, 2010 7:43 AM
  • Thanks for all the replies.

    If this way of getting bytes of a struct should be avoided, is there a suggested way of accessing the bytes of a struct?

    I would like to access the byte (stream) of a struct to be able to send it to a device - am looking for the best possible way to do this.

    Please let me know.

     

    Regards,
    Radha.


    Use Marshal and GCHandle methods to get/set the bytes from a struct. Here's an example that uses them to implement stream extension methods for reading/writing structs from/to a stream. You should be able to modify this to suit (note that you need to add these methods to a static class, like all extension methods).

    /// <summary>Read a struct from a <see cref="Stream"/>.</summary>
    /// <typeparam name="T">The type of the struct to read.
    /// The packing used must match that used to write the struct.</typeparam>
    /// <param name="input">The stream from which to read the struct.</param>
    /// <returns>The struct read in.</returns>
    
    public static T ReadStruct<T>(this Stream input) where T: struct
    {
      byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
    
      if (input.Read(buffer, 0, buffer.Length) != buffer.Length)
      {
        throw new IOException("Premature end of input stream while reading a struct.");
      }
    
      GCHandle gcHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    
      try
      {
        return (T)Marshal.PtrToStructure(gcHandle.AddrOfPinnedObject(), typeof(T));
      }
    
      finally
      {
        gcHandle.Free();
      }
    }
    
    /// <summary>Write a struct to a <see cref="Stream"/>.</summary>
    /// <typeparam name="T">The type of the struct to read.
    /// The packing used must match that used to write the struct.</typeparam>
    /// <param name="output">The stream to which to write the struct.</param>
    /// <param name="data">The struct to write.</param>
    
    public static void WriteStruct<T>(this Stream output, T data) where T: struct
    {
      byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
      GCHandle gcHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    
      try
      {
        Marshal.StructureToPtr(data, gcHandle.AddrOfPinnedObject(), false);
        output.Write(buffer, 0, buffer.Length);
      }
    
      finally
      {
        gcHandle.Free();
      }
    }
    
    

    Monday, November 08, 2010 1:17 PM
  •  

    Hi,

    In C, I could use the following mechanism to access the bytes of a structure:

     

    union
    
    {
    
     unsigned char bytes[10];
    
     struct
    
     {
    
     int x;
    
     int y;
    
     unsigned char p;
    
     unsigned char q;
    
     }
    
    };

    How could I achieve the same in C#?

    Any help in this regard will be highly appreciated.

    Regards,
    Radha.


    No need for this in C#, you can use properties to acheieve the same thing.

    The (usual) motive for unions is to provide an alternative "view" of some storage, this is readily acheived with properties and fixed buffers and unsafe pointers, I haven't tested the struct below but it does compile and I have coded this kind of thing many times and had great success:

     public unsafe struct Union_01
     {
      private fixed byte bytes[10];
      
      public int x
      {
       get{
        fixed (byte * px = bytes)
          {
          int * int_ptr = (int*)(&(px[0]));
          return(*int_ptr);
          }
        }
       set
        {
        fixed (byte * px = bytes)
          {
          int * int_ptr = (int*)(&(px[0]));
          *int_ptr = value;
          }
        }
      }
    
      public int y
      {
       get
       {
        fixed (byte* px = bytes)
        {
         int* int_ptr = (int*)(&(px[4]));
         return (*int_ptr);
        }
       }
       set
       {
        fixed (byte* px = bytes)
        {
         int* int_ptr = (int*)(&(px[4]));
         *int_ptr = value;
        }
       }
      }
     
      public byte p
      {
       get {
        fixed (byte * ptr = bytes)
          return(ptr[8]);
        }
       set {
        fixed (byte * ptr = bytes)
          ptr[8] = value;
        }
      }
    
    
      public byte q
      {
       get
       {
        fixed (byte* ptr = bytes)
         return (ptr[9]);
       }
       set
       {
        fixed (byte* ptr = bytes)
         ptr[9] = value;
       }
      }
     }
    

    FYI the presence of properties has no impact whatsoever on marshaling or storage layout.

    Let me know how this goes !

    Cap'n

     

     

    Monday, November 08, 2010 2:10 PM
  • If your goal is to read/write stucts from/to a stream, you're better off using GC.Alloc() and Marshal methods, otherwise you'll have to mark your program as unsafe!
    Monday, November 08, 2010 2:19 PM
  • For completeness, I've put together a standalone console app that demonstrates how to stream a struct to/from a stream (which could be a file or, in this case, a memory buffer):

    using System;
    using System.IO;
    using System.Runtime.InteropServices;
    
    namespace ConsoleApplication3
    {
      static class Program
      {
        static void Main(string[] args)
        {
          MemoryStream stream = new MemoryStream();
          Test test1 = new Test { X = 1, Y = 2, P = 3, Q = 4 };
    
          stream.WriteStruct(test1);
          var array = stream.ToArray();
          Console.WriteLine("Stream contents: ");
    
          foreach (byte b in array)
          {
            Console.Write(b.ToString("X") + " ");
          }
    
          Console.WriteLine();
          stream.Seek(0, SeekOrigin.Begin);
          Test test2 = stream.ReadStruct<Test>();
          Console.WriteLine("\nRead back:");
    
          Console.WriteLine
          (
            string.Format
            (
              "X = {0}, Y = {1}, P = {2}, Q = {3}", 
              test2.X, test2.Y, test2.P, test2.Q
            )
          );
        }
    
        /// <summary>Read a struct from a <see cref="Stream"/>.</summary>
        /// <typeparam name="T">The type of the struct to read.
        /// The packing used must match that used to write the struct.</typeparam>
        /// <param name="input">The stream from which to read the struct.</param>
        /// <returns>The struct read in.</returns>
    
        public static T ReadStruct<T>(this Stream input) where T: struct
        {
          byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
    
          if (input.Read(buffer, 0, buffer.Length) != buffer.Length)
          {
            throw new IOException("Premature end of input stream while reading a struct.");
          }
    
          GCHandle gcHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    
          try
          {
            return (T)Marshal.PtrToStructure(gcHandle.AddrOfPinnedObject(), typeof(T));
          }
    
          finally
          {
            gcHandle.Free();
          }
        }
    
        /// <summary>Write a struct to a <see cref="Stream"/>.</summary>
        /// <typeparam name="T">The type of the struct to read.
        /// The packing used must match that used to write the struct.</typeparam>
        /// <param name="output">The stream to which to write the struct.</param>
        /// <param name="data">The struct to write.</param>
    
        public static void WriteStruct<T>(this Stream output, T data) where T: struct
        {
          byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
          GCHandle gcHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    
          try
          {
            Marshal.StructureToPtr(data, gcHandle.AddrOfPinnedObject(), false);
            output.Write(buffer, 0, buffer.Length);
          }
    
          finally
          {
            gcHandle.Free();
          }
        }
    
        [StructLayout(LayoutKind.Sequential, Pack=1)]
        public struct Test
        {
          public int X;
          public int Y;
          public byte P;
          public byte Q;
        }
      }
    }
    
    

    Monday, November 08, 2010 2:30 PM
  • Thanks for all the replies.

    If this way of getting bytes of a struct should be avoided, is there a suggested way of accessing the bytes of a struct?

    I would like to access the byte (stream) of a struct to be able to send it to a device - am looking for the best possible way to do this.

    Please let me know.

     

    Regards,
    Radha.

    If the struct contains no reference types, then you can take its address and thus get a (true) pointer to instances of it. That pointer will be the actual address of the start of the raw value type's data fields. However do not assume that the physcal layout is the same as you might get with C or C++ because it may not be, nor can you assume that fields are ordered as declared.

    Even with [StructLayout] the field ordering in managed memory may differ from the ordering of a marshalled instance of the struct, the [StructLayout] simply directs the marshaller it has no impact on the layout in managed memory, so be aware of this.

    If you are just going to take the struct, copy its data and repopulate another instance of the same struct later on, then these details are immaterial.

    The physical ordering and offsets of fields as well as the size of a struct in managed memory are not made available however so extreme caution is called for with this kind of thing.

    Cap'n

     

    Monday, November 08, 2010 2:37 PM
  • Even with [StructLayout] the field ordering in managed memory may differ from the ordering of a marshalled instance of the struct, the [StructLayout] simply directs the marshaller it has no impact on the layout in managed memory, so be aware of this.

    That's not quite true. The Pack value DOES affect the packing of the struct members in managed memory.

     

    Monday, November 08, 2010 2:51 PM
  • "Even with [StructLayout] the field ordering in managed memory may differ from the ordering of a marshalled instance of the struct, the [StructLayout] simply directs the marshaller it has no impact on the layout in managed memory, so be aware of this."

    Where did you get that? The documentation for StuctLayout doesn't make any reference to marshaling, it clearly says that this attribute does:

    "Typically, the common language runtime controls the physical layout of the data fields of a class or structure in managed memory. However, if you want to arrange the class or structure needs in a certain way, you can use StructLayoutAttribute. "

    Monday, November 08, 2010 2:54 PM
  • "Even with [StructLayout] the field ordering in managed memory may differ from the ordering of a marshalled instance of the struct, the [StructLayout] simply directs the marshaller it has no impact on the layout in managed memory, so be aware of this."

    Where did you get that? The documentation for StuctLayout doesn't make any reference to marshaling, it clearly says that this attribute does:

    "Typically, the common language runtime controls the physical layout of the data fields of a class or structure in managed memory. However, if you want to arrange the class or structure needs in a certain way, you can use StructLayoutAttribute. "


    Hello Mike

    The docs are very confusing, please see this post to understand why I said what I did, I am still not 100% clear on what .net does behind the scenes in some scenarios.

    Cap'n

     

    Tuesday, November 09, 2010 2:27 AM
  • Hrm, that looks more like a compiler/runtime bug. If you take out the DateTime field from that struct then everything works as expected. Since DateTime doesn't have a StructLayout attribute I assume you're breaking the rules but the compiler doesn't complain.

    You have to remember that there's more to interop than marshaling. There are also 'fixed' unsafe pointers that can be sent to unmanaged code without any marshaling. People have been used this stuff for a while now (including Microsoft products like XNA) and I've never seen a complaint about it not working.

    Of course, probably nobody tried to place a DateTime in such a struct because its internal layout and its size aren't known. It doesn't make sense to use it in such a case. And the only confusion in the docs is that they don't say what should happen in such a case.

    Tuesday, November 09, 2010 5:57 AM
  • Yeah, that's messed up. This documentation says that special marshalling is done for DateTime (and other types):

    http://social.msdn.microsoft.com/Forums/en-US/csharplanguage/thread/fb84bf1d-d9b3-4e91-823e-988257504b30

    Looks like some weirdness caused by that. Looks bad to me... I think I'll serialize the DateTime.Ticks property if I ever need to interop DateTime with legacy code (which, somehow, I think is never going to happen. :))

    [EDIT] I tested, and if you use Marshal.StructureToPtr() to marshal the struct, it works as expected, which leads me to conclude that the interop would work as expected too.

    Tuesday, November 09, 2010 9:53 AM
  • The order of the fields is not important until you either marshal it or take a pointer to it. Fixing the structure (either explicitly with the fixed keyword, or when marshaling), the order of the fields should be sequential.

    Items of an array are not one after the other in memory either. Until you marshal it, or fix it.

    Tuesday, November 09, 2010 11:29 AM
  • You make it sound as if "fixing" will change the layout of an object. There's no such statement in either the C# spec or the CLI spec. Fixing (aka pinning) only prevents the garbage collection from moving the object.
    Tuesday, November 09, 2010 12:07 PM
  • Items of an array are not one after the other in memory either. Until you marshal it, or fix it.

    Um... Elements of a single-dimensional zero-based array are definitely contiguously allocated (either references or value types) although of course in the case of reference-type elements, the objects that the references refer to are not contiguously allocated.
    Tuesday, November 09, 2010 12:24 PM
  • Maybe I'm not remembering correctly. Maybe it's only multidimensional arrays.

    Tuesday, November 09, 2010 1:27 PM
  • You make it sound as if "fixing" will change the layout of an object. There's no such statement in either the C# spec or the CLI spec. Fixing (aka pinning) only prevents the garbage collection from moving the object.


    Of course, you're right. I must have confused with something else but I don't see what.

    In fact, fixing it doesn't change anything about the strange layout in that Inner structure.

    Tuesday, November 09, 2010 1:36 PM
  • The order of the fields is not important until you either marshal it or take a pointer to it. Fixing the structure (either explicitly with the fixed keyword, or when marshaling), the order of the fields should be sequential.

    Items of an array are not one after the other in memory either. Until you marshal it, or fix it.


    But in the struct I give as an example, the physical ordering is not the same as the declared ordering. So taking a pointer to that and passing it (the pointer) to native code would cause the called code to see the F3 field in the wrong place, the act of taking the address would not magically rearrange the fields.

    Mike has raised a very good point, the presence of the DateTime seems to be causing this. I posted an earlier question/observation about DateTime actually, here is the post and it may all tie in together.

    The upshot is that by having DateTime embedded in the struct (in my other post) we make the entire struct non-blittable.

    A non-blittable struct (by defintion) will have/may have a different physical organization to a native "equivalent".

    So perhaps the bottom line her is that [StructLayout(LayoutKind.Sequential)] is only honored if the struct to which it is applied is blittable (meaning all fields are blittable).

    Do you guys agree with this?

    Cap'n

     

     

    Tuesday, November 09, 2010 1:43 PM
  • Items of an array are not one after the other in memory either. Until you marshal it, or fix it.

    Um... Elements of a single-dimensional zero-based array are definitely contiguously allocated (either references or value types) although of course in the case of reference-type elements, the objects that the references refer to are not contiguously allocated.


    Yes this is quite true, in fact you can use fixed to get an unsafe pointer to a 1D array of primitives and acccess the array elements using pointer [subscripting]. I think the language makes special exceptions for these.

    Cap'n

     

    Tuesday, November 09, 2010 1:47 PM
  • Looks like. Basically there are 2 clients of StructLayout:

    1. The marshaler - this one copies the cotent of a managed object to native memory and in the process it respects the StructLayout attribute, event if some fields are not blittable.
    2. The class loader - if explicit or sequential layout is requested then it arranges the fields in managed memory as requested but only if all fields are blittable.

    So now we know why the compiler doesn't issue a warning for a struct that contains DateTime and has sequential/explicit layout: it can't warn because it's a valid scenario when using marshaling.

    Tuesday, November 09, 2010 2:06 PM
  • So perhaps the bottom line her is that [StructLayout(LayoutKind.Sequential)] is only honored if the struct to which it is applied is blittable (meaning all fields are blittable).


    I guess that's what we can conclude. When required to marshal by ref, the marshaller will pin a struct and pass a pointer only if the struct is blittable. We shouldn't do otherwise.

    I really wonder what I was thinking about in my earlier reply.

    Edit: I guess I had a vague memory of this from the C# specification:

    7.6.10.4 Array creation expressions

    "Except in an unsafe context (§18.1), the layout of arrays is unspecified."

    Tuesday, November 09, 2010 2:13 PM
  • Do you guys agree with this?

    Cap'n

    Aye Cap'n! Seems that way. But I'd go further and say that it really does seem only to apply to the marshalled data.

    I asked a question about this on Stack Overflow:

    http://stackoverflow.com/questions/4132533/why-does-layoutkind-sequential-work-differently-if-a-struct-contains-a-datetime-f

    Tuesday, November 09, 2010 2:14 PM
  • Do you guys agree with this?

    Cap'n

    Aye Cap'n! Seems that way. But I'd go further and say that it really does seem only to apply to the marshalled data.

    I asked a question about this on Stack Overflow:

    http://stackoverflow.com/questions/4132533/why-does-layoutkind-sequential-work-differently-if-a-struct-contains-a-datetime-f


    OK I see, now what if I pass such a struct by ref to a native C function? In this case do we really do any "marshalling" at all?

    By ref will (in effect) pass the struct's address to the C function.

    Of course the C function cannot declare or assume anything about the DateTime field, but nevertheless if we wanted our C code to just look at the "first" field, we'd be in trouble here.

    I do know that compilers will insert padding in order to ensure fields are aligned correctly for their type, and will impose an alignment on the struct itself that depends upon the largest alignment of any of the fields (e.g. a struct that contains a double will always be aligned on an 8 byte boundary, but a stuct that has no fields larger than a short will only need to be aligned on a two byte boundary - for C anyway).

    But this has nothing to do with field ordering.

    Cap'n

     

    Tuesday, November 09, 2010 4:03 PM
  • I'm pretty sure that "by ref" marshals the struct using Marshal.StructureToPtr behind the scenes. It doesn't really pass a raw pointer across.
    Tuesday, November 09, 2010 4:11 PM
  • Hmm... http://msdn.microsoft.com/en-us/library/75dwhxf7.aspx

    "As an optimization, arrays of blittable types and classes that contain only blittable members are pinned instead of copied during marshaling. These types can appear to be marshaled as In/Out parameters when the caller and callee are in the same apartment. However, these types are actually marshaled as In parameters, and you must apply the InAttribute and OutAttribute attributes if you want to marshal the argument as an In/Out parameter."

    Tuesday, November 09, 2010 4:24 PM
  • I'm pretty sure that "by ref" marshals the struct using Marshal.StructureToPtr behind the scenes. It doesn't really pass a raw pointer across.


    Perhaps, but if that is true there is no point at all in passing such an item by ref, the whole point of which is to not do a copy of an item. If the marshaler does indeed create a copy (with an adjusted layout etc) and passes a ref to that copy, I think the compiler should warn us, because passing by ref in that scenario is no different to passing by value, possibly confusing the programmer.

    Cap'n

     

    Tuesday, November 09, 2010 5:21 PM
  • You pass by ref when the function expects a pointer. Why should the compiler complain about the design of the function you're calling? You'd have a warning on most of the system calls.

    Tuesday, November 09, 2010 6:14 PM
  • You pass by ref when the function expects a pointer. Why should the compiler complain about the design of the function you're calling? You'd have a warning on most of the system calls.


    I'm saying it should warn the programmer IF the struct being passed by ref cannot actually be passed by ref. If the struct must be marshaled and a ref passed to the copy, then we might as well pass it by value.

    That's all I' saying.

     

    Cap'n

     

    Tuesday, November 09, 2010 7:20 PM
  • You seem to confuse ref with fixed pointers or something.

    Ref is first and foremost a semantic, I pass some data to a function and I want that function to be able to modify my data. This is normally implemented by passing a pointer to data but it can also be implemented by copying. Copy the data, pass a pointer to the copied data to the function. When the function returns the "copied data" gets copied back over the initial data.

    You cannot replace "pass by ref" by "pass by value" because then you cannot get the modified data back.

    Tuesday, November 09, 2010 7:40 PM
  • You seem to confuse ref with fixed pointers or something.

    Ref is first and foremost a semantic, I pass some data to a function and I want that function to be able to modify my data. This is normally implemented by passing a pointer to data but it can also be implemented by copying. Copy the data, pass a pointer to the copied data to the function. When the function returns the "copied data" gets copied back over the initial data.

    You cannot replace "pass by ref" by "pass by value" because then you cannot get the modified data back.


    Yes you're right I agree, having reviewed what you say. Of course in other lanuages (e.g. Pascal or PL/I) passing by ref is passing the address of a datum, which allows the callee to access the datum in the caller. I guess that in C# the semantic behavior is the main aspect of pass by ref, whether it's implemented one way or another is an implementation detail.

    Cap'n

     

    Tuesday, November 09, 2010 10:32 PM
  • Hi Radha Krishna. S. _,

    Welcome to MSDN Forums!

     

    If your question has been solved, please don’t forget mark/vote the replies which helped you, and this will encourage the other community members to join in discussion and help each one.

     

    If there’s anything unclear, please feel free to let me know.

     

    Best wishes,

    Mike

    *****************************************************

    [All-In-One Code Framework]

    Sample world! You will get more from this world!

    Welcome to the new world!

    Wednesday, November 10, 2010 8:11 AM
  • Yeah, that's messed up. This documentation says that special marshalling is done for DateTime (and other types):

    http://social.msdn.microsoft.com/Forums/en-US/csharplanguage/thread/fb84bf1d-d9b3-4e91-823e-988257504b30

    Looks like some weirdness caused by that. Looks bad to me... I think I'll serialize the DateTime.Ticks property if I ever need to interop DateTime with legacy code (which, somehow, I think is never going to happen. :))

    [EDIT] I tested, and if you use Marshal.StructureToPtr() to marshal the struct, it works as expected, which leads me to conclude that the interop would work as expected too.


    I think "messed up" is a pretty good term here because the issue runs a bit deeper. It actually bit me yesterday in fact!

    I do a lot of high-perf network comms and uses structs as the basis for messages and protocols.

    Anyway, I have a struct message header that contains a DateTime, I knew the position of the timestamp within the header struct was messed up (due to the DateTime field), but this was irrelevant because I only (in effect) ever copy these structs from struct->byte array->struct so their actual layout is immaterial to my C# code.

    However I created a new message struct yesterday (all messages begin with a header field) and had issues during testing, it then all started to "make sense" so to speak.

    Here is bottom line after much fiddling around:

    If a struct A contains a field like DateTime (a struct with LayoutKind.Auto) then A's fields may not be ordered as expected, but if another struct B (with LayoutKind.Sequential) contains a field of type A, then B's fields may not be ordered as expected either.

    This propagates outward, so one might have structs containing fields of other structs that contain fields of other structs and so on, but if any struct in that hierarchy is LayoutKind.Auto every struct in the hierarchy suffers from field layout issues.

    This is potentially pretty bad, because fine, DateTime has LayoutKind.Auto so order it's fields as you see fit BUT structs with LayoutKind.Sequential that contain instances of DateTime should have their fields laid out as defined by their attributes.

    In my case my new message (very oddly) had its own fields messed up and thus its first field was NOT a header struct but some other field, all due to the header itself containing a DateTime.

    This is bad and not documented nor does it make sense, as I say by all means reorder the fields within DateTime, but do not rearrange fields in structs that simply contain fields of DateTime.

    I logged a bug report here.

    Cap'n

    PS I mention a workaround that works fine and should handle any cases of embedded struct's with Auto layout, so my issue is "solved" but this either needs to be fixed or documented in my view.

     

    Wednesday, November 10, 2010 1:05 PM
  • Seriously, you're making this bigger than it is. Maybe it doesn't make sense to you but common sense says that if you really care about your object's memory layout then you're careful what types you put in it. You didn't bother to check if DateTime is a blittable type, nor did you consider the fact that DateTime's size and contents is basically undefined. You got bitten by it and now you're making a bigg fuss about it.
    Wednesday, November 10, 2010 1:29 PM
  • Seriously, you're making this bigger than it is. Maybe it doesn't make sense to you but common sense says that if you really care about your object's memory layout then you're careful what types you put in it. You didn't bother to check if DateTime is a blittable type, nor did you consider the fact that DateTime's size and contents is basically undefined. You got bitten by it and now you're making a bigg fuss about it.


    I beg your pardon? What documentation says "if you really care about your object's memory layout then you're careful what types you put in it."? We are only told that we can use LayoutKind.Sequential or LayoutKind.Auto (or explicit) are we not?

    If people "care" about layout then they apply LayoutKind.Sequential do they not? For the system to ignore that is surely either a bug or a documentation omission - so I'm discusisng it here in a tech forum.

    I never cared if DateTime was blittable did I, why do you assume that? I use pointers sometime and you can take the address of a blittable or non-blittable struct (but not a struct with reference type fields). Blittable is only relevant if passing the struct to native code, I am not doing that here nor am I using Marshal either.

    "Bitten"? by what? I have encountered an issue in either the docs or the .Net field ordering, I have not coded anything at all that is non-standard here.

    Here is the problem, I create a family of structs, they all begin with a field of type MsgHeader. All of them are LayoutKind.Sequential. Now I do not "care" about the layout other than this: I expect each of them to always begin with a MsgHeader (or at least have the MsgHeader in the same position), I expect the first bytes in ever one of these structs to always be the MsgHeader bytes.

    So it is the fact that they do not share equivalent layout that is the issue NOT the actual ordering of the fields as such.

    But because MsgHeader contains a DateTime (non-blittable) not only is MsgHeader ordered oddly (but harmlessly) but every other struct that begins with a MsgHeader is also at risk. Even though every struct's first field is a MsgHeader, some struct do begin with that but some do not.

     

    Cap'n

     

     

    • Edited by Captain Kernel Wednesday, November 10, 2010 2:10 PM clarify
    Wednesday, November 10, 2010 1:47 PM
  • "Bitten"? by what? I have encountered an issue in either the docs or the .Net field ordering, I have not coded anything at all that is non-standard here.


    I think the documentation should mention something about non-blittable structs. I think also that you must have done something unusual. You mention copying a struct into a byte array then copying the byte array into a struct. I don't see how the actual layout of the struct would change anything.

    After checking again, the LayoutKind enum specifically mentions unmanaged memory, but I don't think the meaning of "The members of the object are laid out sequentially, in the order in which they appear when exported to unmanaged memory" is very clear. If you add a comma after "appear" it makes more sense and conforms to the real effect.

    Wednesday, November 10, 2010 2:15 PM
  • All fine but you made 2 threads about this and jumped in the middle of another one with claims about the documentation being confusing and SructLayout not working.

    What's confusing about this:

    "Typically, the common language runtime controls the physical layout of the data fields of a class or structure in managed memory. However, if you want to arrange the class or structure needs in a certain way, you can use StructLayoutAttribute"

    Nothing. It's just that the docs omit to say (or there's a bug in the runtime...) what happens in a corer case like yours. And you keep going on and on about it. Just sayin'...

    Wednesday, November 10, 2010 2:38 PM
  • "Bitten"? by what? I have encountered an issue in either the docs or the .Net field ordering, I have not coded anything at all that is non-standard here.

    I think the documentation should mention something about non-blittable structs. I think also that you must have done something unusual. You mention copying a struct into a byte array then copying the byte array into a struct. I don't see how the actual layout of the struct would change anything.


    Well it's pretty simple. Every msg struct has as its first declared field a MsgHeader (that contains various details like an enum for message type etc).

    Low level network code that extracts received data from a ring buffer needs to examine this header before it can know what kind of message it is, and then it can tell which kind of message struct to populate with the data in the buffer.

    The low level code therefore assumes that every message received begins with a header field and every struct is indeed declared to begin with a header struct.

    The async network management layer is now very fast indeed, uses no serialization, no marshaling etc and I can send and reinstantiate any message struct, but I do need to rely on a common header block.

    If this issue had no workaround (like avoiding the DateTime field which I now do) then I would need to actually physically transmit the header bytes myself as a distinct operation and then send the message itself, I did consider that and it would have worked and had little or no performance impact but is not very clean.

    So that's it, I needed to assume that a bunch of structs that all begin with a common header struct, do indeed all have that header struct at the same position (even if it wasn't loacted at the start, at least put it in the same relative place in every struct).

    Cap'n

     

     

    Wednesday, November 10, 2010 2:42 PM
  • I see. Something unusual, but not something wrong.

    The Standard ECMA-335 states that "The CLI shall lay out the fields in sequential order, based on the order of the fields in the logical metadata table".

    Looks like all the available documentation is stating the fields should be in the defined order, no matter the type of field.

    Wednesday, November 10, 2010 5:08 PM
  • All fine but you made 2 threads about this and jumped in the middle of another one with claims about the documentation being confusing and SructLayout not working.

    What's confusing about this:

    "Typically, the common language runtime controls the physical layout of the data fields of a class or structure in managed memory. However, if you want to arrange the class or structure needs in a certain way, you can use StructLayoutAttribute"

    Nothing. It's just that the docs omit to say (or there's a bug in the runtime...) what happens in a corer case like yours. And you keep going on and on about it. Just sayin'...


    Mike; why do some people like to get personal in these tech discussion? I didn't "jump in" to this thread any more than you or anyone else. I advised the poster and gave a neat way of avoiding his stated problem (with no need to use ANY attributes) and I mentioned in passing, an issue with StructLayout.Sequential, is that a crime? does that violate the site's terms of use?

    And I didn't "make two threads about this" either, they were independent threads and I merely referred to those threads in here, is that a crime too? does that violate the site's terms of use?

    Now before you continue with you diatribe and ill mannered remarks, you show me where the documentation says

    "StructLayout.Sequential does not always physically order the fields in declaration order, this is the case when any of the fields is itself a struct type that has LayoutKind.Auto or is not itself blittable" ?

    Does it say that? No. Would it help if it did say that? Yes. Might it actually be a bug though and the docs are actually already correct? Yes - but I don't know for sure and frankly neither do you.

    It is you who "keeps going on" Mike, you keep going on about me the poster rather than sticking to the topic.

    Regards

    Cap'n

     

     

     

    Wednesday, November 10, 2010 7:17 PM