locked
How to marshall an unknown byte length... RRS feed

  • Question

  • How do you assign the byte array length of an unknown field using the previous field?  In other words, when I read DataLen value, that will be the length for Data to read that many bytes in.  I am trying to marshall from memory stream to a data structure such as below.

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
        struct SampleDataStruct
        {

            public byte HeaderID;
            [MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
            public byte[] Unused;
            public int MessageLen;
            public short ID;
            public short DataLen;
            [MarshalAs(UnmanagedType.ByValArray, SizeConst = ???)]
            public byte[] Data;
            public short Status;
            public int Data2Len;
            [MarshalAs(UnmanagedType.ByValArray, SizeConst = ???)]
            public byte[] Data2;

        }
    Sunday, March 9, 2008 10:48 PM

Answers

  • Marshaling, and the various marshalling attributes, is strictly for managed/unmanaged interop.  For memory management inside .NET there is no need for marshaling.

     

    So I assume you have a structure serialized to disk and you want to get it back.  You'll have to write the code to do this manually.  If you have control over the serialization process then you could just serialize the structure out to disk using the binary serializer.  However if the data is serialized outside your control and the format can't be modified then you'll have to do it manually.  To deserialize the structure you need to read each field in order.  You can use a memory stream for this but it can complicate things.  It really depends upon what format your data resides in.

     

    In the general case you have the data on disk and you want to read it in.  You would normally use BinaryReader for this.  In that case the reader exposes methods to retrieve typed-values directly such as ReadInt32 or ReadBoolean.  You should use these to build up your structure field by field.  In the case of the array you'd use ReadInt32 to read the size in and assign it to the appropriate field.  You could then call ReadBytes and pass the length to get back an array of bytes.  Something like this:

    Code Snippet

    MyStruct ReadStruct ( BinaryReader reader )
    {

       MyStruct myStruct = new MyStruct();

      

       ...

       myStruct.Length = reader.ReadInt32();

       myStruct.Data = reader.ReadBytes(myStruct.Length);  //Creates and returns the array

       ...

       return myStruct;

    }

     

    If you have the data as a byte array (because that is how it is given to you) then you have two choices.  BitConverter can be used to convert a byte array back to typed values and works similar to a reader.  You have to keep track off the position within the array from which to read though.  For the sample of earlier:

     

    Code Snippet

    MyStruct ReadStruct ( byte[] data )
    {
       MyStruct myStruct = new MyStruct();

       int offset = 0;

     

       ...

       myStruct.Length = BitConverter.ToInt32(data, offset);

       offset += 4;

       myStruct.Data = new byte[myStruct.Length];

       Buffer.BlockCopy(data, offset, myStruct.Data, 0, myStruct.Length);

       offset += myStruct.Length;

       ...

     

       return myStruct;
    }

     

     

    Alternatively you can use a memory stream to wrap the byte array and then create a BinaryReader to wrap the stream.  This is a lot of overhead but it avoids keeping track of the offset.  I personally do this in some networking code that I use.  Something like this would work.

     

    Code Snippet

    MyStruct ReadStruct ( byte[] data )
    {
       MyStruct myStruct = new MyStruct();

     

       using(MemoryStream stream = new MemoryStream(data))

       {
          using(BinaryReader reader = new BinaryReader(stream))

          {

             ...

             myStruct.Length = reader.GetInt32();

             myStruct.Data = reader.GetBytes(myStruct.Length);

             ...

          };

       };

     

       return myStruct;
    }

     

     

    Ultimately I find the reader route the best.  If you have to use a byte array then wrapping a reader around it isn't so bad but ideally just use a reader directly rather than reading data into a byte array.

     

    Michael Taylor - 3/10/08

    http://p3net.mvps.org

     

     

    Monday, March 10, 2008 4:45 PM

All replies

  • You can't do that directly because the marshaler doesn't support it.  Just to be clear though we're talking about unmanaged to managed right?  If you have a regular memory stream in .NET then the rules are different.

     

    The problem is that the marshaler needs to be able to allocate the array before it reads the structure in and it can't do that since the field is not fixed.  However since you're dealing with unmanaged code this shouldn't be a problem anyway.  C/C++ does not allow you to allocate a dynamic array on the stack (it always gets allocated in the heap).  Therefore in the C/C++ code Data must either have a fixed size or be dynamically allocated using new/malloc.  Because of this you can't really read in the array by value as it won't actually be a value array (and hence why SizeConst only works with value arrays).

     

    Instead the array is really just a pointer inside the struct that points to the array somewhere on the heap.  To get this to marshal you should change the array to be an IntPtr.  After you read the structure in then you can marshal the array using the Marshal class directly as you'll then have the array length.

     

    Michael Taylor - 3/10/08

    http://p3net.mvps.org

     

    Monday, March 10, 2008 1:00 PM
  • Thank you for your reply.  I am rather a noob with data structures and found myself to look for a solution and learn this.  I am reading a stream of bytes using MemoryStream ms = new MemoryStream(byte[]). 

     

    My goal is to marshall (map) these bytes to a data structure such as above but am unclear how to deal with variable length bytes and would not know the length until I read the previous 4-byte field.  I have heard about the IntPtr.  It seems examples are hard to find.  Should I be looking at a different solution?  Can you lead me to some resource or example code on how to accomplish this?  I am using .NET 2.0 C#.  Very limited knowledge of C/C++.

     

    Thank you very much for you help.

     

     

    Monday, March 10, 2008 3:17 PM
  • Marshaling, and the various marshalling attributes, is strictly for managed/unmanaged interop.  For memory management inside .NET there is no need for marshaling.

     

    So I assume you have a structure serialized to disk and you want to get it back.  You'll have to write the code to do this manually.  If you have control over the serialization process then you could just serialize the structure out to disk using the binary serializer.  However if the data is serialized outside your control and the format can't be modified then you'll have to do it manually.  To deserialize the structure you need to read each field in order.  You can use a memory stream for this but it can complicate things.  It really depends upon what format your data resides in.

     

    In the general case you have the data on disk and you want to read it in.  You would normally use BinaryReader for this.  In that case the reader exposes methods to retrieve typed-values directly such as ReadInt32 or ReadBoolean.  You should use these to build up your structure field by field.  In the case of the array you'd use ReadInt32 to read the size in and assign it to the appropriate field.  You could then call ReadBytes and pass the length to get back an array of bytes.  Something like this:

    Code Snippet

    MyStruct ReadStruct ( BinaryReader reader )
    {

       MyStruct myStruct = new MyStruct();

      

       ...

       myStruct.Length = reader.ReadInt32();

       myStruct.Data = reader.ReadBytes(myStruct.Length);  //Creates and returns the array

       ...

       return myStruct;

    }

     

    If you have the data as a byte array (because that is how it is given to you) then you have two choices.  BitConverter can be used to convert a byte array back to typed values and works similar to a reader.  You have to keep track off the position within the array from which to read though.  For the sample of earlier:

     

    Code Snippet

    MyStruct ReadStruct ( byte[] data )
    {
       MyStruct myStruct = new MyStruct();

       int offset = 0;

     

       ...

       myStruct.Length = BitConverter.ToInt32(data, offset);

       offset += 4;

       myStruct.Data = new byte[myStruct.Length];

       Buffer.BlockCopy(data, offset, myStruct.Data, 0, myStruct.Length);

       offset += myStruct.Length;

       ...

     

       return myStruct;
    }

     

     

    Alternatively you can use a memory stream to wrap the byte array and then create a BinaryReader to wrap the stream.  This is a lot of overhead but it avoids keeping track of the offset.  I personally do this in some networking code that I use.  Something like this would work.

     

    Code Snippet

    MyStruct ReadStruct ( byte[] data )
    {
       MyStruct myStruct = new MyStruct();

     

       using(MemoryStream stream = new MemoryStream(data))

       {
          using(BinaryReader reader = new BinaryReader(stream))

          {

             ...

             myStruct.Length = reader.GetInt32();

             myStruct.Data = reader.GetBytes(myStruct.Length);

             ...

          };

       };

     

       return myStruct;
    }

     

     

    Ultimately I find the reader route the best.  If you have to use a byte array then wrapping a reader around it isn't so bad but ideally just use a reader directly rather than reading data into a byte array.

     

    Michael Taylor - 3/10/08

    http://p3net.mvps.org

     

     

    Monday, March 10, 2008 4:45 PM
  • Thank you Michael.  I forgot about the BinaryReader.  I knew it was available but didn't think twice to use it.  I was reading a FileStream to a byte array.  Wrapping the byte array with a MemoryStream, keeping tracking of offsets and using BitConverter.  Now, I will just wrap the FileStream with a BinaryReader.  It's not so much of keeping track of offsets and using the BitConverter a problem.  I was looking for something along the line of a structure schema you can assign to the reader and it will partition the stream based on that schema.  If the incoming data structure were to change, you can easily change the schema without rewriting code.  I hope that makes sense.

     

    Again thank you for your time.  The last code snippet will suit my purpose.

     

    Monday, March 10, 2008 6:40 PM