none
Marshal.StructureToPtr() where struct contains a byte array of unknown size RRS feed

  • Question

  • Hi to all,

    I'm stuck on what I think should be an easy fix.
    I need to convert the following c++ struct into c#:

    typedef struct MainStruct {
    	uint8_t fieldA;
    	uint8_t fieldB;
    	uint16_t fieldC;
    	uint16_t fieldD;
    	uint16_t fieldE;
    	char data[];
    } MainStruct;
    

    This is my c# implementation:

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
    struct MainStruct {
    	byte fieldA;
    	byte fieldB;
    	UInt16 fieldC;
    	UInt16 fieldD;
    	UInt16 fieldE;
    	[MarshalAs(UnmanagedType.ByValArray)]
            byte[] data;
    }
    With the following code I create an instance of the structure, and, after inserting data, I  convert the whole thing into a pointer to be passed as an argument of an unmanaged DLL.
    MainStruct Instance = new MainStruct();
    Instance.fieldA = 1;
    Instance.fieldB = 2;
    Instance.fieldC = 5;
    Instance.fieldD = 3;
    Instance.fieldE = 0;
    Instance.data = MyArrayOfBytes();
    
    IntPtr ptrRequest = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(MainStruct)) + MyArrayOfBytes.lenght());
    Marshal.StructureToPtr(Instance, ptrRequest, true);
    
    bool success = MyDllFunction(ptrRequest);
    The main problem is that the length of the data array it's different on each instance of the structure and therefore the pointer created from StructureToPtr() is not valid.
    If I try to add "SizeConst" attribute, I obtain a valid pointer that works as expected when passed to the unmanaged dll function.

    In c++ it's very easy, I just need to allocate the required memory to the data field and nothing more, but in C#, after hours of researches I still didn't found a solution.

    Thanks to all
    Tuesday, March 20, 2012 7:33 PM

Answers

  • Hello mark_555,

    1. The Problems.

    1.1 After reading the OP and subsequent posts, I gather that you want the "data" field to be part of the structure when viewed from the C++ DLL.

    1.2 There is no other way to achieve this other than to decorate "data" with the MarshalAsAttribute together with the UnmanagedType.ByValArray and the SizeConst parameters.

    1.2 However, another requirement is that the "data" field must not be fixed in size.

    1.3 This renders impossible the objective in 1.1.

    1.4 With reference to your second post : by declaring "data" to be of type IntPtr and then dynamically allocating memory space for the array, you separate the array from the structure immediately. There is no way for Marshal.StructureToPtr() to somehow copy the contents of the array that "data" points to into "ptrRequest".

    2. Possible Solution.

    2.1 One solution is to define 2 structures. The first one is used only in managed code. The other structure serves as an unmanaged version of the first structure. It is manually and dynamically created (based on the values of the managed structure) to be passed to the unmanaged code.

    2.2 The following is a sample code :

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
    struct MainStruct 
    {
        public byte fieldA;
        public byte fieldB;
        public UInt16 fieldC;
        public UInt16 fieldD;
        public UInt16 fieldE;
        // Do not need to specify how "data"
        // is to be marshaled. It remains 
        // a managed field and it is not blittable.
        //[MarshalAs(UnmanagedType.ByValArray)]
        public byte[] data;
    }
    
    // Define an unmanaged equivalent of MainStruct.
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
    struct UnmanagedMainStruct
    {
        public byte fieldA;
        public byte fieldB;
        public UInt16 fieldC;
        public UInt16 fieldD;
        public UInt16 fieldE;
        // Do not define any field to represent
        // the "data" byte array.
    }

    Notice that the "data" field of MainStruct remains a managed array. This is necessary in order that it be variable in size.

    For the UnmanagedMainStruct structure, there is no equivalent "data" field. This field will be allocated dynamically as will be seen later. It is an UnmanagedMainStruct structure (together with an extension memory for its undeclared "data" field) that will be passed to the C++ function.

    2.3 Here is a sample function that will perform a manual dynamic allocation of an UnmanagedMainStruct together with the extended memory for its undeclared "data" field :

    // Just a helper function to return an initialized 
    // array of bytes.
    private static byte[] ReturnByteArray()
    {
        byte [] byRet = new byte[30];
    
        for (int i = 0; i < byRet.Length; i++)
        {
            byRet[i] = (byte)(i + 1);
        }
    
        return byRet;
    }
    
    private static void DoTest()
    {
        // Allocate a MainStruct and fill in 
        // its fields including its "data" field.
        MainStruct Instance = new MainStruct();
        Instance.fieldA = 1;
        Instance.fieldB = 2;
        Instance.fieldC = 5;
        Instance.fieldD = 3;
        Instance.fieldE = 0;
        Instance.data = ReturnByteArray();
    
        // Allocate an UnmanagedMainStruct structure
        // and fill in its fields based on those of
        // the MainStruct struct "Instance".
        UnmanagedMainStruct UnmanagedInstance = new UnmanagedMainStruct();
        UnmanagedInstance.fieldA = Instance.fieldA;
        UnmanagedInstance.fieldB = Instance.fieldB;
        UnmanagedInstance.fieldC = Instance.fieldC;
        UnmanagedInstance.fieldD = Instance.fieldD;
        UnmanagedInstance.fieldE = Instance.fieldE;
    
        // We now dynamically allocate space for an
        // UnmanagedMainStruct structure with extended
        // memory space for the extra (undeclared) 
        // "data" field.
        IntPtr ptrRequest = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(UnmanagedMainStruct)) + Instance.data.Length);
        // Transfer the existing field values of 
        // "UnmanagedInstance" to the memory space
        // pointed to by "ptrRequest".
        Marshal.StructureToPtr(UnmanagedInstance, ptrRequest, true);
        // Calculate the offset to the undeclared "data" 
        // field which is the end of an UnmanagedMainStruct 
        // structure.
        int OffsetToByteArray = Marshal.SizeOf(typeof(UnmanagedMainStruct));
        // Obtain the memory address of this undeclared 
        // "data" field based on the offset.
        IntPtr pAddressOfByteArray = new IntPtr(ptrRequest.ToInt32() + OffsetToByteArray);
        // Copy the contents of the "data" field of 
        // "Instance" to the "data" field of "ptrRequest".
        Marshal.Copy(Instance.data, 0, pAddressOfByteArray, Instance.data.Length);
    
        bool success = MyDllFunction(ptrRequest);
    
        Marshal.FreeHGlobal(ptrRequest);
        ptrRequest = IntPtr.Zero;
    }
    

    The DoTest() function performs this dynamic manual allocation for an unmanaged version of MainStruct. Please read the comments for an explanation of how this can be done.

    3. Generalizing The Technique Into a Custom Marshaler.

    3.1 Please let me know if it works for you.

    3.2 If you find the technique suitable for usage, it may be further generalized into a custom marshaler.

    - Bio.


    Please visit my blog : http://limbioliong.wordpress.com/

    • Marked as answer by mark_555 Friday, March 23, 2012 12:45 AM
    Wednesday, March 21, 2012 2:47 PM

All replies

  • "In c++ it's very easy, I just need to allocate the required memory to the data field and nothing more,"

    Its just as easy in C#.  Allocate the required memory to the data field and pass the ptr you get from the allocation.  You can use GCHandle.Alloc or Marshal.AllocHGlobal.  I prefer GCHandle.

    Tuesday, March 20, 2012 8:34 PM
  • Hi John and thanks for your fast reply. Yes, I had already tried this method before posting, but without success.
    This is what I had done:
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
    struct MainStruct {
    	byte fieldA;
    	byte fieldB;
    	UInt16 fieldC;
    	UInt16 fieldD;
    	UInt16 fieldE;
    	IntPtr data;
    }
    
    MainStruct Instance = new MainStruct();
    Instance.fieldA = 1;
    Instance.fieldB = 2;
    Instance.fieldC = 5;
    Instance.fieldD = 3;
    Instance.fieldE = 0;
    Instance.data = Marshal.AllocHGlobal(MyArrayOfBytes.lenght());
    Marshal.Copy(MyArrayOfBytes, 0, Instance.data, MyArrayOfBytes.Length);
    
    IntPtr ptrRequest = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(MainStruct)) + MyArrayOfBytes.lenght());
    Marshal.StructureToPtr(Instance, ptrRequest, true);
    bool success = MyDllFunction(ptrRequest);
    I checked the validity of the final IntPtr and I can confirm that it is 100% valid because I can get the full struct back again using the PtrToStructure method. I captured the data sent to the dll function and I can clearly see that bytes are formatted in a different way respect to "UnmanagedType.ByValArray" method.
    The function that I'm calling is DeviceIoControl and, as far as I know, the pinvoke declaration only accepts an IntPtr value as lpInBuffer.

    I really don't know how to solve that problem, maybe some sort of hack that force  "SizeConst" to accept a custom variable ?



    Tuesday, March 20, 2012 10:23 PM
  • I don't understand.  What does that Structure have to do with calling DeviceIoControl?  You get the elements of the structure from the returned buffer.  Call the function, get the required size of the buffer, allocate the buffer and call the function again.
    Edit:

    "The function that I'm calling is DeviceIoControl and, as far as I know, the pinvoke declaration only accepts an IntPtr value as lpInBuffer."

    It's your definition.  You can call it anyway you wish.  pinvoke declarations are one coders idea of how the function should be called, not necessarily the best, or even adequate way.

    • Edited by JohnWein Tuesday, March 20, 2012 10:55 PM
    Tuesday, March 20, 2012 10:36 PM
  • The function accept a buffer representing a custom struct. In this case, I don't have to call the function again. Once the pointer is passed, the function continue on its own.
    I didn't mentioned DeviceIoControl and I have used a generic name for the sample code to avoid any confusion and also because I think the problem is not the function, but the way the data is marshalled.

    Using the following method, output is valid and everything works, but as side adverse, I have to provide the size of the array that unfortunately change every time:
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 7)]
    internal byte[] data;

    The alternative method (data array as ptr) does not work because the resulted output is different from the method above. Even with a sample array of the same lenght (7) the resulted bytes are different.

    The only way I can get this to work is to marshal the array as ByValArray. Thanks
    Tuesday, March 20, 2012 11:10 PM
  • Hello mark_555,

    1. The Problems.

    1.1 After reading the OP and subsequent posts, I gather that you want the "data" field to be part of the structure when viewed from the C++ DLL.

    1.2 There is no other way to achieve this other than to decorate "data" with the MarshalAsAttribute together with the UnmanagedType.ByValArray and the SizeConst parameters.

    1.2 However, another requirement is that the "data" field must not be fixed in size.

    1.3 This renders impossible the objective in 1.1.

    1.4 With reference to your second post : by declaring "data" to be of type IntPtr and then dynamically allocating memory space for the array, you separate the array from the structure immediately. There is no way for Marshal.StructureToPtr() to somehow copy the contents of the array that "data" points to into "ptrRequest".

    2. Possible Solution.

    2.1 One solution is to define 2 structures. The first one is used only in managed code. The other structure serves as an unmanaged version of the first structure. It is manually and dynamically created (based on the values of the managed structure) to be passed to the unmanaged code.

    2.2 The following is a sample code :

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
    struct MainStruct 
    {
        public byte fieldA;
        public byte fieldB;
        public UInt16 fieldC;
        public UInt16 fieldD;
        public UInt16 fieldE;
        // Do not need to specify how "data"
        // is to be marshaled. It remains 
        // a managed field and it is not blittable.
        //[MarshalAs(UnmanagedType.ByValArray)]
        public byte[] data;
    }
    
    // Define an unmanaged equivalent of MainStruct.
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
    struct UnmanagedMainStruct
    {
        public byte fieldA;
        public byte fieldB;
        public UInt16 fieldC;
        public UInt16 fieldD;
        public UInt16 fieldE;
        // Do not define any field to represent
        // the "data" byte array.
    }

    Notice that the "data" field of MainStruct remains a managed array. This is necessary in order that it be variable in size.

    For the UnmanagedMainStruct structure, there is no equivalent "data" field. This field will be allocated dynamically as will be seen later. It is an UnmanagedMainStruct structure (together with an extension memory for its undeclared "data" field) that will be passed to the C++ function.

    2.3 Here is a sample function that will perform a manual dynamic allocation of an UnmanagedMainStruct together with the extended memory for its undeclared "data" field :

    // Just a helper function to return an initialized 
    // array of bytes.
    private static byte[] ReturnByteArray()
    {
        byte [] byRet = new byte[30];
    
        for (int i = 0; i < byRet.Length; i++)
        {
            byRet[i] = (byte)(i + 1);
        }
    
        return byRet;
    }
    
    private static void DoTest()
    {
        // Allocate a MainStruct and fill in 
        // its fields including its "data" field.
        MainStruct Instance = new MainStruct();
        Instance.fieldA = 1;
        Instance.fieldB = 2;
        Instance.fieldC = 5;
        Instance.fieldD = 3;
        Instance.fieldE = 0;
        Instance.data = ReturnByteArray();
    
        // Allocate an UnmanagedMainStruct structure
        // and fill in its fields based on those of
        // the MainStruct struct "Instance".
        UnmanagedMainStruct UnmanagedInstance = new UnmanagedMainStruct();
        UnmanagedInstance.fieldA = Instance.fieldA;
        UnmanagedInstance.fieldB = Instance.fieldB;
        UnmanagedInstance.fieldC = Instance.fieldC;
        UnmanagedInstance.fieldD = Instance.fieldD;
        UnmanagedInstance.fieldE = Instance.fieldE;
    
        // We now dynamically allocate space for an
        // UnmanagedMainStruct structure with extended
        // memory space for the extra (undeclared) 
        // "data" field.
        IntPtr ptrRequest = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(UnmanagedMainStruct)) + Instance.data.Length);
        // Transfer the existing field values of 
        // "UnmanagedInstance" to the memory space
        // pointed to by "ptrRequest".
        Marshal.StructureToPtr(UnmanagedInstance, ptrRequest, true);
        // Calculate the offset to the undeclared "data" 
        // field which is the end of an UnmanagedMainStruct 
        // structure.
        int OffsetToByteArray = Marshal.SizeOf(typeof(UnmanagedMainStruct));
        // Obtain the memory address of this undeclared 
        // "data" field based on the offset.
        IntPtr pAddressOfByteArray = new IntPtr(ptrRequest.ToInt32() + OffsetToByteArray);
        // Copy the contents of the "data" field of 
        // "Instance" to the "data" field of "ptrRequest".
        Marshal.Copy(Instance.data, 0, pAddressOfByteArray, Instance.data.Length);
    
        bool success = MyDllFunction(ptrRequest);
    
        Marshal.FreeHGlobal(ptrRequest);
        ptrRequest = IntPtr.Zero;
    }
    

    The DoTest() function performs this dynamic manual allocation for an unmanaged version of MainStruct. Please read the comments for an explanation of how this can be done.

    3. Generalizing The Technique Into a Custom Marshaler.

    3.1 Please let me know if it works for you.

    3.2 If you find the technique suitable for usage, it may be further generalized into a custom marshaler.

    - Bio.


    Please visit my blog : http://limbioliong.wordpress.com/

    • Marked as answer by mark_555 Friday, March 23, 2012 12:45 AM
    Wednesday, March 21, 2012 2:47 PM
  • Hi Bio and many thanks for your professional and detailed solution. Before reading your post I decided to follow John suggestion to investigate on a possible different pinvoke declaration and turned out that DeviceIoControl seems to accept also an array of bytes instead of just plain IntPtr. 
    This is a common declaration:
    [DllImport("kernel32.dll", ExactSpelling = true, SetLastError = true, CharSet = CharSet.Auto)]
    static extern bool DeviceIoControl(IntPtr hDevice, uint dwIoControlCode,
    IntPtr lpInBuffer, uint nInBufferSize,
    IntPtr lpOutBuffer, uint nOutBufferSize,
    out uint lpBytesReturned, IntPtr lpOverlapped);

    I changed it directly into this:
    [DllImport("kernel32.dll", ExactSpelling = true, SetLastError = true, CharSet = CharSet.Auto)]
    static extern bool DeviceIoControl(IntPtr hDevice, int dwIoControlCode,
    byte[] MainStructIn , int MainStructInSize,
    byte[] MainStructOut, int MainStructOutSize ,
    out int lpBytesReturned, IntPtr lpOverlapped);

    I'm not sure if this declaration is safe or not, but I have seen that the "coredll" equivalent use directly arrays of bytes as parameter and, to my suprise, works also with the kernel32 version.

    I have removed MainStruct and now, without any marshalling operation, I just send the data in the following way:

    byte[] MainStruct = new byte[OFFSET +  MyArrayOfBytes.Length]; //OFFSET(8) = the lenght in bytes of FieldA + FieldB + FieldC + FieldD + FieldE
    BitConverter.GetBytes(FieldA).CopyTo(ControlTransfer, 0);
    BitConverter.GetBytes(FieldB).CopyTo(ControlTransfer, 1);
    BitConverter.GetBytes(FieldC).CopyTo(ControlTransfer, 2);
    BitConverter.GetBytes(FieldD).CopyTo(ControlTransfer, 4);
    BitConverter.GetBytes(FieldE).CopyTo(ControlTransfer, 6);
    Array.Copy(MyArrayOfBytes, 0, MainStruct, OFFSET, MyArrayOfBytes.Lenght);
    
    bool success = DeviceIoControl(TheOtherRequiredParameters, MainStruct, MainStruct.Lenght, TheOthersRequiredParameters);
    With the old struct there was the advantage to create an instance in every part of the code, but I can always stick the new code in a function that returns a the filled array.
    The only thing that leave me worried is the memory. From the managed side seems to be that there's nothing to release, but I don't know if passing of bytes arrays to DeviceIoControl can cause adverse effects in the long time or any sort of corruption. For now everything seems to work flawlessly.

    I've read carefully your solution and I saved all of your instructions in the case that in future I need it, but for now I don't think it's the case to write so much code just for passing an array of bytes to a function. Thanks for so much effort, I'll mark your solution as accepted anyway.

    Thanks


    Friday, March 23, 2012 12:45 AM