none
why MemoryMappedFiles only work with simple structures not with arrays inside a structures. RRS feed

  • Question

  • I have a very complex and large object containing arrays of structures an primitive types in C++ code. We were using this object in C# (using marshalling /unmarshalling) and then pass it to C++ to do some process and then return back to C# for further processing. The problem is this marshalling and unmarshalling is taking too much time across the boundary (few seconds). Since our program is time critical and we do not own the C++ code( large object belong to 3rd party), we have to use the object as it is across the boundary. 

    So I wanted to shared a memory between the C# and C++ processes for the large object or for array of large objects. Such that I create the object once and then pass the pointer between the C# and C++ code. I read article related to MemoryMappedFile in .NetFramework 4 and C++.

    http://blogs.msdn.com/b/salvapatuel/archive/2009/06/08/working-with-memory-mapped-files-in-net-4.aspx

    http://msdn.microsoft.com/en-us/library/windows/desktop/aa366551(v=vs.85).aspx

    http://www.codeproject.com/Articles/138290/Programming-Memory-Mapped-Files-with-the-NET-Frame

    But the problem is it only works on simple structure. Please see below. 

     public struct Point
        {
            public int x;
            public int y;
        }

        public struct MyData
        {
            public int myInt;
            public Point myPt;

        }


    But gives error 

           

     public struct Point
        {
            public int x;
            public int y;
        }

        public struct MyData
        {
            public int myInt;
    public Point[] myPt;

            public MyData(int count)
            {
                myInt = 0;
                myPt = new Point[count];
            }
        }



    I just tried to run some prototype using above article and ran into problem

    Here is the C# code that creates the MMF (shared memory)

    Class SharedMemory.cs

                                   

    using System;
    using System.IO;
    using System.Collections.Generic;
    using System.IO.MemoryMappedFiles;
    using System.Threading;
    using System.Runtime.InteropServices;
    namespace PinnedObjectTest
    {
        public class SharedMemory<T> where T : struct
        {
            // Constructor
            public SharedMemory(string name, int size)
            {
                smName = name;
                smSize = size;
            }

            // Methods
            public bool Open()
            {
                try
                {
                    // Create named MMF
                    mmf = MemoryMappedFile.CreateOrOpen(smName, smSize);

                    // Create accessors to MMF
                    accessor = mmf.CreateViewAccessor(0, smSize,
                                   MemoryMappedFileAccess.ReadWrite);

                    // Create lock
                    smLock = new Mutex(true, "SM_LOCK", out locked);
                }
                catch
                {
                    return false;
                }

                return true;
            }

            public void Close()
            {
                accessor.Dispose();
                mmf.Dispose();
                smLock.Close();
            }

            public T Data
            {
                get
                {
                    T dataStruct;
    accessor.Read<T>(0, out dataStruct);
                    return dataStruct;
                }
                set
                {
                    smLock.WaitOne();
    accessor.Write<T>(0, ref value);
                    smLock.ReleaseMutex();
                }
            }

            // Data
            private string smName;
            private Mutex smLock;
            private int smSize;
            private bool locked;
            private MemoryMappedFile mmf;
            private MemoryMappedViewAccessor accessor;
        }

        public struct Point
        {
            public int x;
            public int y;
        }

        public struct MyData
        {
            public int myInt;
            public Point[] myPt;

            public MyData(int count)
            {
                myInt = 0;
                myPt = new Point[count];
            }
        }
    }

    Main Program  code:

                    

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.IO.MemoryMappedFiles;

    namespace PinnedObjectTest
    {
        class Program
        {
            static void Main(string[] args)
            {

            SharedMemory<MyData> shmem = 
                 new SharedMemory<MyData>("ShmemTest", 9000);

            if(!shmem.Open()) return;

            MyData data = new MyData(1000);

            // Read from shared memory
            data = shmem.Data;
            Console.WriteLine("CurrentTime: {0}", DateTime.Today.Date.TimeOfDay);
            Console.WriteLine("MyInt: {0}", data.myInt);
            for (int i = 0; i < data.myPt.Length; i++)
            {
                Console.WriteLine("{0}, {1}", data.myPt[i].x, data.myPt[i].y);
            }
            Console.WriteLine("CurrentTime: {0}", DateTime.Today.Date.TimeOfDay);
            // Press any key to exit
            Console.ReadKey();

            // Close shared memory
            shmem.Close();        

            }
        }
    }

    The above code fail @

    accessor.Read<T>(0, out dataStruct); 

    An unhandled exception of type 'System.ArgumentException' occurred in mscorlib.dll

    Additional information: The specified Type must be a struct containing no references.

    The C++ code looks something like this

    UnmanagedSharedMemory.h

     

    typedef struct
    {
    int x;
    int y;
    }Point;

    typedef struct
    {
    int myInt;
         Point myPt[1000];
    }Data;

    UnmanagedSharedMemory.cpp

    // UnmanagedSharedMemory.cpp : Defines the entry point for the console application.
    //

    #include <windows.h>
    #include <stdio.h>
    #include <conio.h>
    #include <tchar.h>
    #include "UnmanagedSharedMemory.h"
    #pragma comment(lib, "user32.lib")

    #define BUF_SIZE 9000
    TCHAR szName[]=TEXT("ShmemTest");

    int _tmain()
    {

       HANDLE hMapFile;
       void* pBuf;

       hMapFile = OpenFileMapping(
                       FILE_MAP_ALL_ACCESS,   // read/write access
                       FALSE,                 // do not inherit the name
                       szName);               // name of mapping object

       if (hMapFile == NULL)
       {
          _tprintf(TEXT("Could not open file mapping object (%d).\n"),
                 GetLastError());
          return 1;
       }

       pBuf = (void*) MapViewOfFile(hMapFile, // handle to map object
                   FILE_MAP_ALL_ACCESS,  // read/write permission
                   0,
                   0,
                   BUF_SIZE);

       if (pBuf == NULL)
       {
          _tprintf(TEXT("Could not map view of file (%d).\n"),
                 GetLastError());

          CloseHandle(hMapFile);
     
          return 1;
       }

       Data* data = (Data*) pBuf;
       printf("Buf->MyInt (%d).\n",data->myInt);
       data->myInt = 33;
       for(int i = 0; i < 1000; i++)
       {
    data->myPt[i].x = i;
    data->myPt[i].y = 1000 - i;
       }
       UnmapViewOfFile(pBuf);

       CloseHandle(hMapFile);
       data = NULL;
       return 0;
    }

    Please help how to work with shared memory with structures that have arrays and we do not need to do the copying across the boundary every time. How can we fix this issue. 

    Any help is highly appreciated !

     

    Nadeem

    Tuesday, December 17, 2013 9:21 PM

Answers

  • This has nothing to do with marshaling, marshaling is not used when dealing with memory mapped files. That's exactly the reason you can't use a type that contains references like MyData.

    The only way to get that data into a memory mapped file is to manually write its fields:

    accessor.Write(0, ref data.myInt); 
    accessor.WriteArray(4, data.myPt, 0, data.myPt.Length);
    

    Well, actually there's a way to use marshaling too but that would probably defeat your purpose, to be efficient. Try it though, maybe it's good enough:

    byte *ptr = null;
    try {
        accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        Marshal.StructureToPtr(value, (IntPtr)ptr, false);
    }
    finally {
        if (ptr != null)
            accessor.SafeMemoryMappedViewHandle.ReleasePointer();
    }
    
    This is for the set accessor of your Data property. The get accessor will need similar code but with Mar

    shal.PtrToStructure instead. Though if the array size isn't const like in your example I'm not sure how this could work...

    Wednesday, December 18, 2013 1:03 PM
    Moderator

All replies

  • The default marshaller cannot marshal variable-length arrays. You need to serialize the array to flat memory (e.g. using  BinaryFormatter, BitConverter for primitive types or DataContractSerializer).


    Visual C++ MVP

    Tuesday, December 17, 2013 11:03 PM
  • Can you please give an example.

    Do you mean something like this

      public struct MyData
        {
            public int myInt;

           [MarshalAs(UnManaged.ValArray, SizeConst = 1000)]

           public Point[] myPt;

        }

    So we will still be doing the copy in the shared memory. If we are still serializing across the boundary of C# and C++ then how shared memory is different than regular marshal/unmarshal. Also, Flat structure will be complex as I need to keep track of each and every entries byte length.


    Nadeem

    Wednesday, December 18, 2013 3:46 AM
  • This has nothing to do with marshaling, marshaling is not used when dealing with memory mapped files. That's exactly the reason you can't use a type that contains references like MyData.

    The only way to get that data into a memory mapped file is to manually write its fields:

    accessor.Write(0, ref data.myInt); 
    accessor.WriteArray(4, data.myPt, 0, data.myPt.Length);
    

    Well, actually there's a way to use marshaling too but that would probably defeat your purpose, to be efficient. Try it though, maybe it's good enough:

    byte *ptr = null;
    try {
        accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        Marshal.StructureToPtr(value, (IntPtr)ptr, false);
    }
    finally {
        if (ptr != null)
            accessor.SafeMemoryMappedViewHandle.ReleasePointer();
    }
    
    This is for the set accessor of your Data property. The get accessor will need similar code but with Mar

    shal.PtrToStructure instead. Though if the array size isn't const like in your example I'm not sure how this could work...

    Wednesday, December 18, 2013 1:03 PM
    Moderator
  • Thanks ! I will give it a try and give update if I ran into some problem.

    Nadeem

    Thursday, December 19, 2013 3:08 PM