none
Copy unmanaged data RRS feed

  • Question

  • I have following code:
    List<MyStruct> listMyStructs = new List<MyStruct>();
    for (int elOfInterest = 0; elOfInterest < nRecords; elOfInterest++)
    {
      IntPtr ptrElOfInterest = new IntPtr(ptrMyData.ToInt64() + Marshal.SizeOf(typeof(MyStruct)) * elOfInterest);
      MyStruct objStruct = ((MyStruct)(Marshal.PtrToStructure(ptrElOfInterest, typeof(MyStruct))));
    
      listMyStructs.Add(objStruct);
    }
    
    
    

    Here, ptrMyData --> is pointing to first record out of 'nRecords'. I am getting this data from unmanaged library. 

    "Above code works fine. That is, My list 'listMyStructs' gives me what I expect in it."

    Now, problem is, this operation takes more time as the 'nRecords' (number of records) increases. It is always having a possibility that number of records will grow more and more. i.e. Number of records can be in multiples of thousand.

    As my application is time critical. my job is to reduce this time somehow.

    Anybody having an idea how I can achieve this?


    Santosh.
    Thursday, January 27, 2011 7:07 AM

All replies

  • Can you please check which of the two statements take more time. Comment the lines individually and take the reading once again, so you can focus on that.
    Please Mark as Answered If this answers your question Or UnMark as Answered if it did not.
    Happy to Help :)
    My Blog
    Thursday, January 27, 2011 12:36 PM
  • Hi Santosh,

    You may use Stopwatch Class to monitor which statement costs a lot of time.


    Cookie Luo[MSFT]
    MSDN Community Support | Feedback to us
    Get or Request Code Sample from Microsoft
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Monday, January 31, 2011 3:58 AM
  • first of all, thank u very much for ur replies. I used query performance counter to check the time to excute each of the statement. And i found it is very less. If u look at my problem definition where i mentioned that as 'nRecords' increases, the time to excute 'for loop' changes in the same proportion. What i would like to know from u if there exists any other technique than one which i hv implemented will take less time to copy my unmanaged data. I know MSDN will not let me down to get rid of this time critical issue.
    Santosh.
    Monday, January 31, 2011 9:25 AM
  • Sorry, Adavesh as I'm replying so late. But, How could you make 'IntPtr' to 'Null' as it is a non-nullable type?
    Santosh.
    Tuesday, February 8, 2011 5:05 AM
  • If you're working with a large number of records then you will save a lot of time in that loop if you create the List<MyStruct> with an initial capacity that is large enough to avoid a lot of resizing.  Resizing Lists can be costly in performance-critical applications; particularly in tight-loops. 

    In your code you can check this by just setting the initial capacity to nRecords before the loop and seeing if there's a performance difference.  My guess is that it will be a noticeable improvement.

    ShaneB

    Tuesday, February 8, 2011 5:28 AM
  • Hi Shane,

    Thanks for you reply. I checked my code and measured time.

    1. Setting 'Capacity' for 'List'.(Your suggestion)

    25.57 seconds.

    2. Without Setting 'Capacity'. (My older implementation)

    25.67 seconds.

    Hence, I don't think it served my purpose.

    Am I doing anything wrong in the for loop? Or is there any other way to do this very efficiently considering time criticality?


    Santosh.
    Tuesday, February 8, 2011 7:00 AM
  • Hmmm...it's difficult to say without knowing more about the program. 

    Can you elaborate on what type of data it is?  Specifically:

    1) Does it change often? 

    2) How many records are you typically getting from the unmanaged library? 

    3) How is MyStruct defined? 

    4) Can the unmanaged data be cached into managed memory before your function? 

    I'm just thinking there may be a way to avoid marshalling in 1 struct at a time. 

    Any more info you can give would be helpful.  If it's a publicly-available unmanaged library, maybe I can even duplicate the problem on my machine which would help even more.

    ShaneB

    Tuesday, February 8, 2011 2:49 PM
  • Shane,

    Precisely I would define my problem as:

    Unfortunately, the unmanaged library I'm using is not publicly available and I do not redefine/change the same.

    This library allows me to get a binary data/stream(byte array) from it.

    I will answer your questions as:

    type of data-->

    1) Does it change often? 

    ->Yes, for every run of my app I see a different data. I mean it changes very often.

    2) How many records are you typically getting from the unmanaged library? 

    ->The number of records are in multiples of thousands.

    3) How is MyStruct defined? 

    -> MyStruct { Int32 var1; Int var2; Int var3;}

    4) Can the unmanaged data be cached into managed memory before your function? 

    -> I can't capture this unmanaged data before my function. As my design doesn't allow me to do so. It has to happen in this function only.

    The unmanaged library has a function like:

    bool UnmanagedFunction(BYTE *data);

    The Managed function:

    bool ManagedFunction(IntPtr ptrMyData)

    {

    NativeMethods.UnmanagedFunction(ptrMyData);

    }

    Now, I have this:

    List<MyStruct> listMyStructs = new List<MyStruct>();
    for (int elOfInterest = 0; elOfInterest < nRecords; elOfInterest++)
    {
     IntPtr ptrElOfInterest = new IntPtr(ptrMyData.ToInt64() + Marshal.SizeOf(typeof(MyStruct)) * elOfInterest);
     MyStruct objStruct = ((MyStruct)(Marshal.PtrToStructure(ptrElOfInterest, typeof(MyStruct))));
    
     listMyStructs.Add(objStruct);
    }

    Let me know if you still need more info as I want to solve this problem as early as possible.
    At the same time let me also know if I'm doing anything wrong. :)



    Santosh.
    Wednesday, February 9, 2011 4:54 AM
  • I threw together a quick test to see if I could duplicate the performance issue and my results were very different.  It took approximately 1/2 a second (on a slow netbook) on average to load 500K structs from an unmanaged memory pointer into the list. 

    My code is as follows:

     

    struct MyStruct 
    { 
      int var1; 
      int var2; 
      int var3;
    }
    
    ...
    
    void Test()
    {
      int nRecords = 500000;
      IntPtr ptrMyData = Marshal.AllocHGlobal(nRecords * Marshal.SizeOf(typeof(MyStruct)));
      try
      {
        List<MyStruct> listMyStructs = new List<MyStruct>();
        for (int elOfInterest = 0; elOfInterest < nRecords; elOfInterest++)
        {
          IntPtr ptrElOfInterest = new IntPtr(ptrMyData.ToInt64() + Marshal.SizeOf(typeof(MyStruct)) * elOfInterest);
          MyStruct objStruct = ((MyStruct)(Marshal.PtrToStructure(ptrElOfInterest, typeof(MyStruct))));
    
          listMyStructs.Add(objStruct);
        }
      }
      finally
      {
        Marshal.FreeHGlobal(ptrMyData);
      }
    }
    

     

    Unfortunately it's difficult to help more without being able to duplicate the issue.  You could try copying the buffer given to you by the dll all at once to a managed int[] (the struct has only Int32s right?) using one of the Marshal.Copy() overloads and then loop through the managed array.  I suspect that will be faster but who knows. 

    The only other option I can see to improve performance would be to use unsafe code...

    HTH,

    ShaneB

     


    Thursday, February 10, 2011 1:05 AM
  •   /// <summary>
      /// This Class Represents 'My Structure'.
      /// </summary>
      [StructLayout(LayoutKind.Sequential)]
      public class MyStruct : IEquatable<MyStruct>
      {
        #region Data members
        /// <summary> Integer 1</summary>
        private Int32 integerVar1;
        /// <summary> Integer 2</summary>
        private Int32 integerVar2;
        /// <summary> Integer 3</summary>
        private Int32 integerVar3;
        /// <summary>Unsigned Integer</summary>
        private UInt64 unsignedIntVar;
        /// <summary>A byte</summary>
        private byte byteVar; 
        #endregion
    
        #region Constructors
        /// <summary> Default Constructor </summary>
        public MyStruct() { }
    
        public MyStruct(Int32 integerVar1, Int32 integerVar2, Int32 integerVar3, UInt64 unsignedIntVar, byte byteVar)
        {
          this.integerVar1 = integerVar1;
          this.integerVar2 = integerVar2;
          this.integerVar3 = integerVar3;
          this.unsignedIntVar = unsignedIntVar;
          this.byteVar = byteVar;
        } 
        #endregion
    
        #region IEquatable<MyStruct> Members
    
        /// <summary>
        /// Returns the hash code for this instance.
        /// </summary>
        /// <returns> Hash Code </returns>
        public override int GetHashCode()
        {
          return (
            this.integerVar1.GetHashCode() ^
            this.integerVar2.GetHashCode() ^
            this.integerVar3.GetHashCode() ^
            this.unsignedIntVar.GetHashCode() ^
            this.byteVar.GetHashCode()
            );
        }
    
        /// <summary>
        /// Returns a value indicating whether this is instance
        /// is equal to a specified object.
        /// </summary>
        /// <param name="obj"> Object to to check equality </param>
        /// <returns> True: If the instances are equal; False: If instances doesn't match. </returns>
        public override bool Equals(object obj)
        {
          MyStruct myStruct = obj as MyStruct;
          if (null == myStruct)
            return false;
    
          return Equals(myStruct);
        }
    
        /// <summary>
        /// Returns a value indicating whether this is instance
        /// is equal to specified 'MyStruct' instance.
        /// </summary>
        /// <param name="other"> Instance of 'MyStruct' to check equality </param>
        /// <returns> True: If the instances are equal; False: If instances doesn't match. </returns>
        public bool Equals(MyStruct other)
        {
          if (integerVar1 != other.integerVar1)
            return false;
    
          return integerVar1 == other.integerVar1;
        }
    
        /// <summary>
        /// Returns a value indicating whether the specified objects are equal.
        /// </summary>
        /// <param name="indexTableItem1"> Instance 1st of 'MyStruct' </param>
        /// <param name="indexTableItem2"> Instance 2nd of 'MyStruct' </param>
        /// <returns> True: If the instances are equal; False: If instances doesn't match. </returns>
        public static bool operator ==(MyStruct myStruct1, MyStruct myStruct2)
        {
          return myStruct1.Equals(myStruct2);
        }
    
        /// <summary>
        /// Returns a value indicating non-equality of the specified objects, if any.
        /// </summary>
        /// <param name="indexTableItem1"> Instance 1st of 'MyStruct' </param>
        /// <param name="indexTableItem2"> Instance 2nd of 'MyStruct' </param>
        /// <returns></returns>
        public static bool operator !=(MyStruct myStruct1, MyStruct myStruct2)
        {
          return !myStruct1.Equals(myStruct2);
        }
    
        #endregion
    
        #region Properties
        /// <summary>
        /// Integer 1
        /// </summary>
        public Int32 IntegerVar1
        {
          set { integerVar1 = value; }
          get { return integerVar1; }
        }
    
        /// <summary>
        /// Integer 2 
        /// </summary>
        public Int32 IntegerVar2
        {
          set { integerVar2 = value; }
          get { return integerVar2; }
        }
    
        /// <summary>
        /// Integer 3
        /// </summary>
        public Int32 IntegerVar3
        {
          set { integerVar3 = value; }
          get { return integerVar3; }
        }
    
        /// <summary>
        /// Unsigned Integer
        /// </summary>
        public UInt64 UnsignedIntVar
        {
          set { unsignedIntVar = value; }
          get { return unsignedIntVar; }
        }
    
        /// <summary>
        /// A Byte Variable
        /// </summary>
        public byte ByteVar
        {
          set { byteVar = value; }
          get { return byteVar; }
        } 
        #endregion
    
      }
    

    I have extended 'MyStruct' as above. Hence, I think i have left with only choice to use 'unsafe code'. Actually I already had a thought of it and trying to avoid the same :). 

    If you still find anything better solution to this. i'll be grateful to you.

    Thanks.


    Santosh.
    Thursday, February 10, 2011 6:55 AM
  • As a side note, I suggest you consider rewriting your GetHashCode() function.  As it stands now, it will return the same value when it shouldn't because the order of variables isn't taken into account.

    For example, if you have:

    a = 1, b = 2, c = 3

    your implementation (which is effectively a ^ b ^ c) will return the same hash code as:

    a = 3, b = 2, c = 1

    At the risk of pointing out the obvious, Int32.GetHashCode() and byte.GetHashCode() just return the actual variable's value.  UInt64.GetHashCode() of course has to actually calculate a value in order to return an integer. 

    Anyway, I'm no hash algorithm expert but I wanted to point out that you may run into problems with what you currently have.

    ShaneB

    Thursday, February 10, 2011 10:11 PM
  • Thank you very much for your suggestion. :)
    Santosh.
    Friday, February 11, 2011 4:31 AM