none
Why is DateTime not blittable?

    Question

  • I'm a little unclear on something.

    Decimal and DateTime are just structs, the Decimal type contains four Int32 fields, and because these are blittable a Decimal is indeed blittable.

    DateTime is a struct too, it contains a single UInt64 field, because this is blittable I'd expect DateTime to be blittable but it isn't.

    My understanding is that any struct or class that has LayoutKind.Sequential AND contains ONLY blittable fields, is itself blittable.

    My research into this suggests that DateTime, although it is a struct, is not formatted with LayoutKind.Sequential and this is why it isn't blittable. It seems to have been defined with Auto layout and this is the reason it isn't blittable.

    However, I don't know how or why this was done by MS, the C# language rules say that ALL structs have LayoutKind.Sequential applied BY DEFAULT.

    Does anyone know more about this odd behaviour from DateTime?

    Incidentally, you can test for blittable easily by just calling:

    GCHandle pinStructure = GCHandle.Alloc(item, GCHandleType.Pinned);

    If no exception is thrown, then the type of 'item' is a blittable type.

    I suspect that they deliberately made DateTime non-blittable to avoid confusion with existing Win32 date time structs, making DateTime blittable migh lead people to believe they can marshal it to one of these Win32 types.

    Cap'n

     

     

     

    Saturday, June 26, 2010 8:44 PM

All replies

  • Hi,

       Thanks for your post. I use the .NET reflector to check the definitions of DateTime and Decimal struct as below:

       DateTime:

    [Serializable]
    public struct DateTime : IComparable, IFormattable, IConvertible, ISerializable, IComparable<DateTime>, IEquatable<DateTime>
    {
     // Fields
     private ulong dateData;
     private const string DateDataField = "dateData";
     private const int DatePartDay = 3;
     private const int DatePartDayOfYear = 1;
     private const int DatePartMonth = 2;
     private const int DatePartYear = 0;
     private const int DaysPer100Years = 0x8eac;
     private const int DaysPer400Years = 0x23ab1;
     private const int DaysPer4Years = 0x5b5;
     private const int DaysPerYear = 0x16d;
     private const int DaysTo10000 = 0x37b9db;
     private const int DaysTo1601 = 0x8eac4;
     private const int DaysTo1899 = 0xa9559;
     private static readonly int[] DaysToMonth365;
     private static readonly int[] DaysToMonth366;
     private const long DoubleDateOffset = 0x85103c0cb83c000L;
     private const long FileTimeOffset = 0x701ce1722770000L;
     private const ulong FlagsMask = 13835058055282163712L;
     private const ulong KindLocal = 9223372036854775808L;
     private const ulong KindLocalAmbiguousDst = 13835058055282163712L;
     private const int KindShift = 0x3e;
     private const ulong KindUnspecified = 0L;
     private const ulong KindUtc = 0x4000000000000000L;
     private const ulong LocalMask = 9223372036854775808L;
     private const long MaxMillis = 0x11efae44cb400L;
     internal const long MaxTicks = 0x2bca2875f4373fffL;
     public static readonly DateTime MaxValue;
     private const int MillisPerDay = 0x5265c00;
     private const int MillisPerHour = 0x36ee80;
     private const int MillisPerMinute = 0xea60;
     private const int MillisPerSecond = 0x3e8;
     internal const long MinTicks = 0L;
     public static readonly DateTime MinValue;
     private const double OADateMaxAsDouble = 2958466.0;
     private const double OADateMinAsDouble = -657435.0;
     private const long OADateMinAsTicks = 0x6efdddaec64000L;
     private const long TicksCeiling = 0x4000000000000000L;
     private const string TicksField = "ticks";
     private const ulong TicksMask = 0x3fffffffffffffffL;
     private const long TicksPerDay = 0xc92a69c000L;
     private const long TicksPerHour = 0x861c46800L;
     private const long TicksPerMillisecond = 0x2710L;
     private const long TicksPerMinute = 0x23c34600L;
     private const long TicksPerSecond = 0x989680L;
    
     // Methods
     .....}

     

       Decimal:

    [Serializable, StructLayout(LayoutKind.Sequential), ComVisible(true)]
    public struct Decimal : IFormattable, IComparable, IConvertible, IComparable<decimal>, IEquatable<decimal>
    {
     private const int SignMask = -2147483648;
     private const int ScaleMask = 0xff0000;
     private const int ScaleShift = 0x10;
     private const int MaxInt32Scale = 9;
     [DecimalConstant(0, 0, (uint) 0, (uint) 0, (uint) 0)]
     public static readonly decimal Zero;
     [DecimalConstant(0, 0, (uint) 0, (uint) 0, (uint) 1)]
     public static readonly decimal One;
     [DecimalConstant(0, 0x80, (uint) 0, (uint) 0, (uint) 1)]
     public static readonly decimal MinusOne;
     [DecimalConstant(0, 0, uint.MaxValue, uint.MaxValue, uint.MaxValue)]
     public static readonly decimal MaxValue;
     [DecimalConstant(0, 0x80, uint.MaxValue, uint.MaxValue, uint.MaxValue)]
     public static readonly decimal MinValue;
     private static uint[] Powers10;
     private int flags;
     private int hi;
     private int lo;
     private int mid;
     //Methods
     ...
    }

       As we can see, Decimal type doesn't only contain four Int32 fields as it shown in the .NET reflector. And DateTime doesn't only contain a single UInt64 field either. But all the fields in the Decimal type are blittable, while DateTime's are not.

        DateTime contains several String type fields. According to here, String is non-blittable. So I think this is the reason DateTime is not blittable. Hope this could help you through this.

     

     


    Please mark the right answer at right time.
    Thanks,
    Sam
    • Edited by SamAgain Monday, June 28, 2010 7:31 AM refine
    Monday, June 28, 2010 7:23 AM
  • > I suspect that they deliberately made DateTime non-blittable to avoid confusion with existing Win32 date time structs

    I think that's a sensible line of reasoning.  To be blittable, it should have a clean, documented representation that makes sense in the context of the existing unmanaged Windows APIs.  On the contrary, DateTime's internal representation is a tick count along with "kind" bits that tell whether it is DateTimeKind.Utc, Local, or Unspecified.  Even more intriguing, for DateTimeKind.Local there is a representation of whether it is a daylight saving ambiguous time.  These are all .NET implementation details.

     

     

    • Proposed as answer by SamAgain Friday, July 02, 2010 3:15 AM
    Monday, June 28, 2010 11:13 PM
  • Be Careful Sam !

    All those const fields are technically static fields, so they don't appear as part of the memory used by an instance of the struct and therefore do not affect its "blittability" so to speak.

    The only instance fields are the one's I mentioned, and every instance field in both Decimal and DateTime is blittable.

    If you use reflector again, you will see that Decimal has the [StructLayout(LayoutKind.Sequential)] attribute BUT DateTime does not.

    DateTime appears as "auto" in Red Gate's reflector but Decimal appears as "sequential".

    It is the "auto" that makes DateTime non blittable I think, BinaryCoder seems to agree with this (by which I mean, he didn't disagree).

    Can you tell me what reflector tool you used? because in your listing Decimal has an explicit [StructLayout(LayoutKind.Sequential)] attribute, but DateTime doesn't have any attributes, this puzzles me because for a struct the default is "sequential" when no attribute is supplied.

    Cap'n

     

     

    Tuesday, June 29, 2010 12:52 AM
  • Hi Captain,

       Thanks for your pointing out. I am not sure about the reason. But I cannot reject your explanation, either. I am using the RedGate .NET Reflector 6.1.0.11. I double check the reflect result of DateTime, and it does have only one attribute [Serializable].


    Please mark the right answer at right time.
    Thanks,
    Sam
    Tuesday, June 29, 2010 7:38 AM
  • Hi Captain,

        Just a second thought, even the static or const fields in the DateTime struct don't appear in the memory for an instance of DateTime in managed code, does that mean that they are irrelevant when blitting is concerned?


    Please mark the right answer at right time.
    Thanks,
    Sam
    Friday, July 02, 2010 3:14 AM
  • Hi Captain,

        Just a second thought, even the static or const fields in the DateTime struct don't appear in the memory for an instance of DateTime in managed code, does that mean that they are irrelevant when blitting is concerned?


    Please mark the right answer at right time.
    Thanks,
    Sam


    So far as I am aware this is true. per/type fields like statics or consts (but NOT readonly, this is per/instance) are physically stored in the underlying type descriptor, not in the memory block used for instance fields.

    A blittable struct is a struct that specifically is guaranteed not be rearranged when manipulated by Marshal.StructureToPtr or Marshal.PtrToStructure.

    That's all it means, the term "blittable" only has meaning within the context of the .net marshaler.

    What has puzzled me is HOW DateTime is marked as having layout auto, there seems to be no attribute for it and the default layout for stucts is always sequential, so in the absence of any explicit layout attribute it really should be blittable.

    Cap'n

     

     

    Friday, July 02, 2010 12:07 PM
  • You are correct.  For a struct, the C# compiler will automatically use sequential.  Thus, for DateTime, they must have wrote [StructLayout(LayoutKind.Auto)] in the code.  (See how this is done in Mono's implementation of DateTime: http://www.koders.com/csharp/fidB9B10CF238378B924E027B47A8C08E2793CCF0D1.aspx)

    This looks like a bug in Reflector that it does not show this StructLayoutKind.Auto in its C# mode.  Switch to IL disassembly to see what is really going on.  We see that for DateTime the MSIL keyword "auto" appears and for Decimal the MSIL keyword "sequential" appears.

     

    Saturday, July 03, 2010 12:35 AM
  • You are correct.  For a struct, the C# compiler will automatically use sequential.  Thus, for DateTime, they must have wrote [StructLayout(LayoutKind.Auto)] in the code.  (See how this is done in Mono's implementation of DateTime: http://www.koders.com/csharp/fidB9B10CF238378B924E027B47A8C08E2793CCF0D1.aspx)

    This looks like a bug in Reflector that it does not show this StructLayoutKind.Auto in its C# mode.  Switch to IL disassembly to see what is really going on.  We see that for DateTime the MSIL keyword "auto" appears and for Decimal the MSIL keyword "sequential" appears.

     


    Yes, that's true I noticed these in the assembly code too, I think you are correct, this is a small bug in Reflector. However, the Mono implementation is surely just a "copy" of the MS one? I doubt the designers said "Hmm, this DateTime struct really should be layout=auto". By which I mean, I can see no functional reason for this (other than preventing user misunderstandings about how DateTime maps to unmanaged Win32 structs).

     

    Thanks

    Cap'n

     

    Saturday, July 03, 2010 2:09 PM
  • Don't read too much into it.  (If you actually need to blit DateTime, your design needs revised!)  We're not likely to get the DateTime developer to jump in on this thread and tell us the exact reasoning.  My guess is that the developer was familiar with StructLayoutKind, realized that Sequential was not required by this struct because it is not intended for any interop scenarios, and thus stamped StructLayoutKind.Auto onto it.

     

    Tuesday, July 06, 2010 11:58 PM
  • Don't read too much into it.  (If you actually need to blit DateTime, your design needs revised!)  We're not likely to get the DateTime developer to jump in on this thread and tell us the exact reasoning.  My guess is that the developer was familiar with StructLayoutKind, realized that Sequential was not required by this struct because it is not intended for any interop scenarios, and thus stamped StructLayoutKind.Auto onto it.

     


    Yes I think this is what happened. The reason the issue came up though is that it misleads people into thinking the struct is not blittable because of its field types, most people would assume this and move on. I needed to verify this and then discovered that it was the attribute.

    The lessson here is that one can't assume a non-blittable struct/class contains any non-blittable fields.

    Cap'n

     

    Friday, July 09, 2010 2:29 PM
  • So what is the best way to do a GCHandle.Alloc on a DateTime array?

     

    thanks


    w.
    Wednesday, April 13, 2011 4:55 PM
  • So what is the best way to do a GCHandle.Alloc on a DateTime array?

     

    thanks


    w.


    Can you explain this a bit more?

    Are you seeking a way to get the address of an array of DateTimes in managed code?

    Cap'n

     


    Wednesday, April 13, 2011 5:01 PM
  • This is the kind of scenario where I need some huge arrays of DateTime and double and compute data on it then release them and do again with another set, the "well design" approach are way to slow compare to what I have right now, problem is time to time the array is move and so my pointer are lost....

    Note: I'm using c# as the app has a lot of different part that made sens in c#, so convenient and fast to dev. Also c# seem to handle as well and sometime better huge amout of memory than c++ as few articles mention
    http://social.msdn.microsoft.com/forums/en-US/csharplanguage/thread/0990061a-0618-495f-81da-48badf4db2fb/

     

    I have been able to improve perf by a lot by using pointers and other tricks that are not necessarely "well design" but waiting 1 min instead of 3h to do the same computation make me not care if some isolated piece of the design are not that "well design"

    So, I have huge arrays of date and double that I do computation on, after sometime I realize I increase the speed a lot by using simple normal arrays and using pointers on them (so I pin it once and then move the pointers, calling fixed everyti medefy the purpose of pointers to my app), but of course when object goes in Gen 2 they are compacted and move so my pointers are not good anymore, so I'm looking at different solution so I can rely on the pointers, I can see few direction but I'm not sure yet what will work:

    - GC.KeepAlive
    - or Having some king of event feedback from GC so I know when the big arrays have been move and so I can set the pointers again
    - Is there a way to know when the pointer doesn't point to a managed object anymore?
    - I feel I can solve this issue for double with GCHandle.Alloc, but what do I do for dateTime? I will guess the date to long conversion will be too costly? I can't use GCHandle.Alloc with DateTime as DateTime is un-blittable.
    - Is there a way to tell the GC to not compact the memory?
    - Is there a way to not allow the GC to promote a long live object as Gen 2?
    - I could of course create some large object so they are not compacted. but the definition might change thought time, so I need to make sure it is pin....(The large object heap contains objects that are 85,000 bytes and larger. Very large objects on the large object heap are usually arrays. It is rare for an instance object to be extremely large. )

    Thanks a lot for your help

    w

     


    w.
    Wednesday, April 13, 2011 5:25 PM
  • This is the kind of scenario where I need some huge arrays of DateTime and double and compute data on it then release them and do again with another set, the "well design" approach are way to slow compare to what I have right now, problem is time to time the array is move and so my pointer are lost....

    Note: I'm using c# as the app has a lot of different part that made sens in c#, so convenient and fast to dev. Also c# seem to handle as well and sometime better huge amout of memory than c++ as few articles mention
    http://social.msdn.microsoft.com/forums/en-US/csharplanguage/thread/0990061a-0618-495f-81da-48badf4db2fb/

     

    I have been able to improve perf by a lot by using pointers and other tricks that are not necessarely "well design" but waiting 1 min instead of 3h to do the same computation make me not care if some isolated piece of the design are not that "well design"

    So, I have huge arrays of date and double that I do computation on, after sometime I realize I increase the speed a lot by using simple normal arrays and using pointers on them (so I pin it once and then move the pointers, calling fixed everyti medefy the purpose of pointers to my app), but of course when object goes in Gen 2 they are compacted and move so my pointers are not good anymore, so I'm looking at different solution so I can rely on the pointers, I can see few direction but I'm not sure yet what will work:

    - GC.KeepAlive
    - or Having some king of event feedback from GC so I know when the big arrays have been move and so I can set the pointers again
    - Is there a way to know when the pointer doesn't point to a managed object anymore?
    - I feel I can solve this issue for double with GCHandle.Alloc, but what do I do for dateTime? I will guess the date to long conversion will be too costly? I can't use GCHandle.Alloc with DateTime as DateTime is un-blittable.
    - Is there a way to tell the GC to not compact the memory?
    - Is there a way to not allow the GC to promote a long live object as Gen 2?
    - I could of course create some large object so they are not compacted. but the definition might change thought time, so I need to make sure it is pin....(The large object heap contains objects that are 85,000 bytes and larger. Very large objects on the large object heap are usually arrays. It is rare for an instance object to be extremely large. )

    Thanks a lot for your help

    w

     


    w.


    Hi, thanks for explaining the problem.

    This does sound interesting, and I tell you my immediate ideas here.

    First, don't store "DateTime" instances, all DateTime actually is a a long and a load of methods/properties, so I would store arrays of long and doubles.

    Next if the "date time" and doubles are always paired, then define a new struct that contains a long (date time) and double, call this new type DateDouble (for the sake of argument).

    Next add a property to the new struct that converts to/from DateTime and "hide" the long.

    This struct is blittable, and getting a pointer to it is easy in C#.

    Next I would allocate a chunk of memory from the unmanaged heap that can store the array you need (of these new structs) and get a pointer to that.

    Copy that pointer to a pointer of type DateDouble * dd_ptr;

    Now you can simply index that pointer: dd_ptr[0], dd_ptr[23] etc etc.

    The GC will never get involved and the allocated block of memory is (to all intents and purposes) just like a large static array.

    I would do this and I'm very confident it would be very fast indeed and eliminate all GC worries, I've done stuff more complicated than this myself several times and had great success.

     

    Cap'n

     

     


    • Edited by Captain Kernel Wednesday, April 13, 2011 7:12 PM clarify
    • Proposed as answer by wil70 Thursday, April 14, 2011 3:43 PM
    Wednesday, April 13, 2011 7:10 PM
  • Thanks a lot Cap'n, I will look into it, I was staying on DateTime thinking the long conversion may be too long as I probably use them billion of time and so I didn't want to pay the cost of conversion everytime and moving 2 ptrs (++ or --) is pretty fast.

    In your experience the long to datetime conversion (and vis versa) wasn't bad?

    Everything works well in my app expect I have few smaller arrays that are not that are not big enouight and so are not put in the LOH and so are compacted by GC Gen2 (if I increase their size it works really well, with the speed I need etc). By using long that will make those work with GCHandle.Alloc and so should work... I guess I will need to make some test to see what the speed really is.

    Thanks a lot again.

    w

     

    Wednesday, April 13, 2011 8:21 PM
  • Thanks a lot Cap'n, I will look into it, I was staying on DateTime thinking the long conversion may be too long as I probably use them billion of time and so I didn't want to pay the cost of conversion everytime and moving 2 ptrs (++ or --) is pretty fast.

    In your experience the long to datetime conversion (and vis versa) wasn't bad?

    Everything works well in my app expect I have few smaller arrays that are not that are not big enouight and so are not put in the LOH and so are compacted by GC Gen2 (if I increase their size it works really well, with the speed I need etc). By using long that will make those work with GCHandle.Alloc and so should work... I guess I will need to make some test to see what the speed really is.

    Thanks a lot again.

    w

     

    Hi, and no probs, I get a lot of help here and I'm glad to do my bit.

    Well since the DateTime struct itself is just a long with a load of operators, I wouldn't expect much of a hit by storing just the long and working with that, I've not done any benchmarking but never noticed any hotspots here.

    By creating your own unmanaged array of structs (using Marshal.AllocHGlobal) you have direct access to memory via a managed pointer, ensure that GC keep out of things, and minimize storage space used.

    I assume the arrays you create are 1D?

    Cap'n

     

     

    • Proposed as answer by wil70 Thursday, April 14, 2011 3:43 PM
    Wednesday, April 13, 2011 9:50 PM
  • yes (for now), how does that work in term of memory, when I do a new DateTime(aLong),

    Pitty I can not cast the long as DateTime or just make the DateTime tick point to a long (so I could have a Pool of empty DateTime)

    I will run some test

    thanks again

    w

     

     


    w.
    Wednesday, April 13, 2011 10:21 PM
  • Here is another solution I read somewhere else propose by viorel_

    "I order to pin the array, try this approach. In the constructor of Dates, start a thread. Inside the thread procedure (which is a member of Dates), execute ‘fixed (DateTime* daptr = &_dateTimeArray[0])’, then block the execution of the thread. For example use an AutoResetEvent and wait for a signal. In your main thread, in the constructor, make sure your thread has fixed the data (use another event), and then continue your job. When you want to unpin and free the data, set the AutoResetEvent in order to let the thread to exist from the fixed block. I hope this makes sense."

    Sounds pretty good also.


    w.
    Thursday, April 14, 2011 3:44 PM
  • Here is another solution I read somewhere else propose by viorel_

    "I order to pin the array, try this approach. In the constructor of Dates, start a thread. Inside the thread procedure (which is a member of Dates), execute ‘fixed (DateTime* daptr = &_dateTimeArray[0])’, then block the execution of the thread. For example use an AutoResetEvent and wait for a signal. In your main thread, in the constructor, make sure your thread has fixed the data (use another event), and then continue your job. When you want to unpin and free the data, set the AutoResetEvent in order to let the thread to exist from the fixed block. I hope this makes sense."

    Sounds pretty good also.


    w.

    This may "work", howver it is poor design in my opinion, such approaches should only be used if there is absolutely no other way.

    The additional complexity of synchronising the threads must be taken into account and you must consider how this design would behave if your code was ever used in a future multithreaded setting or extended by another developer.

    From what you say, I see no reason at all for storing the arrays in the managed heap, and the threading idea is actually just a way of dealing with the GC, if you go for the unmanaged heap you avoid the GC altogether and place less of a burden on the GC and reduce interference with the GC scheduling.

    Storing the array in the unmanaged heap will allow you to achieve the highest performance targets in my opinion.

     

    Cap'n

     

     

     

     

    Sunday, April 17, 2011 8:23 PM
  • It's funny but I guess TimeStamp is possible blittable!!

    Let me explain why I think so.

    We know than TimeStamp and DateTime is 8 byte, 64 bit integer. Measuring 100 ns ticks.

    Based on that I wrote unmanaged dll (Delphi) library (Dllimport, unsafe code) sharing the following structure in memory:

    [StructLayout(LayoutKind.Sequential, Pack = 8)] //Quad word align ! for default in Delphi 2010
        public struct  record_quote

        {

            public double Open, High, Low, Close, TrdPrice, BidPrice, AskPrice;
            public int TrdVolume, CumVolume, ID;
            public int TimeZoneOffset_sec;
            public TimeSpan TimeZoneOffset;
            public Int64 DT_GMT;

    }      

    Everething was ok (including TimeStamp!)

    then I added one line to the structure (DLL modified too)

            public DateTime DT;

    wow I'v got wrong values for all (!) fields like it was strange "shift" in data structure.

    So, obviously DateTime have unexpected behaviour ...

    But good news - probably you can use TimeStamp instead.

    Thx.

     

     

    Saturday, June 04, 2011 12:36 AM