none
BitConverter - How to write a data type into a pre-allocated byte array? RRS feed

  • Question

  • I currently convert a data type value into a byte array by using BitConverter.GetBytes().

    Each GetBytes-call allocates a tiny byte-array that I immediately copy to a bigger byte array. The tiny byte-arrays are just temporary garbage collected by the GC.

    How can I avoid producing these tiny byte arrays?

    I am looking for a counterpart to e.g. BitConverter.ToSingle that writes directly into an existing byte array.

    I also noticed that GUID, IPAddress and other basic classes use the same API pattern; a serialized value of one of these data types comes via temporary byte array. Any way to avoid this?

    This might not be a problem for most applications, but in a game less garbage means a lower GC run frequency and each GC run results in a major frame drop.

    Kind Regards,

    Keld Ølykke

    Friday, February 22, 2013 10:36 AM

Answers

  • In case of Guid, it is possible to write bytes in this manner (supposing the output array contains enough space):

    public void ToBytes(Guid g, byte[] output, ref int outputOffset)

    {

           unsafe

           {

                  fixed (byte* b = &output[outputOffset])

                  {

                         IntPtr p = new IntPtr(b);

                         Marshal.StructureToPtr(g, p, false);

                         outputOffset += Marshal.SizeOf(g);

                  }

           }

    }


    • Edited by Viorel_MVP Friday, February 22, 2013 1:29 PM
    • Proposed as answer by Mike FengModerator Tuesday, February 26, 2013 9:36 AM
    • Marked as answer by KeldØlykke Wednesday, February 27, 2013 6:35 AM
    Friday, February 22, 2013 1:28 PM

All replies

  • Sometimes it is possible to use pointers. Which data type are you converting to bytes using BitConverter.GetBytes()?

    Friday, February 22, 2013 12:28 PM
  • The Getbyte() method has multiple overrides.  The default is to put the data at offset 0 in the byte[] but you can specify the start positiion and the length.

    You may want to use a memorystream which uses an expandable byte array which will allocate more memory as you add data.  For the memorystream to expand you must use the constructor and not specify the size of the stream.


    jdweng

    Friday, February 22, 2013 12:37 PM
  • I have made my own 2's complement serialization of short, int and long.

    I haven't tried to do something similar with float and double -  I guess it is possible, but I feel I shouldn't be doing it.

    DateTime and TimeStamp basically can be serialized with a long.

    I find GUID and IPAddress (and thus IPEndPoint) out of my reach, since I basically can't reach their internal format without producing temporary byte arrays.

    Kind Regards,

    Keld Ølykke

    Friday, February 22, 2013 12:47 PM
  • The Getbyte() method has multiple overrides.  The default is to put the data at offset 0 in the byte[] but you can specify the start positiion and the length.

    You may want to use a memorystream which uses an expandable byte array which will allocate more memory as you add data.  For the memorystream to expand you must use the constructor and not specify the size of the stream.


    jdweng

    Do you refer to the GetByte() in Buffer? Sounds like you have a class and method that I haven't.

    Kind Regards,

    Keld Ølykke

    Friday, February 22, 2013 12:54 PM
  • Why not use a structure or a class?  Last year I wrote functions to convert from byte[8] to float and double; and the reverse using the IEEE standard.  We have a database where all the data was being stored with a identify string name and 64 bits (8 bytes) value.  When extracting the the float and double numbers I had to convert  the bytes to float/double before writing the numbers to string.

        class ReceiveData
        {
            public int firstInteger { get; set; }
            public int secondInteger { get; set; }
            public long thirdLong { get; set; }
            public float fourthFloat { get; set; }
            public double fifthDouble { get; set; }
        }


    jdweng

    Friday, February 22, 2013 1:01 PM
  • I don't really want to make my own struct version of all future data types. Furthermore, I don't see I can with some data types e.g. GUID. 

    I basically just want to be able to add more pairs of functions to our serializer/deserializer class:

           /// <summary>
            /// Serializes a float.
            /// </summary>
            /// <param name="input">input to serialize</param>
            /// <param name="output">existing byte array to serialize to, or null if new array should be created</param>
            /// <param name="outputOffset">offset in existing byte array, or 0</param>
            /// <returns>new array, if output is null, otherwise null</returns>
            [SuppressMessage("Microsoft.Design", "CA1045")] // all the reference parameters
            byte[] ToBytes(float input, ref byte[] output, ref int outputOffset);
    
            /// <summary>
            /// Deserializes a float from bytes. 
            /// </summary>
            /// <param name="input">input byte array to deserialize from</param>
            /// <param name="inputOffset">offset in input to deserialize from</param>
            /// <param name="outputBytesRead">variable that will be incremented with bytes read</param>
            /// <returns>the deserialized new object</returns>
            [SuppressMessage("Microsoft.Naming", "CA1720")] // yes, method name includes a primitive name
            [SuppressMessage("Microsoft.Design", "CA1045")] // all the reference parameters
            float ToFloat(byte[] input, int inputOffset, ref int outputBytesRead);

    and I don't want these functions to allocate temporary byte buffers.

    Kind Regards,

    Keld Ølykke


    • Edited by KeldØlykke Friday, February 22, 2013 1:11 PM
    Friday, February 22, 2013 1:10 PM
  • In case of Guid, it is possible to write bytes in this manner (supposing the output array contains enough space):

    public void ToBytes(Guid g, byte[] output, ref int outputOffset)

    {

           unsafe

           {

                  fixed (byte* b = &output[outputOffset])

                  {

                         IntPtr p = new IntPtr(b);

                         Marshal.StructureToPtr(g, p, false);

                         outputOffset += Marshal.SizeOf(g);

                  }

           }

    }


    • Edited by Viorel_MVP Friday, February 22, 2013 1:29 PM
    • Proposed as answer by Mike FengModerator Tuesday, February 26, 2013 9:36 AM
    • Marked as answer by KeldØlykke Wednesday, February 27, 2013 6:35 AM
    Friday, February 22, 2013 1:28 PM
  • That is interesting... a bit hacky ... I'll have a whack at it.

    I have unit tests, so I can verify the results against the GetBytes() implementation.

    Thx

    Kind Regards,

    Keld Ølykke

    Friday, February 22, 2013 1:34 PM
  • That is interesting... a bit hacky ... I'll have a whack at it.


    It's not hacky - look inside BitConvertor with Reflector - that's basically what it's doing.

    -cd Mark the best replies as answers!

    Friday, February 22, 2013 3:59 PM
    Moderator
  • Hi Keld,

    Do you have any update?

    Best regards,


    Mike Feng
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Tuesday, February 26, 2013 9:36 AM
    Moderator
  • This seems to be the only solution for a Guid. 

    I still think it is hacky, since - as far as I can tell - this approach relies on the fields within the Guid class. If some poor guy got away with adding another private non-static field to the Guid structure, all code in the world using this approach would start writing more than 16 bytes. 

    In this case I recon it might be ok, but I still miss a proper public api pattern for this datatype-to-bytearray service.

    Thanks for your answer Viorel_


    • Edited by KeldØlykke Wednesday, February 27, 2013 6:43 AM
    Wednesday, February 27, 2013 6:42 AM
  • Thx Carl,

    I just reflected BitConverter and that was very good call from you. Thanks.

    Obviously, I don't know the IL assumptions for this kind of code to work, but this way of poking in memory is new to me.

    As an example:

    [__DynamicallyInvokable, SecuritySafeCritical]
    public unsafe static byte[] GetBytes(float value)
    {
        return BitConverter.GetBytes(*(int*)(&value));
    }

    Thanks for pointing me in this direction. Still think this is a hacky approach that could do with an api pattern instead.

    Kind Regards,

    Keld Ølykke

    Wednesday, February 27, 2013 6:48 AM
  • Hi Mike,

    Yes. Think this is as far that this thread can go. Thanks for heads up :)


    • Edited by KeldØlykke Wednesday, February 27, 2013 6:50 AM
    Wednesday, February 27, 2013 6:49 AM
  • Thanks for pointing me in this direction. Still think this is a hacky approach that could do with an api pattern instead.

    No question of that!  The BitConvertor class should be upgraded with overloads on GeBytes that will write to specific offsets in an existing array.  I'd suggest posting this suggestion on connect or uservoice.

    -cd Mark the best replies as answers!

    Wednesday, February 27, 2013 3:36 PM
    Moderator