locked
Structure Array To Byte Array And Vice Versa RRS feed

  • Question

  • Is there any fast method to convert an array of structures to a byte array.

    I have to convert millions of structures(e.g single,short etc).

    It takes too much time to save every structure separately.


    Allow time to reverse.

    Thursday, February 19, 2015 5:05 PM

Answers

  • If you mark your structure with <Serialisable> attribute, then you can serialise the whole array to a MemoryStream using BinaryFormatter, then obtain the bytes from MemoryStream (or serialise directly to file, database, etc.). Then you can deserialise the array.

    It is also possible to write a portion of code in C# or C++/CLR, where you can obtain the bytes easier.


    • Edited by Viorel_MVP Thursday, February 19, 2015 7:09 PM
    • Proposed as answer by IronRazerz Thursday, February 19, 2015 7:22 PM
    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 7:07 PM
  • Hi,

     Viorel beat me to it but, i was thinking the same thing. Mark your structure as Serializable and use the BinaryFormatter class to Serialize the whole list to a MemoryStream and then you can Deserialize back to a List of your structure.

    Here is a short basic example i just threw together to test it and it seems to work pretty good.

    Imports System.Runtime.Serialization.Formatters.Binary
    
    Public Class Form1
        Private Byts() As Byte
    
        <Serializable()> _
        Private Structure MyStruct
            Public str As String
            Public int As Integer
            Public dbl As Double
            Public sngl As Single
            Public shrt As Short
        End Structure
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
            Dim sList As New List(Of MyStruct)
    
            Dim s As New MyStruct
            s.str = "First"
            s.int = 50
            s.dbl = 33.523
            s.sngl = 12.33343
            s.shrt = 255
            sList.Add(s)
    
            s = New MyStruct
            s.str = "Second"
            s.int = 23
            s.dbl = 72.881
            s.sngl = 1.003
            s.shrt = 64
            sList.Add(s)
    
            Byts = GetBytesFromStructureList(sList)
        End Sub
    
        Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click
            Dim sList As List(Of MyStruct) = GetStructureListFromBytes(Byts)
    
            For Each s As MyStruct In sList
                RichTextBox1.AppendText(s.str & vbNewLine)
                RichTextBox1.AppendText(s.int.ToString & vbNewLine)
                RichTextBox1.AppendText(s.dbl.ToString & vbNewLine)
                RichTextBox1.AppendText(s.sngl.ToString & vbNewLine)
                RichTextBox1.AppendText(s.shrt.ToString & vbNewLine)
                RichTextBox1.AppendText("---------------------" & vbNewLine)
            Next
        End Sub
    
        Private Function GetBytesFromStructureList(ByVal structList As List(Of MyStruct)) As Byte()
            Dim bts() As Byte
            Using ms As New IO.MemoryStream
                Dim formatter As New BinaryFormatter
                formatter.Serialize(ms, structList)
                bts = ms.GetBuffer
            End Using
            Return bts
        End Function
    
        Private Function GetStructureListFromBytes(ByVal bytesArray() As Byte) As List(Of MyStruct)
            Dim lst As New List(Of MyStruct)
            Using ms2 As New IO.MemoryStream(bytesArray)
                Dim formatter As New BinaryFormatter
                lst = CType(formatter.Deserialize(ms2), List(Of MyStruct))
            End Using
            Return lst
        End Function
    End Class


    If you say it can`t be done then i`ll try it

    • Edited by IronRazerz Thursday, February 19, 2015 7:30 PM
    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 7:22 PM
  • Another one:

    Imports System.Runtime.InteropServices
    
    Public Class Main
    
       'Make sure the struct only contains value types and the StructLayout attribute is attached. No prob if just Single() or Integer().
    
       <StructLayout(LayoutKind.Sequential)> _
       Public Structure OpenDMLStandardIndexEntry
          Public Offset As UInteger
          Public Size As UInteger
       End Structure
    
       Public Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (ByVal Destination As IntPtr, ByVal Source As IntPtr, ByVal Length As IntPtr)
    
       Shared Sub Main()
    
          Dim Entries(3) As OpenDMLStandardIndexEntry
          Dim Bytes = GetBytes(Entries)
    
       End Sub
    
       Shared Function GetBytes(Of TStruct)(ByVal Values As TStruct()) As Byte()
    
          Dim StructSize = Marshal.SizeOf(GetType(TStruct))
          Dim Result As Byte()
          Dim gchSource, gchDest As GCHandle
    
          If Values Is Nothing Then Throw New ArgumentNullException("Values")
    
          ReDim Result((Values.Length * StructSize) - 1)
    
          If Values.Length > 0 Then
             gchSource = GCHandle.Alloc(Values, GCHandleType.Pinned)
    
             Try
                gchDest = GCHandle.Alloc(Result, GCHandleType.Pinned)
    
                Try
                   CopyMemory(gchDest.AddrOfPinnedObject, gchSource.AddrOfPinnedObject, New IntPtr(Result.Length))
                Finally
                   gchDest.Free()
                End Try
             Finally
                gchSource.Free()
             End Try
          End If
    
          Return Result
    
       End Function
    
    End Class
    


    Armin

    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 8:16 PM
  • I tried.... But I do not think it is perfect solution:

        Public Shared Function GetBytes(Of Struct As Structure)(Obj() As Struct) As Byte()
            Dim Size = Runtime.InteropServices.Marshal.SizeOf(GetType(Struct))
            Dim Bytes(Size * Obj.Count - 1) As Byte
            Dim Ptr As IntPtr = Runtime.InteropServices.Marshal.AllocHGlobal(Size)
            For i = 0 To Obj.Count - 1
                Runtime.InteropServices.Marshal.StructureToPtr(Obj(i), Ptr, False)
                Runtime.InteropServices.Marshal.Copy(Ptr, Bytes, i * Size, Size)
            Next
            Runtime.InteropServices.Marshal.FreeHGlobal(Ptr)
            Return Bytes
        End Function
        Public Shared Function ToStructures(Of Struct As Structure)(Bytes() As Byte) As Struct()
            Dim Size = Runtime.InteropServices.Marshal.SizeOf(GetType(Struct))
            Dim Structs(Bytes.Count / Size - 1) As Struct
            Dim Ptr As IntPtr = Runtime.InteropServices.Marshal.AllocHGlobal(Size)
            For i = 0 To Structs.Count - 1
                Runtime.InteropServices.Marshal.Copy(Bytes, i * Size, Ptr, Size)
                Structs(i) = Runtime.InteropServices.Marshal.PtrToStructure(Ptr, GetType(Struct))
            Next
            Runtime.InteropServices.Marshal.FreeHGlobal(Ptr)
            Return Structs
        End Function


    Allow time to reverse.

    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 5:36 PM

All replies

  • I tried.... But I do not think it is perfect solution:

        Public Shared Function GetBytes(Of Struct As Structure)(Obj() As Struct) As Byte()
            Dim Size = Runtime.InteropServices.Marshal.SizeOf(GetType(Struct))
            Dim Bytes(Size * Obj.Count - 1) As Byte
            Dim Ptr As IntPtr = Runtime.InteropServices.Marshal.AllocHGlobal(Size)
            For i = 0 To Obj.Count - 1
                Runtime.InteropServices.Marshal.StructureToPtr(Obj(i), Ptr, False)
                Runtime.InteropServices.Marshal.Copy(Ptr, Bytes, i * Size, Size)
            Next
            Runtime.InteropServices.Marshal.FreeHGlobal(Ptr)
            Return Bytes
        End Function
        Public Shared Function ToStructures(Of Struct As Structure)(Bytes() As Byte) As Struct()
            Dim Size = Runtime.InteropServices.Marshal.SizeOf(GetType(Struct))
            Dim Structs(Bytes.Count / Size - 1) As Struct
            Dim Ptr As IntPtr = Runtime.InteropServices.Marshal.AllocHGlobal(Size)
            For i = 0 To Structs.Count - 1
                Runtime.InteropServices.Marshal.Copy(Bytes, i * Size, Ptr, Size)
                Structs(i) = Runtime.InteropServices.Marshal.PtrToStructure(Ptr, GetType(Struct))
            Next
            Runtime.InteropServices.Marshal.FreeHGlobal(Ptr)
            Return Structs
        End Function


    Allow time to reverse.

    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 5:36 PM
  • If you mark your structure with <Serialisable> attribute, then you can serialise the whole array to a MemoryStream using BinaryFormatter, then obtain the bytes from MemoryStream (or serialise directly to file, database, etc.). Then you can deserialise the array.

    It is also possible to write a portion of code in C# or C++/CLR, where you can obtain the bytes easier.


    • Edited by Viorel_MVP Thursday, February 19, 2015 7:09 PM
    • Proposed as answer by IronRazerz Thursday, February 19, 2015 7:22 PM
    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 7:07 PM
  • Hi,

     Viorel beat me to it but, i was thinking the same thing. Mark your structure as Serializable and use the BinaryFormatter class to Serialize the whole list to a MemoryStream and then you can Deserialize back to a List of your structure.

    Here is a short basic example i just threw together to test it and it seems to work pretty good.

    Imports System.Runtime.Serialization.Formatters.Binary
    
    Public Class Form1
        Private Byts() As Byte
    
        <Serializable()> _
        Private Structure MyStruct
            Public str As String
            Public int As Integer
            Public dbl As Double
            Public sngl As Single
            Public shrt As Short
        End Structure
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
            Dim sList As New List(Of MyStruct)
    
            Dim s As New MyStruct
            s.str = "First"
            s.int = 50
            s.dbl = 33.523
            s.sngl = 12.33343
            s.shrt = 255
            sList.Add(s)
    
            s = New MyStruct
            s.str = "Second"
            s.int = 23
            s.dbl = 72.881
            s.sngl = 1.003
            s.shrt = 64
            sList.Add(s)
    
            Byts = GetBytesFromStructureList(sList)
        End Sub
    
        Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click
            Dim sList As List(Of MyStruct) = GetStructureListFromBytes(Byts)
    
            For Each s As MyStruct In sList
                RichTextBox1.AppendText(s.str & vbNewLine)
                RichTextBox1.AppendText(s.int.ToString & vbNewLine)
                RichTextBox1.AppendText(s.dbl.ToString & vbNewLine)
                RichTextBox1.AppendText(s.sngl.ToString & vbNewLine)
                RichTextBox1.AppendText(s.shrt.ToString & vbNewLine)
                RichTextBox1.AppendText("---------------------" & vbNewLine)
            Next
        End Sub
    
        Private Function GetBytesFromStructureList(ByVal structList As List(Of MyStruct)) As Byte()
            Dim bts() As Byte
            Using ms As New IO.MemoryStream
                Dim formatter As New BinaryFormatter
                formatter.Serialize(ms, structList)
                bts = ms.GetBuffer
            End Using
            Return bts
        End Function
    
        Private Function GetStructureListFromBytes(ByVal bytesArray() As Byte) As List(Of MyStruct)
            Dim lst As New List(Of MyStruct)
            Using ms2 As New IO.MemoryStream(bytesArray)
                Dim formatter As New BinaryFormatter
                lst = CType(formatter.Deserialize(ms2), List(Of MyStruct))
            End Using
            Return lst
        End Function
    End Class


    If you say it can`t be done then i`ll try it

    • Edited by IronRazerz Thursday, February 19, 2015 7:30 PM
    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 7:22 PM
  • Another one:

    Imports System.Runtime.InteropServices
    
    Public Class Main
    
       'Make sure the struct only contains value types and the StructLayout attribute is attached. No prob if just Single() or Integer().
    
       <StructLayout(LayoutKind.Sequential)> _
       Public Structure OpenDMLStandardIndexEntry
          Public Offset As UInteger
          Public Size As UInteger
       End Structure
    
       Public Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (ByVal Destination As IntPtr, ByVal Source As IntPtr, ByVal Length As IntPtr)
    
       Shared Sub Main()
    
          Dim Entries(3) As OpenDMLStandardIndexEntry
          Dim Bytes = GetBytes(Entries)
    
       End Sub
    
       Shared Function GetBytes(Of TStruct)(ByVal Values As TStruct()) As Byte()
    
          Dim StructSize = Marshal.SizeOf(GetType(TStruct))
          Dim Result As Byte()
          Dim gchSource, gchDest As GCHandle
    
          If Values Is Nothing Then Throw New ArgumentNullException("Values")
    
          ReDim Result((Values.Length * StructSize) - 1)
    
          If Values.Length > 0 Then
             gchSource = GCHandle.Alloc(Values, GCHandleType.Pinned)
    
             Try
                gchDest = GCHandle.Alloc(Result, GCHandleType.Pinned)
    
                Try
                   CopyMemory(gchDest.AddrOfPinnedObject, gchSource.AddrOfPinnedObject, New IntPtr(Result.Length))
                Finally
                   gchDest.Free()
                End Try
             Finally
                gchSource.Free()
             End Try
          End If
    
          Return Result
    
       End Function
    
    End Class
    


    Armin

    • Marked as answer by Youjun Tang Friday, February 27, 2015 9:06 AM
    Thursday, February 19, 2015 8:16 PM
  • Another one:



    Armin

    Armin,

    Hey (young man). I recognize your name from the before times. CIS, QB or somewhere after? I doubt you remember me I had other names. You helped me a lot as I recall.

    Tom


    Thursday, February 19, 2015 9:08 PM