none
Filling a large Array with random Bytes RRS feed

  • Question

  • I am creating a Hard Drive Eraser using normal Filesystem calls, it repeatedly writes files until the disk is 100% full. I wold like the option to fill the write buffer with random bytes but am getting out of memory exceptions trying to fill a 1 Gig buffer. I have sort of fixed that but I still see out of memory when I change from 1GB to 512MB and back to 1GB - Looping through the array works fine but is agonizingly slow. Have a look at my sample app please and any suggestions would be great. Using much smaller files avoids this but I don't want to leave a 4 TB drive with tens of thousands of files if possible.

    Form Layout:

    Code:

    Option Strict On
    Imports System.Security.Cryptography
    
    Public Class Form1
        Private Sizes As New List(Of Integer)
        Private SizeText As New List(Of String)
        Private Filesize As Integer = 0
        Private RNG As New RNGCryptoServiceProvider
        Private R As New Random
        Private Buffer(1) As Byte
        Private SW As New Stopwatch
    
        Private Function ShowFriendlyNumber(ByVal number As Long) As String
            Dim TB As Long = 1024L * 1024L * 1024L * 1024L
            Dim GB As Long = 1024L * 1024L * 1024L
            Dim MB As Long = 1024L * 1024L
            Dim KB As Long = 1024L
            ShowFriendlyNumber = "0"
            If number > TB Then
                Return Format((number / TB), "F2") & " TB"
            ElseIf number > GB Then
                Return Format((number / GB), "F2") & " GB"
            ElseIf number > MB Then
                Return Format((number / MB), "F2") & " MB"
            ElseIf number > KB Then
                Return Format((number / KB), "F2") & " KB"
            Else
                Return number.ToString & " Bytes"
            End If
        End Function
    
        Private Sub DisplayBuffer(Method As String)
            TxtResults.Text = "Buffer Size = " & ShowFriendlyNumber(Buffer.Length) & " - Took " &
                               SW.Elapsed.ToString & " Using " & Method & vbNewLine
            TxtResults.AppendText("First 100 Bytes:" & vbNewLine)
            For I = 0 To 99
                TxtResults.AppendText(Buffer(I).ToString("X2") & " ")
            Next
            TxtResults.AppendText(vbNewLine & vbNewLine & "Last 100 Bytes" & vbNewLine)
            For I = Buffer.Length - 100 To Buffer.Length - 1
                TxtResults.AppendText(Buffer(I).ToString("X2") & " ")
            Next
        End Sub
    
        Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
            Sizes.AddRange({1, 64, 128, 256, 512, 1024})
            SizeText.AddRange({"1 MiB", "64MiB", "128 MiB", "256 MiB", "512 MiB", "1 GiB"})
            CBSize.Items.Clear()
            CBSize.Items.AddRange(SizeText.ToArray)
            CBSize.SelectedIndex = 0
        End Sub
    
        Private Sub CBSize_SelectedIndexChanged(sender As Object, e As EventArgs) Handles CBSize.SelectedIndexChanged
            Filesize = Sizes(CBSize.SelectedIndex) * 1024 * 1024 - 1
        End Sub
    
        Private Sub BtnUseCrypto_Click(sender As Object, e As EventArgs) Handles BtnUseCrypto.Click
            Try
                SW.Reset()
                SW.Start()
                If Buffer.Length <> Filesize + 1 Then
                    ReDim Buffer(Filesize) ' Seems to fix using 1GiB multiple times - 512Mib to 1GiB still errors
                End If
                RNG.GetBytes(Buffer) 'Exception Always HERE
                SW.Stop()
                DisplayBuffer("Crypto")
            Catch ex As Exception
                TxtResults.Text = "Exception - " & ex.Message & ex.StackTrace & vbNewLine
            End Try
    
        End Sub
    
        Private Sub BtnOnebyOne_Click(sender As Object, e As EventArgs) Handles BtnOnebyOne.Click
            SW.Reset()
            SW.Start()
            ReDim Buffer(Filesize)
            For I As Integer = 0 To Buffer.Length - 1
                Buffer(I) = CByte(R.Next(0, 256))
            Next
            SW.Stop()
            DisplayBuffer("Looping")
        End Sub
    End Class
    

    On my system, a 1 GB buffer takes 10 sec using the Crypto Library and 40 seconds using the loop. I'd really prefer something even faster even if it isn't Cryptographically secure so I could fill each file with different data.  See the bold text in the code.

    Thursday, August 15, 2019 1:20 AM

Answers

  • Try this:

       SW.Start()

       ReDim Buffer(FileSize)

       R.NextBytes(Buffer)

       SW.Stop()

     

    Maybe you can also fill the buffer with zero or any other value.

    • Marked as answer by Devon_Nullman Thursday, August 15, 2019 1:56 PM
    Thursday, August 15, 2019 5:35 AM

All replies

  • Try this:

       SW.Start()

       ReDim Buffer(FileSize)

       R.NextBytes(Buffer)

       SW.Stop()

     

    Maybe you can also fill the buffer with zero or any other value.

    • Marked as answer by Devon_Nullman Thursday, August 15, 2019 1:56 PM
    Thursday, August 15, 2019 5:35 AM
  • Thanks Viorel - that is only slightly slower than the System.Security,Cryptography method and doesn't cause any exceptions. ReDim fills the array with zeros, as far as other values, I'll have to look at some of the P/Invoke stuff but I don't think I really need it. Erasing a disk for super high security usually wants a sequence written like:

    All Zeros - All bits cleared
    All 0xFF - All bits set
    All 0x55 - 0101 0101‬
    All 0xAA - 1010 1010
    All Random
    Repeat several times

    This is intended from someone that is donating or disposing of a computer or Hard Drive and will certainly be enough.

    Thursday, August 15, 2019 2:05 PM
  • Update - Solved Filling the buffer quickly with any one Value

    Imports System.Runtime.InteropServices ' Added
    
    Public Class Form1
    
        <DllImport("msvcrt.dll", EntryPoint:="memset", CallingConvention:=CallingConvention.Cdecl, SetLastError:=False)>
        Public Shared Function MemSet(dest As IntPtr, c As Integer, byteCount As Integer) As IntPtr
        End Function ' Added
    
    
    
        Private Sub BtnMemset_Click(sender As Object, e As EventArgs) Handles BtnMemset.Click
            SW.Reset()
            SW.Start()
            Dim Filler As Integer = CInt(NUDFill.Value) ' NUDFill is a NumericUpDown
            If Buffer.Length <> Filesize Then
                ReDim Buffer(Filesize - 1)
            End If
            Dim pinned = GCHandle.Alloc(Buffer, GCHandleType.Pinned)
            MemSet(pinned.AddrOfPinnedObject(), Filler, Buffer.Length)
            SW.Stop()
            DisplayBuffer("Memset")
        End Sub
    
    
    Buffer Size = 1024.00 MiB - Took 00:00:00.8461908 Using Memset
    First 10 Bytes:
    AA AA AA AA AA AA AA AA AA AA 
    
    Last 10 Bytes
    AA AA AA AA AA AA AA AA AA AA 
    

    Nice.......

    Thursday, August 15, 2019 2:58 PM