none
Problem with 32bit Distribution RRS feed

  • Question

  • Hello,

     my current project reads a file of 3MB entropy data to find 32 bit variances. Problems

    is a 4GB container is nearly impossible to have in memory. So, I used the following code;

    for ( int y = 0; y < 1024; y++ )
    {
        int z = y * 4;
    
        ushort least = ( ushort ) ( ( this.ioBuffer[ z + 1 ] * 256 ) + this.ioBuffer[ z ] );
        ushort most = ( ushort ) ( ( this.ioBuffer[ z + 3 ] * 256 ) + this.ioBuffer[ z + 2 ] );
    
        uint data = ( uint ) ( ( most * 65536 ) + least );
    
        int x = this.Data32Bit.IndexOf( data );
    
        if ( x == -1 )
        {
            this.Data32Bit.Add( data );
            this.Count32Bit.Add( 1 );
        }
        else
        {
            this.Count32Bit[ x ]++;
        }
    }

     Before anyone ask, the Data32Bit and Count32Bit are object List<uint> and List<int> respectively.

    Also the the object ioBuffer is byte[ 4096 ] array used in BinaryReader class method ReadBytes().

    The code works and performance is slow as expected.    If you could suggest code the would

    boost performance then I would be very appreciative.

     Thanks :)


    • Edited by User3DX Monday, January 15, 2018 11:26 PM wrong title
    Monday, January 15, 2018 11:26 PM

Answers

  • I'd recommend to just increase the buffer size and make the end condition of for loop to "y < (this.ioBuffer.Length / 4)".

    The I/O is the bottleneck, considering you're doing simple operations. In the past I did write a sector copy program that reads 4kb a time, and get tremendous performance by simply increasing the buffer size to read to 256kb.

    • Marked as answer by User3DX Tuesday, January 16, 2018 6:02 PM
    Tuesday, January 16, 2018 4:04 AM
    Answerer
  • Check if this minor adjustment works:

       for( int y = 0; y < 4096; y += 4 )

       {

          uint data = BitConverter.ToUInt32( ioBuffer, y );

          // . . .

       }

     

    Consider replacing lists with a Dictionary<uint, int>. The keys will represent the data got from file. The values will represent the counts. This should work faster than IndexOf. Also compare with SortedDictionary<uint, int>.

    You can also consider parallel threads: one that reads into buffer, other ones that process the data.

    Or start reading the next buffer using FileStream.ReadAsync before the above loop. After the loop, wait until the data are available, switch the buffers, and repeat the procedures.






    • Marked as answer by User3DX Tuesday, January 16, 2018 6:02 PM
    • Edited by Viorel_MVP Wednesday, January 17, 2018 5:21 AM Fixed: 1024→4096
    Tuesday, January 16, 2018 6:12 AM

All replies

  • I'd recommend to just increase the buffer size and make the end condition of for loop to "y < (this.ioBuffer.Length / 4)".

    The I/O is the bottleneck, considering you're doing simple operations. In the past I did write a sector copy program that reads 4kb a time, and get tremendous performance by simply increasing the buffer size to read to 256kb.

    • Marked as answer by User3DX Tuesday, January 16, 2018 6:02 PM
    Tuesday, January 16, 2018 4:04 AM
    Answerer
  • Check if this minor adjustment works:

       for( int y = 0; y < 4096; y += 4 )

       {

          uint data = BitConverter.ToUInt32( ioBuffer, y );

          // . . .

       }

     

    Consider replacing lists with a Dictionary<uint, int>. The keys will represent the data got from file. The values will represent the counts. This should work faster than IndexOf. Also compare with SortedDictionary<uint, int>.

    You can also consider parallel threads: one that reads into buffer, other ones that process the data.

    Or start reading the next buffer using FileStream.ReadAsync before the above loop. After the loop, wait until the data are available, switch the buffers, and repeat the procedures.






    • Marked as answer by User3DX Tuesday, January 16, 2018 6:02 PM
    • Edited by Viorel_MVP Wednesday, January 17, 2018 5:21 AM Fixed: 1024→4096
    Tuesday, January 16, 2018 6:12 AM
  • Hello,

     Thank you cheong00 and Viorel, your solution for code performance boost was spot-on!

    I implemented a larger IO buffer, 4K to 16K.  Used the Dictionary class, btw was faster,

    for the 32bit container.  And rewrote the code to make use of BitConverter, which was

    way better than my method.

     

     In short, in 32bit mode, my project went from long to 0.22 seconds blank screen.

     

     Thank you very much! :)

    Tuesday, January 16, 2018 6:01 PM