none
Two FileStream instances used to access same file RRS feed

  • Question

  • We have a multi-threaded application that writes and reads large amounts of raw binary data (floating point arrays) to a single file.  Access to the file is maintained by a single class, the DataFileManager. Two methods, Write(...) and Read(...) are used by clients that need to write and read data.  The object essentially acts as a cache for data, with all data buffered in the file itself.

    When constructed, DataFileManager creates two FileStreams - one for writing and one for reading:

    FileStreamWriter = File.Open(filename, FileMode.OpenOrCreate, FileAccessWrite, FileShare.Read);
    FileStreamReader = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);

    These streams remain open for the life of the object, until closed in Dispose().  The Write() and Read() methods use a lock, to prevent simultaneous writing and reading by different threads. The important code from each:

    public float[,] Read(int start, int numBytes) {
      ... limit checks ...
      lock (ThisLock) {
        BinaryReader reader = new BinaryReader(FileStreamReader);
        byte[] buffer = reader.ReadBytes(numBytes);
    
        ... code to copy byte[] to float[,] ...
      }
    }
    
    public void Write(float[,] data) {
      ... limit checks ...
      lock (ThisLock) {
        byte[] buffer = new buffer[...]
        ... code to copy float[,] to buffer[]
        FileStreamWriter.Write(buffer, 0, numBytes);
      }
    }
     

    There are two properties of DataFileManger, TotalBytesWritten and BytesAvailableForReading, which do nothing more than return the Length property of the FileStreamWriter and FileStreamReader, respectively.  I have code in another class that checks the TotalBytesWritten to insure data can be read before calling Read().  The Read() method, however, checks BytesAvailableForReading and throws an exception if the read request exceeds the number of bytes available, as indicated by the length property of the FileStreamReader.

    On rare occasions, the Length property of the FileStream used for writing is greater than the Length for the stream used for reading, and the exception is thrown.  Today when debugging, I expanded the properties of these FileStream objects at the point the exception was being thrown and sure enough, the Length of the stream writer was 1024 bytes greater than the stream reader.  I went to another thread to check some values and returned to this code and found the Length properties of the two streams were now the same!  Somehow, touching or reading properties seems to update their values.

    I'm not sure what I should do.  This is code that has been in the product for a year and seems to work 99.999% of the time.  My first instinct is to switch to checking the FileStreamReader Length before starting a read, but I'm concerned the FileStream object for reading will not recognize the newly written data (by the FileStreamWriter) and will hang forever waiting for the file size to grow.

    Has anyone had experience using two FileStream objects simultaneously, to write and read the same file?

    Thursday, June 16, 2016 3:59 PM

Answers

  • And my only point that is important is this one:

    [quote]

    I mean by using List<byte> you have .Count to use that'll report the exact number of bytes to be written.

    [/quote]

    At the instant the Write() call is executed, exactly how much bytes does the buffer contains will be passed to the API. You should never experience buffer overflow/underflow problem this way.

    It's much like if you use some form internally to keep track of how many goods you buy and sell, but occasionally the warehouse would complain there is not enough goods to send out, why won't you contact the warehouse to get the exact number of goods there first?
    Wednesday, June 29, 2016 1:13 AM
    Answerer

All replies

  • Why not declare buffer in Write() as List<byte> and use buffer.Add() in your "code to copy float[] to buffer[]"? In that case the FileStreamWriter.Write() call can be rewritten as:

    FileStreamWriter.Write(buffer.ToArray(), 0, buffer.Count);


    Friday, June 17, 2016 1:53 AM
    Answerer
  • I need to convert the float[,] array to a byte array.  The exact (unsafe) code is:

    byte[] buffer = new byte[numBytes];
    unsafe {
    	fixed (byte* bufPtr = &(buffer[0])) {
    		float* outPtr = (float*)bufPtr;
    		int i = 0;
    		for (int r = 0; r < numberSamples; r++) {
    			fixed (float* inPtr = &(data[r, 0])) {
    				for (int c = 0; c < numberChannels; c++) {
    					outPtr[i++] = inPtr[c];
    				}
    			}
    		}
    	}
    	FileStreamWriter.Write(buffer, 0, numBytes);
    }
    

    But the Write() isn't a problem.  The FileStreamReader's Length doesn't match the FileStreamWriter Length.

    Since I posted this, I added a call: FileStreamWriter.Flush();  I have not seen the problem, but that doesn't mean it's fixed.  I was never able to come up with a reliable way to make this fail.  I will just have to test and test and test to see if I ever see this failure agin.

    Friday, June 17, 2016 3:21 PM
  • I mean by using List<byte> you have .Count to use that'll report the exact number of bytes to be written.

    And you need not use unsafe function at all, .NET framework has covered your need. Use it BitConverter to convert the float to byte[] and that use .AddRange() to add the returned byte[] to List<byte>.

    Saturday, June 18, 2016 10:47 AM
    Answerer
  • I mean by using List<byte> you have .Count to use that'll report the exact number of bytes to be written.

    And you need not use unsafe function at all, ... Use it BitConverter to convert the float to byte[] and that use .AddRange() to add the returned byte[] to List<byte>.

    That gets bytes for a single floating point value.  I read and write 2-D (float[,]) arrays that are typically 256 x 3000 values.  If I used this on every floating point value and then either wrote each 4-byte value in a separate read/write call, my performance would slow to a crawl.

    I don't think I'm communicating my problem clearly (I have no issue with the float/byte conversion).  My issue is that I have two FileStream objects - one for writing and one for reading.  It seems the length for the writing stream sometimes is longer than the read stream.  I'm just trying to confirm this and if using Flush() on the write stream will solve the problem.

    Monday, June 20, 2016 9:57 PM
  • Hi John,

    >>"There are two properties of DataFileManger, TotalBytesWritten and BytesAvailableForReading

    You need to lock these properties when you access them.

    Best Regards,
    Li Wang


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, June 28, 2016 8:16 AM
    Moderator
  • And my only point that is important is this one:

    [quote]

    I mean by using List<byte> you have .Count to use that'll report the exact number of bytes to be written.

    [/quote]

    At the instant the Write() call is executed, exactly how much bytes does the buffer contains will be passed to the API. You should never experience buffer overflow/underflow problem this way.

    It's much like if you use some form internally to keep track of how many goods you buy and sell, but occasionally the warehouse would complain there is not enough goods to send out, why won't you contact the warehouse to get the exact number of goods there first?
    Wednesday, June 29, 2016 1:13 AM
    Answerer