Parllel I/O -- No advantage?


  • Sorry if this is the wrong forum- I am new here.

    I am reading  large blocks of data (_open, _read)  in from two different files, each on different devices, processing the data and repeating the cycle. This can be done 2 ways:

    I. Reade FILE1 into Buffer1. (say 500 MB). When done, read FILE2 into buffer2 (also 500 MB).  Then process data, repeat.

    II. Do the same thing in 2 different threads. Thread1 reads File1, thread2 reads File2. (Supposedly) at the same time. process data. repeat.

    I have never found scenario II to be any faster than I. I have tried this on, maybe, 4 different HP machines, 2 different Dells, running XP/Pro and 2 different Dells, using any combination of internally attached discs (SATA/PATA), USB attached, Ethernet attached (1GB), the results are the same. I have also done this on a Vista machine. No change. Incidentally, doing a _read of 500 Mb is SLOW compared to blocking it to a sequence of 32 KB reads, all cases.

    Am I doing something wrong? I have not extracted code small enough to include here; I will do that if there is any interest (will take me a week or more).

    Is it all a limitation on the architecture of most cheap machines? I never saw this on other Unix-like machines (Sequent, Convex, Cray, HP, Alliant, Sun, even Sun 386-types).

    Any insight is most appreciated.

    Jimmy da V












    Wednesday, December 21, 2011 3:54 AM


  • You're Doing It Wrong™. You need to extract a bit of data, and process it, then extract a bit more. Right now, you're blocking the CPU forever waiting for the data from the disk, then it gets to process. If the processing is light, then you're spending virtually all your time waiting on the disk. Having two threads wait on the disk at once won't make any difference because you're still waiting on the disk which can only serve up data at a certain rate. You need to interleave the I/O and the processing so the CPU and the disk drive can work at the same time. 
    Monday, December 26, 2011 12:57 AM