I am reading large blocks of data (_open, _read) in from two different files, each on different devices, processing the data and repeating the cycle. This can be done 2 ways:
I. Reade FILE1 into Buffer1. (say 500 MB). When done, read FILE2 into buffer2 (also 500 MB). Then process data, repeat.
II. Do the same thing in 2 different threads. Thread1 reads File1, thread2 reads File2. (Supposedly) at the same time. process data. repeat.
I have never found scenario II to be any faster than I. I have tried this on, maybe, 4 different HP machines, 2 different Dells, running XP/Pro and 2 different Dells, using any combination of internally attached discs (SATA/PATA), USB attached, Ethernet
attached (1GB), the results are the same. I have also done this on a Vista machine. No change. Incidentally, doing a _read of 500 Mb is SLOW compared to blocking it to a sequence of 32 KB reads, all cases.
Am I doing something wrong? I have not extracted code small enough to include here; I will do that if there is any interest (will take me a week or more).
Is it all a limitation on the architecture of most cheap machines? I never saw this on other Unix-like machines (Sequent, Convex, Cray, HP, Alliant, Sun, even Sun 386-types).
You're Doing It Wrong™. You need to extract a bit of data, and process it, then extract a bit more. Right now, you're blocking the CPU forever waiting for the data from the disk, then it gets to process. If the processing is light, then you're spending
virtually all your time waiting on the disk. Having two threads wait on the disk at once won't make any difference because you're still waiting on the disk which can only serve up data at a certain rate. You need to interleave the I/O and the processing so
the CPU and the disk drive can work at the same time.