Reading a text file - fast

Question

Hi all,

I have a bunch of text files, actually 1TB in all. I have to read each file line by line. Currently, I am using streamreader and readline method and it seems to take forever.

Are there other methods to read a text file really, really fast?

Can the read of the text file itself be multi-threaded?

Any ideas will be highly appreciated.

Thanks.

A

Answer 1

Can the read of the text file itself be multi-threaded?

Hi ,

As far as I know,you should try to use ThreadPool to solve the problem.

The ThreadPool class can process asynchronous I/O.

Answer 2

If you use threading for file reading it will further slow down the process because the disk head will have to move between different areas on your hard drive to support parallel reading of multiple files.

It is helpful only if you have multiple hard drives..

Try BinaryReader if you know the contents and format of the file, it should provide slightly better performance..

PS: BinaryReader or any other FileStream is slower on IsolatedStorage when compared to regular FileStorage