locked
How to handle large data set? RRS feed

  • Question

  • Hi,

     

    I have 2 questions as follow:

    1) How to read a file (any format) and convert into aray?

    2) If the data array is large (in GByte), is it advisable to use ADO and SQL query? what about MS Access?

    Any better idea on how to save and retrieve large dataset?

     

    Thank you!

    Thursday, October 25, 2007 7:42 AM

Answers

  •  brain_storm wrote:

    1) How to read a file (any format) and convert into aray?

    You'll have to write a parser. There's no SomeFileToArray()-function. Read and process the data as you require, and put it into a suitable container.


     brain_storm wrote:

    2) If the data array is large (in GByte), is it advisable to use ADO and SQL query? what about MS Access?

    Any better idea on how to save and retrieve large dataset?



    If it's that large, keeping it in-memory is out of the question in either case. You'll have to work with memory mapped files or address windowing extensions (AWE). If you can rely on specialized storage solutions, such as a SQL database, that's far superior (in terms of development time, speed, accessability and so forth) to writing your own specialized data structure.
    Thursday, October 25, 2007 10:25 AM