locked
Performance for one million records RRS feed

  • Question

  • I perform a query that retrieves one million records. I am using a SqlDataReader and looping through each reader.Read() to build up an object using reflection.

    The original implementation took around 15 minutes to build a million objects.

    I have done a few simple things.

    1. Hold onto the Properties (instead of calling GetProperties in each read)

    2. I build a dictionary lookup to use the Ordinal value for the reader (instead of using the string name ie reader["value'])

    3. Used the Get of the reader instead of casting (ie replaced reader[0] as int? with reader[0].GetInt32())

    The current time taken is now 9 minutes 20ish seconds.

    What should I do next?

    1. Store the GetGetMethod / GetSetMethod of each propertyinfo.

    2. Instead go right into DynamicMethod implementation.

    3. For the reader would performing a GetValues instead of doing reader[0]... reader[15] help.

     

    Also, the WCF endpoint does not let me push those million objects through the pipe to the client giving a generic exception.

    I am currently using Buffered mode. Is a million objects beyond the capabilities of buffered mode?

    1. Should I implement streaming.

    Or

    2. Should I try to identify if I can compress the million objects and configure app.config settings to allow this.

     

    Friday, July 30, 2010 1:07 PM

Answers

All replies