none
Strategy for inserting a billion records RRS feed

  • Question

  • The C# application process thread has finished processing 20 million records to generate a billion new records. The uploader thread is damn slow having completed only 2.5 million uploads.

    I am using SqlBulkCOpy to dump .1 million records per batch on a pooled connection.

     

    How can I speed up the upload process?

    Wednesday, September 3, 2008 1:06 AM

All replies

  • You could try to use SQL Server Integration Services. SSIS designed for bulk data integrations, but billion records will not be inserted in a matter of seconds. What is your performance right now?

     

    Wednesday, September 3, 2008 9:59 AM
    Moderator
  • Difficult to say. What is the source of the upload? I wouldn't expect that many rows to be inserted that quickly using the SqlBulkCopy class.

     

    Wednesday, September 3, 2008 5:10 PM