How to reduce search time with a huge Dataset (1 million recs) RRS feed

  • General discussion

  • I am doing a project in VS 2008 for comparing and appending except duplicates records project in which several database's records are appended to a single table without duplicates.

    Database/Tables are of same structure. databases have master and details tables.

    Tables = master1, Details1, master2, details2

    Steps are :
        (i)   Take a Record from Master2, search in Master1 for duplicacy
        (ii)  If not found, add in Master1. Search for all related records in Details2 and add in   Details1.
        (iii) If Found, it won't be added. Search for  all related records in Details2 and search one by one in details1 for match. If not found, append.

        Search keys are 3 for master records and 5 keys for details

    The program is fine for even 100,000 records(almose 30-45 secs) but as the size increases it takes more time (20-25 minuts for 300,000 records). Please help.


    Thursday, February 16, 2012 7:06 AM

All replies

  • Omprakash,

    Why you changed deliberatly the default setting "Question" to "Discussion", a discussion never starts with the word "How" but can start with "Why is it not done like"?


    • Edited by Cor Ligthert Thursday, February 16, 2012 8:43 AM
    Thursday, February 16, 2012 8:42 AM