none
Remove Duplicate lines from text file

    Question

  • Hi,

    I have a text file having structure

    [section1]

    key1 = value1

    key2 = value2

    :

    [section2]

    key = value

    :

    I have to remove lines from above file by searching section and key.

    If key found twice,have to delete all entries associated with that key.

    Can anybody help?

    Thank you!

    Friday, June 22, 2012 12:46 PM

Answers

All replies

  • Hi,

    how does your file look like, what to erase, and what is should be after deleting?

    From your code above its all very unclear what to do. You didnt even specify any duplictes.

    ---

    You have to know that removing lines from file directly will not do. You will have to read whole file, and then create a new object and save it as the same filename.

    What to do inbetween, answer on my 1st question.


    Mitja

    Friday, June 22, 2012 12:51 PM
  • Add each line to a dictionary

    put the Dictionary<>.Add() in a try catch loop

    if it is a duplicate value it will not be added to the dictionary but will throw and exception so will be caught so you can keep track of how many duplicates you have etc.

    This is the simplest way to do what you are wanting.


    Digital Forensic Software Developer
    CCS LABS Digital Forensic Software
    Mark as Answer or Vote up if useful thank you!

    Friday, June 22, 2012 1:21 PM
  • Would you please check-out below threads

    http://social.msdn.microsoft.com/forums/en-us/csharpgeneral/thread/CBF5BAE9-D2E5-4C72-A547-FFAFFA1C28C2

    http://social.msdn.microsoft.com/forums/en-us/smallbasic/thread/2FCB49D3-5E75-43A4-A8B2-24E58B2D816B


    Regards,
    Ahmed Ibrahim
    SQL Server Setup Team
    My Blog
    This posting is provided "AS IS" with no warranties, and confers no rights. Please remember to click "Mark as Answer" and "Vote as Helpful" on posts that help you.
    This can be beneficial to other community members reading the thread.

    Friday, June 22, 2012 1:24 PM
  •  Using a Dictionary is a great solution here, you can check if the key already exists in the dictionary, and if not insert new key with a value.

    On the end of file reading you have a dictionary with a unique keys with coresponding value.

    If you wanna save only these data to a new file, simply use StreamWriter class, and do the loop throgh the dictionary (as my example foreach bellow) and writhe each line (WriteLine method).

    Here`s the code:

            void ReadFile_to_UniqueKeys()
            {
                string file = @"C:\myFile.txt";
                Dictionary<string, string> dic = new Dictionary<string, string>();
                using (StreamReader sr = new StreamReader(file))
                {
                    string line;
                    string[] data = null;
                    while ((line = sr.ReadLine()) != null)
                    {
                        if (line.Contains("="))
                        {
                            data = line.Split('='); //data[0] is key, data[1] is value
                            if (!dic.ContainsKey(data[0]))
                                dic.Add(data[0], data[1]);
                        }
                    }
                }
                //now you have all unique keys here, to get them you can do:
                foreach (KeyValuePair<string, string> kvp in dic)
                {
                    string key = kvp.Key;
                    string value = kvp.Value;
                }
            }


    Mitja

    Friday, June 22, 2012 2:09 PM
  • Add each line to a dictionary

    put the Dictionary<>.Add() in a try catch loop

    if it is a duplicate value it will not be added to the dictionary but will throw and exception so will be caught so you can keep track of how many duplicates you have etc.

    This is the simplest way to do what you are wanting.

    Exceptions are expensive; you're far better off just using the 'Contains' method before trying to add anything.
    Friday, June 22, 2012 2:31 PM