locked
Find duplicate and remove from List<T> by LINQ RRS feed

  • Question

  •                            //if duplicate data found then remove duplicate data by LINQ
                               xmlDocTargetFile = XDocument.Load(strFilePath);
    
                               //removing duplicate data
                               xmlDocTargetFile.Descendants("TickerBrokerDateFormatMap").GroupBy(row =>
                               new
                               {
                                   StandardDate = row.Element("StandardDate").Value,
                                   Broker_Id = row.Attribute("Broker_Id").Value,
                                   BrokerTab_Id = row.Attribute("BrokerTab_Id").Value
                               })
                               //.Where(grp => grp.Count() > 1)
                               .SelectMany(m => m.Skip(1)).Remove();
                               
                               //save file again
                               xmlDocTargetFile.Save(strFilePath);

    The above code worked when i deal with xml file and use XDocument class but when  try to do the same thing using List<T>

    then Remove() function was not available.

    suppose i stored employee data in List<T> and want to remove duplicate employee based on EmployeeID and DepartmentID.

    how to achieve the same this way using Skip(1) and Remove()

    thanks


    • Edited by Sudip_inn Sunday, August 30, 2020 9:03 AM
    Sunday, August 30, 2020 9:02 AM

Answers

  • Check an example about employees. It uses Skip(1) too:

    class Employee

    {

           public int EmployeeID;

           public int DepartmentID;

           // . . .

    }

    . . .

    var data = new List<Employee>

    {

           new Employee{ EmployeeID = 1, DepartmentID = 100},

           new Employee{ EmployeeID = 2, DepartmentID = 200},

           new Employee{ EmployeeID = 1, DepartmentID = 200},

           new Employee{ EmployeeID = 2, DepartmentID = 200},

           new Employee{ EmployeeID = 2, DepartmentID = 200},

           new Employee{ EmployeeID = 1, DepartmentID = 200},

    };


    data

           .GroupBy( e => new { e.EmployeeID, e.DepartmentID } )

           .SelectMany( e => e.Skip( 1 ) )

           .All( e => { data.Remove( e ); return true; } );


    • Edited by Viorel_MVP Sunday, August 30, 2020 1:24 PM
    • Marked as answer by Sudip_inn Monday, August 31, 2020 7:00 AM
    Sunday, August 30, 2020 1:23 PM

All replies

  • Check an example about employees. It uses Skip(1) too:

    class Employee

    {

           public int EmployeeID;

           public int DepartmentID;

           // . . .

    }

    . . .

    var data = new List<Employee>

    {

           new Employee{ EmployeeID = 1, DepartmentID = 100},

           new Employee{ EmployeeID = 2, DepartmentID = 200},

           new Employee{ EmployeeID = 1, DepartmentID = 200},

           new Employee{ EmployeeID = 2, DepartmentID = 200},

           new Employee{ EmployeeID = 2, DepartmentID = 200},

           new Employee{ EmployeeID = 1, DepartmentID = 200},

    };


    data

           .GroupBy( e => new { e.EmployeeID, e.DepartmentID } )

           .SelectMany( e => e.Skip( 1 ) )

           .All( e => { data.Remove( e ); return true; } );


    • Edited by Viorel_MVP Sunday, August 30, 2020 1:24 PM
    • Marked as answer by Sudip_inn Monday, August 31, 2020 7:00 AM
    Sunday, August 30, 2020 1:23 PM
  • Hi Sudip_inn,

    Thank you for posting here.

    Or you can get the first one directly instead of deleting the objects after the first one.

    MyClassList result = (MyClassList)serializer.Deserialize(reader); List<MyClass> list = result.Items; var re = from element in list group element by new { element.State } into groups select groups.First();

      List<MyClass> myClasses = re.ToList();

    If you need to rewrite it into a file, you can use XmlSerializer.

                using (var writer = new FileStream(strFilePath, FileMode.Create))
                {
                    XmlSerializer ser = new XmlSerializer(typeof(List<MyClass>),
                        new XmlRootAttribute("users"));
    
                    ser.Serialize(writer, myClasses);
                }

    Best Regards,

    Timon


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, August 31, 2020 7:00 AM
  • why people use SelectMany along with skip(1) ?

    .SelectMany( e => e.Skip( 1 ) )

    select many actually select all data...am i right ?

    please tell me reason behind why SelectMany() used in this scenario ?

    thanks

    Monday, August 31, 2020 7:00 AM