Removing duplacate values from field RRS feed

  • Question

  • Hello,

    Is there a way of removing duplicate values from csv file fields on import? I have a csv file which looks like this:

    ID func_1                                      func_2                                       func_3

    1 carrier//carrier//bowman//bowman holder//holder//dreamer//holder  killer//giver//killer

    What I need to do is remove the duplicate values in fields so it will look like this:

    ID func_1               func_2                  func_3

    1 carrier//bowman   holder//dreamer    killer//giver

    Is there a way of achieving it without using Script task?

    Tuesday, April 26, 2011 9:21 AM


All replies

  • Hi

    Use Script Component Transformation after Data source

    and for the fields split the column as String Array. whit '//' as separater

    and check duplicates in that array

    other wise insert only distinct values..


    Tuesday, April 26, 2011 9:39 AM
  • you can use a database table, to fill data inside this, set primary key on table. and then ignore error output , also set oledb destination access mode as OpenRowSet. ( this is one of available approaches and not the best )
    Tuesday, April 26, 2011 9:46 AM
  • Is that one row you've shown, or do the slashes and spaces denote row separators?

    To remove duplicates from a single "cell" - you can use SortDeduplicateDelimitedString.

    To remove duplicates from a rowset, here are several techniques.

    Todd McDermid's Blog Talk to me now on
    • Marked as answer by Tony Chain Wednesday, May 4, 2011 7:06 AM
    Wednesday, April 27, 2011 9:39 PM
  • Using Script task or component would be the easiest way to achieve this.

    Else you would have to put the data to a staging table and then query the table to eleminate dupes from each record for each columns.

    My Blog    |    Ask Me
    Thursday, April 28, 2011 2:55 PM