CSV

CSV Duplicate Row Remover

Removes repeated rows, matching either on the whole row or on named key columns.

Loading the tool…

Processing happens locally in your browser. What you paste or load is processed by this page and is not uploaded to a server. Nothing is stored unless you use a control that says it stores something, and you can clear anything this site has kept from the privacy page.

How to use this tool

  1. Paste the CSV.
  2. Leave the key columns blank to match whole rows, or name the columns that identify a record.
  3. Remove duplicates and check the count in the summary against what you expected.

What csv duplicate remover does

Duplicates arrive in extracts for ordinary reasons: a job that ran twice, a join that fanned out, or two source systems that both hold the customer. Removing them requires deciding what "the same" means. Matching on the entire row only removes byte-identical copies; matching on a key column removes rows that describe the same entity even when a timestamp or a comment differs.

The first occurrence of each key is kept and later ones are dropped, so if the file is already in a meaningful order — most recent last, say — you may want to reverse it before deduplicating. The summary reports how many rows were removed, which is worth checking: a much larger number than expected usually means the key you chose is not as unique as you thought.

Frequently asked questions

The first occurrence in file order. If you want the most recent kept instead, sort the file so the newest rows come first before deduplicating.

Yes. List the column names separated by commas and rows are compared on that combination, which is how a composite key behaves.

Usually because the key columns you chose are less unique than assumed. Run the column profiler first and compare the distinct count against the row count.