Text

Remove Duplicate Lines

Removes repeated lines while keeping the original order, or shows you only the duplicates instead.

Loading the tool…

Processing happens locally in your browser. What you paste or load is processed by this page and is not uploaded to a server. Nothing is stored unless you use a control that says it stores something, and you can clear anything this site has kept from the privacy page.

How to use this tool

  1. Paste the list.
  2. Decide what counts as the same line: surrounding spaces and letter case can each be ignored.
  3. Choose whether to keep the first of each line, only the lines that repeat, or only the lines that appear once.
  4. Select De-duplicate. The summary reports how many lines went in and how many came out.

What duplicate line remover does

De-duplicating a list is easy; de-duplicating it without scrambling the order is the part that matters. Sorting a file to remove duplicates destroys whatever order the file had, which is fine for a word list and wrong for a log, a changelog or a query result. Here the first occurrence of each line stays exactly where it was and later copies are dropped.

The other two modes are often more useful than de-duplication itself. Keeping only the duplicates tells you which values repeat, which is how you find a broken join or a double-loaded batch. Keeping only the lines that appear exactly once tells you what is missing from the other side of a comparison. Whitespace and case can each be ignored when deciding what counts as the same line.

Frequently asked questions

No. The first occurrence of each line stays exactly where it was and later copies are removed. Sorting to de-duplicate — which is the usual shortcut — destroys the order, and that matters for a log, a changelog or a query result.

Change Keep to "Only lines that repeat". That inverts the tool: instead of removing repeats it shows you nothing but them, which is how you find a broken join, a double-loaded batch or a key that is not as unique as somebody claimed.

Something invisible differs — usually a trailing space or a non-breaking space. Turn on ignoring surrounding spaces first. If they still do not match, paste both into the character frequency counter, which names every character by its Unicode code point.