Remove duplicate lines

Keeps the first occurrence of every line and preserves the original order.

Paste a list — emails, keywords, URLs, anything one per line — and duplicates disappear. Comparison ignores leading and trailing spaces.

How to use it

1. Paste your list, one item per line.
2. Press Remove duplicates.
3. Optionally sort the result, then press Copy.

What this tool does

Every line of the pasted text is compared with the lines before it. The first time a line appears it is kept in place; every later repeat is dropped. The order of what remains is exactly the order you pasted, which is what you want when the list is ranked or chronological.

Details that decide the result:

• Leading and trailing spaces are ignored, so apple and apple  are the same line. This catches most copy-paste artefacts.
• Comparison is case sensitive: Apple and apple both survive. Convert the case first with the case converter if you want them merged.
• Blank lines are left alone so paragraph spacing survives.
• Internal spacing matters: john smith and john  smith are different. Run the text cleaner first if that is a risk.

Examples

Combining buttons covers most real cleanups. Remove duplicates then Sort A→Z gives a unique alphabetical list — the equivalent of sort -u on the command line. Sorting first and deduplicating second gives the same set, just discovered in a different order.

When you need it

• Email and contact lists. Two exports merged into one file almost always overlap; sending the same campaign twice to the same address is the visible symptom.
• Keyword research. Exports from different tools repeat the same terms, and a deduplicated list is what you actually plan content against.
• URL lists for crawling or redirects. Duplicate rows in a redirect map produce conflicting rules.
• Log analysis. Pulling out unique error messages from thousands of lines shows how many distinct problems there are, not how many events.
• Vocabulary and reference lists. Any list built up by hand over time collects repeats.
• SKUs and product codes. Duplicates in an import file cause a partial upload that is painful to unwind.

A useful habit before deduplicating: normalise the list first. Mixed capitalisation, stray trailing spaces and inconsistent internal spacing all create duplicates that no exact-match comparison can see. Lowercase the list with the case converter, run the text cleaner, then deduplicate. On a merged export of a few thousand email addresses that order routinely removes two or three times as many rows as deduplicating the raw file does.

What the tool cannot catch is a duplicate that is genuinely written differently: bob@example.com and bob+news@example.com reach the same inbox but are different strings, and example.com/page and example.com/page/ are the same page with different text. Those need a rule of their own before the comparison runs.

Doing it without this tool

Every major program can do this; the menu paths are just easy to forget.

Excel

1. Select the column or range.
2. Data → Data Tools → Remove Duplicates.
3. Tick My data has headers if row 1 is a header, choose which columns define a duplicate, press OK. Excel reports how many rows it removed.

To keep the original and get a unique copy instead, use Data → Advanced → Copy to another location with Unique records only ticked. In Microsoft 365, =UNIQUE(A2:A500) produces a live list that updates itself. To find duplicates without deleting anything, use Home → Conditional Formatting → Highlight Cells Rules → Duplicate Values.

Google Sheets

1. Select the range.
2. Data → Data cleanup → Remove duplicates.
3. Confirm which columns to analyse and press Remove duplicates.

Or leave the source untouched and put =UNIQUE(A2:A) in an empty column.

Notepad++

• With the TextFX plugin: TextFX → TextFX Tools → Sort lines case sensitive and remove duplicates. This sorts as a side effect.
• Without plugins, keep the order: Search → Replace, tick Regular expression and . matches newline, find ^(.*?)$\s+?^(?=.*^\1$) and replace with nothing.
• For adjacent duplicates only, find ^(.*)(\r?\n\1)+$ and replace with $1.

Command line

• sort -u file.txt — unique and sorted.
• awk '!seen[$0]++' file.txt — unique while keeping the original order, the closest match to what this page does.
• uniq alone only removes adjacent duplicates, so it needs sorted input first.
• PowerShell: Get-Content file.txt | Select-Object -Unique.

Privacy: your text never leaves your browser

Deduplication runs in JavaScript on your device. The list is never uploaded, never logged and never stored; reloading the page clears it.

This is the one tool where privacy is nearly always relevant, because the lists people deduplicate are email addresses, customer records and internal URLs. Pasting a subscriber list into a server-side tool is a data transfer that may need a contract behind it; pasting it here is not, because nothing leaves the tab.

Related tools

See all text tools →

Frequently asked questions