Remove duplicate words
Deletes repeated words and keeps the first of each. Different from removing duplicate lines.
Paste text and press the button: every word that already appeared earlier is deleted, and the first occurrence stays where it was. Word order is preserved, so the result still reads in the same sequence you wrote it.
Words or lines — which one do you actually need?
This is the single most common mix-up in this corner of the internet, and picking the wrong tool quietly ruins your data. The two operations look similar and do completely different things.
Removing duplicate words looks inside every line. Given red green red blue green it returns
red green blue. It treats the text as a stream of words and does not care where the line breaks are.
Removing duplicate lines compares whole lines against each other. Given a list where red green red
appears twice on separate lines, it keeps one of those lines intact — including the repeated words inside it.
Rule of thumb: if your text is a paragraph, you almost certainly want duplicate words. If your text is a list with one item per line, you want duplicate lines — the button for that is right here too, and there is a dedicated page with more options.
How the comparison works
Two words count as the same when they match after lowercasing and after stripping punctuation attached to them.
That means Cat, cat, cat, and cat. are one word. This is what
people almost always want, because the alternative — exact byte comparison — leaves cat, and
cat both in the output and looks broken.
Punctuation is not thrown away. It stays attached to whichever occurrence survives, so a sentence still ends with its full stop. Whitespace is collapsed to single spaces, because deleting a word from the middle of a line otherwise leaves a double gap.
What people use this for
- Keyword and tag lists. Tags collected from several posts overlap heavily. Deduplicating the words before you paste them into a CMS stops the same tag being created twice.
- Cleaning up dictated or transcribed text. Speech-to-text repeats words when someone stutters or restarts a sentence — "the the meeting", "I I think". One pass removes them.
- Trimming AI or template output. Generated copy loops on the same adjectives. Seeing the deduplicated version shows you instantly how repetitive the draft really is.
- Building a vocabulary list. Paste an article, remove duplicates, and you have every distinct word in it — useful for language learners and for checking terminology consistency.
- Search queries and prompts. Long queries assembled from fragments repeat terms, which adds nothing and sometimes skews results.
A caution about meaning
Deduplicating words destroys grammar. "The cat sat on the mat" becomes "The cat sat on mat" because the second "the" is a duplicate. That is correct behaviour for a word list and wrong for prose you intend to publish. Use this on lists, tags and raw material — not as a final edit of something a human will read. If your goal is to find repetition rather than delete it, the word frequency counter shows counts without touching the text.
Doing it without this tool
Excel or Google Sheets: there is no single function. You would split the cell into columns with
TEXTSPLIT (or Data → Text to Columns), transpose it into rows, apply UNIQUE, then join the
result with TEXTJOIN. In older versions without UNIQUE it is a pivot table or a helper column
with COUNTIF.
Notepad++: the built-in "Remove duplicate lines" works on lines only. For words you need a regular expression
in Find & Replace with "Regular expression" ticked: search for \b(\w+)(\s+\1\b)+ and replace with
\1. Note this only removes duplicates that are adjacent — it will not catch a repeat five words
later, which is the case most people actually have.
Command line: tr ' ' '\n' < file.txt | awk '!a[tolower($0)]++' | tr '\n' ' ' does the job on
Linux and macOS, though it flattens all line breaks.
Word: nothing built in. The usual workaround is a VBA macro, which most people cannot run on a work laptop because macros are blocked by policy.
Runs entirely in your browser. Your text is never uploaded — open DevTools, Network tab, and you will see zero requests while you use this tool.
How to use it
- Paste or type your text into the box.
- Press "Remove duplicate words" — the first occurrence of each word stays, later repeats are deleted.
- If your text is a list with one item per line, press "Remove duplicate lines" instead.
- Press Undo if the result is not what you expected, then copy or download the text.
Related tools
Frequently asked questions
Duplicate words are compared within the whole text regardless of line breaks, so "red green red" becomes "red green". Duplicate lines are compared as complete lines against each other, so a repeated line is deleted but words repeated inside one line are kept. Use words for paragraphs and lines for lists.
Yes. Comparison ignores case and any punctuation attached to the word, so "Cat", "cat," and "cat." all count as one word. The first occurrence keeps its original capitalisation and punctuation.
No. It removes a repeat anywhere in the text, however far apart the occurrences are. Regular expression tricks in editors like Notepad++ usually only catch adjacent repeats, which is why they miss most duplicates.
It can. Removing every repeated word deletes articles and prepositions that grammar needs, so prose comes out ungrammatical. That is expected — the tool is meant for lists, tags and raw text, not for a final draft.
No fixed limit. Everything runs in your browser, so the practical ceiling is your device memory; documents of several hundred thousand words are fine on a normal laptop.
No. The page contains no server-side processing for this tool — the text stays in the browser tab and is never sent, logged or stored.