Remove Duplicate Words

What this tool is used for

• Cleaning a keyword or tag list where terms repeat.
• Removing an accidentally doubled word in a draft.
• Deduplicating a list of terms pasted from several sources.
• Producing a unique vocabulary from a passage.
• Tidying a list where the same word appears in different positions.

Frequently Asked Questions

Does it remove all repeats or only adjacent ones?

These are genuinely different operations. Removing consecutive repeats fixes the classic typo 'the the'; removing all repeats keeps only the first occurrence of every word, which destroys normal prose since 'the' legitimately recurs.

Is matching case-sensitive?

By default yes, so 'The' and 'the' are distinct. For typo removal you almost certainly want case-insensitive matching, since a duplicated word at a sentence boundary appears as 'The the'.

Does punctuation affect matching?

Yes — 'word' and 'word,' are different tokens, so a duplicate split across a comma is missed. Stripping punctuation before comparison catches more, at the cost of changing what the tokens mean.

Will it damage legitimate repetition?

It can. 'Had had', 'that that' and 'is is' are grammatical in English, and repetition is a deliberate rhetorical device. Review the changes rather than applying them blindly to prose.

What is this actually for?

Cleaning up dictated or OCR'd text where words genuinely double, deduplicating keyword and tag lists, and tidying auto-generated content. It is a cleanup tool, not a style tool.

Are hyphenated and contracted words treated as one token?

They should be — splitting `well-known` or `don't` produces fragments that then look like duplicates of other fragments. Tokenising on whitespace rather than on every non-letter is what keeps them intact.

Does word order survive?

Yes — the first occurrence keeps its position and later ones are dropped, so the text stays readable. Sorting the unique words instead would destroy the sentence, which is a different operation entirely.

What is lost when duplicates are removed?

Emphasis and frequency. Repetition is often deliberate in prose, and it is the entire signal in a word-frequency analysis — which is why this belongs to editing rather than to text analysis.

Common errors and gotchas

• Removing a repetition that was deliberate, such as 'had had' or 'that that'.
• Not deciding on case sensitivity, so 'The' and 'the' count as different or the same unexpectedly.
• Ignoring punctuation attached to words, which makes duplicates look distinct.
• Removing duplicates across the whole text when only adjacent repeats were the problem.
• Losing word order that carried meaning, since removal keeps only the first position.