Duplicate Line Remover
Remove duplicate lines from text, keeping only unique entries.
Original Lines: 0Unique Lines: 0Removed Lines: 0Characters: 0
Removing duplicate lines from text
This tool walks through your text line by line and removes any line that has already appeared, preserving the original order. Case insensitivemode treats "Apple" and "apple" as duplicates. Trim whitespace ignores leading and trailing spaces so indented versions of the same line are considered equal. Keep last occurrence reverses the preference — instead of keeping the first time a line appears, it keeps the most recent one. Ideal for cleaning email lists, log files, and CSV data. All processing runs privately in your browser.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Deduplicating a list of values pasted from several sources.
- Reducing a log extract to its distinct messages.
- Producing a unique set for a config or an allowlist.
- Cleaning a file after concatenating two exports.
- Finding out how many distinct entries a list contains.
Frequently Asked Questions
- Is the original order preserved?
- Yes, in both modes. Keeping the first occurrence walks forwards; Keep last occurrence walks backwards and reverses the result, so survivors still appear in their original sequence. Sorting is left to a separate tool, because doing it here would destroy the ordering of logs and CSV rows.
- What does Trim whitespace change?
- It compares lines with leading and trailing whitespace stripped, so indented copies of a line count as duplicates. It also quietly rescues Windows text: splitting on the newline alone leaves a carriage return on every line, so with trimming off, a line ending CRLF never matches the same line ending LF.
- Why aren't visually identical accented lines matched?
- Because the comparison is exact and café has two Unicode spellings — one composed é, or e followed by a combining acute. They render identically and are different strings. Normalise to NFC first if the text came from mixed sources, macOS filenames especially.
- Is case-insensitive matching locale-aware?
- No — it lowercases without a locale, which is exactly right for ASCII and imperfect beyond it. German STRASSE and straße stay distinct, because the sharp s does not lowercase into a double s, and Turkish dotted and dotless i do not fold the way a Turkish reader would expect.
- What happens to blank lines?
- They are deduplicated like anything else, so a file padded with blank separators keeps exactly one. A trailing newline also counts as a final empty line, which is why the Original Lines figure can read one higher than the number of visible entries you pasted.
Common errors and gotchas
- Not deciding on case sensitivity, so the same value in two cases is one entry or two.
- Overlooking trailing whitespace, which makes visually identical lines distinct.
- Keeping the first occurrence when the last one held the corrected value.
- Removing repeats that were meaningful, such as a tally or a count.
- Missing that differing line endings make otherwise identical lines differ.
Related Text Tools tools
Case Converter
Convert between Sentence, Title, camelCase, snake_case, and more.
Word Counter
Count words, characters, and analyze keyword density.
Text to Binary
Convert text into its 8-bit binary representation.
Binary to Text
Decode 8-bit binary back into readable text.
Text Reverser
Reverse text by characters, words, or lines.
Remove Duplicate Lines
Delete repeated lines and keep your list unique.
Remove Empty Lines
Strip blank and whitespace-only lines from text.
Sort Lines Alphabetically
Sort lines A–Z or Z–A, case-sensitive or not.