Remove Duplicate Lines
Delete repeated lines and keep your list unique.
Deduplicating Lists Instantly
This tool groups your text by line and then reports whichever view you need. Enable case insensitiveto treat "Apple" and "apple" as the same, or ignore surrounding spacesso indentation differences don't create false uniques. Great for cleaning up email lists, log files, and CSV columns — all processed privately in your browser.
- Keep first occurrence — the classic dedupe, original order preserved.
- Keep last occurrence — keeps the final version of each line instead of the first. The one you want when later lines supersede earlier ones, as in an append-only log or a config where the last assignment wins.
- Show only duplicates — inverts the tool: lists just the lines that appeared more than once, so you can see what was repeated rather than what survived.
- Count occurrences — a tab-separated
countthen line, most frequent first. Effectivelysort | uniq -c | sort -rn, and it pastes straight into a spreadsheet.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Deduplicating a list of emails, ids or URLs pasted from several sources.
- Cleaning a log extract down to the distinct messages.
- Producing a unique set of values for a config or an allowlist.
- Finding out how many distinct entries a list actually contains.
- Removing repeated rows after concatenating two exports.
How it works in practice
A worked example
Two exports have been concatenated and the same entry appears in both, sometimes capitalised differently, and you need to know how many entries there really are.
alpha Beta alpha gamma beta alpha
keep first alpha / Beta / gamma / beta keep last Beta / gamma / beta / alpha only duplicates alpha count each 3 alpha · 1 Beta · 1 gamma · 1 beta case-insensitive, count each 3 alpha · 2 Beta · 1 gamma
Four readings of one grouping pass, which is why they agree with each other about what a duplicate is. Keep-first preserves the original order of the survivors; keep-last returns the final occurrence of each, in the order those final occurrences appear, which is a genuinely different order and not merely different text. Only-duplicates answers a different question entirely — which entries repeated at all — and the count mode sorts by frequency with ties broken by first appearance, so the output is stable across runs.
The edge case that catches people
The last line is the one that catches people. Turning on case-insensitive matching merges Beta and beta into one entry of two, and the spelling that survives is whichever appeared first — so the output has silently chosen a capitalisation for you. That is almost always what you want for email addresses, whose domains are case-insensitive by specification, and almost never what you want for identifiers or filenames on a case-sensitive filesystem. Decide before you run it, because the discarded spelling is not recoverable from the result.
When not to use this tool
It compares lines, so anything where a record is not a line will be shredded. A CSV whose quoted fields contain newlines has records spanning several lines, and deduplicating that file by line destroys rows rather than merging them; the CSV tools here parse properly and are the right stop. The same applies to indented JSON or YAML, where identical-looking lines are structure rather than data. And a list that is meant to repeat — a tally, a log of events, anything where the count is the information — loses its meaning the moment it becomes a set.
Frequently Asked Questions
- Which occurrence is kept?
- The first, with later duplicates removed — so the original order of the surviving lines is preserved. That is different from sorting and then deduplicating, which is what `sort -u` does and which discards the input order entirely.
- Does it compare lines exactly?
- By default yes, including whitespace — so `foo` and `foo ` are two different lines. That is usually a surprise rather than a preference, which is why trimming first is worth doing when the data came from a copy-paste or a spreadsheet.
- Is the comparison case-sensitive?
- It is an option, and the right answer depends on the data. Email addresses are case-insensitive in the domain part by specification and usually in practice, so `[email protected]` and `[email protected]` are almost always the same person — while two identifiers differing in case may not be.
- What about lines that differ only by invisible characters?
- They survive as distinct, which is the most confusing failure. A trailing carriage return from a Windows file, a non-breaking space, or a zero-width character all make two visually identical lines unequal — and the only way to see it is to inspect the bytes.
- How does this scale to a large file?
- Deduplication uses a hash set, so it is linear in the number of lines and holds every distinct line in memory. That is fine for hundreds of thousands of lines in a browser and is the point at which a command-line tool becomes the better choice.
Common errors and gotchas
- Not deciding on case sensitivity, where the same address in two cases is one entry or two.
- Overlooking trailing whitespace, which makes two visually identical lines distinct.
- Assuming order is preserved, when some implementations sort as a side effect.
- Deduplicating lines that were meant to repeat, such as a count or a tally.
- Missing that different line endings make otherwise identical lines differ.