Duplicate Line Remover
Clean repeated lines with first or last retention, inspect occurrence counts and original positions, sort naturally, process independent files or compare original and cleaned lists.
Processing happens locally in your browser. Your lists are not uploaded or saved.Paste one entry per line, choose matching and retention rules, then Clean lines. The complete cleaned list appears below with occurrence counts and original positions. Files, presets and batch processing stay local.
Clean, count and compare text lines
Paste one entry per line or import a UTF-8 .txt file, choose matching and output options, then select Clean lines. Edits mark previous results as outdated until you clean again. Clear, examples, presets and imports offer Undo; Cancel stops a running task.
Keep the first or last occurrence, keep one representative of repeated groups, retain every repeated member, or keep only singletons. Original order follows retained source positions: keeping the last copy of apple, banana, apple produces banana, apple. Counts never decorate the cleaned text.
Case and edge-space matching are optional. Output trimming is separate and can make otherwise distinct groups look identical. Unicode NFC or NFKC changes matching keys only; original spelling stays in the result unless output trimming is enabled. Normalization is not fuzzy or accent-insensitive matching.
Optional prefix grouping compares a number of Unicode code points or text before a literal delimiter. It can deliberately combine different full lines. Leave it off for whole-line deduplication. A missing delimiter uses the whole matching key.
Sort, inspect and export results
Character sorting keeps its original ordinal behavior. Natural sorting compares unsigned ASCII digit runs by numeric magnitude without losing precision; other characters use Unicode code-point order. Numerically tied forms such as 2 and 02 retain their relative order.
Counts and original one-based line positions are paginated for display. Repeated-only display filtering does not restrict exports. Counts CSV, position CSV and full position JSON contain all groups. CSV line_json cells contain JSON-encoded source text, preserving control characters and preventing source lines from becoming spreadsheet formulas.
Download the full cleaned TXT or copy it. If copying is blocked, select the result for manual copying. Settings presets contain options only, with no source lines, filenames or positions. Built-in presets can be adjusted before cleaning.
Send original and cleaned lists to List Comparison Tool through a five-minute memory-only handoff. Accept explicitly there, review its existing matching settings, and choose Compare lists. Cleanup normalization and prefix rules are not transferred; Undo restores previous comparison drafts.
Files, limits and privacy
Windows, Unix and older Mac line breaks are supported; a final line ending adds no extra entry. UTF-8 decoding is strict. The optional BOM setting removes one leading marker from imports, without changing pasted text. Invalid imports preserve your current input and result.
Process up to ten .txt files independently into an indexed ZIP, rather than combining them. Each file allows 5 MiB and 100,000 lines; a batch allows 20 MiB and 200,000 lines combined. Every file must succeed before a ZIP is offered. Filenames are omitted from its manifest.
Prepared matching keys are limited to 20 MiB, settings imports to 32 KiB, and reports or batch ZIPs to 32 MiB. Oversized work fails explicitly without partial or silently truncated downloads. Bounded on-screen previews do not truncate the full TXT or reports.
Processing stays in your browser. Inputs and the comparison handoff are not uploaded or placed in a URL or browser storage. Downloads can contain your private text and positions. For quoted spreadsheet records spanning multiple lines, use CSV Duplicate Remover.
Stars and feedback
Loading…
What should we improve next?
Help shape the next update to Duplicate line remover. Tell us what would make it better for you.
Prefer email? feedback@tooltulip.com
Tool reference #044