Back to public tools
All tools
Text utility

Duplicate Line Remover

Clean repeated lines with first or last retention, inspect occurrence counts and original positions, sort naturally, process independent files or compare original and cleaned lists.

Processing happens locally in your browser. Your lists are not uploaded or saved.

Paste one entry per line, choose matching and retention rules, then Clean lines. The complete cleaned list appears below with occurrence counts and original positions. Files, presets and batch processing stay local.

Source text and matching

0 / 5,242,880 UTF-8 bytes. Up to 100,000 parsed lines; invalid or oversized input is retained, never truncated.

Normalization changes matching keys only; output keeps the chosen original text unless trimming is selected. NFKC can combine compatibility characters that look or behave differently. Case matching uses JavaScript lowercase conversion rather than full language-specific case folding.

Keep, select and sort

Keep-last follows the last retained positions in source order. All-occurrences mode keeps every member of a repeated group. Natural sorting compares unsigned ASCII digit runs by magnitude, without converting them to floating-point numbers; leading-zero ties retain source order. Matching rules and output sorting are separate.

Reusable cleanup presets and batches

Presets are versioned settings-only JSON up to 32 KiB. Batches clean files independently: 1–10 files, 5 MiB / 100,000 lines each, at most 20 MiB / 200,000 lines combined. The complete ZIP contains indexed cleaned outputs and a manifest without input names. An invalid file rejects the complete batch; original drafts/files are unchanged.

Clean, count and compare text lines

Paste one entry per line or import a UTF-8 .txt file, choose matching and output options, then select Clean lines. Edits mark previous results as outdated until you clean again. Clear, examples, presets and imports offer Undo; Cancel stops a running task.

Keep the first or last occurrence, keep one representative of repeated groups, retain every repeated member, or keep only singletons. Original order follows retained source positions: keeping the last copy of apple, banana, apple produces banana, apple. Counts never decorate the cleaned text.

Case and edge-space matching are optional. Output trimming is separate and can make otherwise distinct groups look identical. Unicode NFC or NFKC changes matching keys only; original spelling stays in the result unless output trimming is enabled. Normalization is not fuzzy or accent-insensitive matching.

Optional prefix grouping compares a number of Unicode code points or text before a literal delimiter. It can deliberately combine different full lines. Leave it off for whole-line deduplication. A missing delimiter uses the whole matching key.

Sort, inspect and export results

Character sorting keeps its original ordinal behavior. Natural sorting compares unsigned ASCII digit runs by numeric magnitude without losing precision; other characters use Unicode code-point order. Numerically tied forms such as 2 and 02 retain their relative order.

Counts and original one-based line positions are paginated for display. Repeated-only display filtering does not restrict exports. Counts CSV, position CSV and full position JSON contain all groups. CSV line_json cells contain JSON-encoded source text, preserving control characters and preventing source lines from becoming spreadsheet formulas.

Download the full cleaned TXT or copy it. If copying is blocked, select the result for manual copying. Settings presets contain options only, with no source lines, filenames or positions. Built-in presets can be adjusted before cleaning.

Send original and cleaned lists to List Comparison Tool through a five-minute memory-only handoff. Accept explicitly there, review its existing matching settings, and choose Compare lists. Cleanup normalization and prefix rules are not transferred; Undo restores previous comparison drafts.

Files, limits and privacy

Windows, Unix and older Mac line breaks are supported; a final line ending adds no extra entry. UTF-8 decoding is strict. The optional BOM setting removes one leading marker from imports, without changing pasted text. Invalid imports preserve your current input and result.

Process up to ten .txt files independently into an indexed ZIP, rather than combining them. Each file allows 5 MiB and 100,000 lines; a batch allows 20 MiB and 200,000 lines combined. Every file must succeed before a ZIP is offered. Filenames are omitted from its manifest.

Prepared matching keys are limited to 20 MiB, settings imports to 32 KiB, and reports or batch ZIPs to 32 MiB. Oversized work fails explicitly without partial or silently truncated downloads. Bounded on-screen previews do not truncate the full TXT or reports.

Processing stays in your browser. Inputs and the comparison handoff are not uploaded or placed in a URL or browser storage. Downloads can contain your private text and positions. For quoted spreadsheet records spanning multiple lines, use CSV Duplicate Remover.

Stars and feedback

Loading…

Tool reference #044