Duplicate Line Remover

How to remove duplicate lines

1. Paste your list or import one UTF-8 .txt file. Small lists process live; larger lists process in a local worker after a short pause.
2. Choose matching rules, blank filtering, optional output trimming and output order.
3. Review the four counts, then copy or download the complete UTF-8 unique-lines.txt result.

Worked example

apple
banana
apple

The default result is apple followed by banana: three input lines, two output lines, one duplicate removed, and zero blanks removed.

Matching does not change retained text

By default, matching is case-sensitive, edge whitespace matters, blank lines are removed, output text is unchanged, and first-occurrence order is preserved. Case-insensitive matching uses locale-independent Unicode lowercasing, not full linguistic equivalence: there is no accent folding, fuzzy matching or Unicode normalization.

Match trimming runs before optional lowercasing and affects comparison keys only. For example, apple and apple match when edge trimming is enabled, but the first original apple remains in the result unless Trim output lines is enabled.

Output trimming is independent and happens after deduplication. With match trimming off, distinct entries can become identical-looking output lines. The tool warns about those collisions and does not secretly deduplicate again.

Line parsing, blanks and counts

CRLF, LF and lone CR separate entries; output separators are LF. Empty input has zero entries, and one terminal separator does not add an extra entry. Internal empty lines are entries. Whitespace-only lines, including tabs, are blank for the blank-filter option.

For a\n\na\n, there are three entries: one blank is filtered, one duplicate is removed, and the output is a. When blank filtering is off, blank lines participate in the same matching rules as other lines.

Blank removals are counted separately. Duplicates removed equals nonblank candidates minus unique retained keys when blank filtering is on; with it off, all entries are candidates. Output-line count describes retained entries, even when an empty entry serializes as an empty string.

Sorting and limitations

Optional ascending/descending sorting compares retained output strings using JavaScript ordinal UTF-16 string ordering. This is deterministic, not browser-locale or natural-number ordering: 10 comes before 2 in ascending order.

Limits are 5 MiB UTF-8 and 100,000 parsed lines, with no silent truncation. Inputs up to 32 KiB process immediately; larger inputs debounce for 200 ms and use a cancellable bundled worker. Input/option changes discard old results and worker replies. Counts are pending while processing or when no valid result is available.

Only UTF-8 .txt files are imported. Removing one leading BOM is optional and applies only to imports. Invalid UTF-8 leaves your previous input unchanged. Downloads always contain the complete result, not a preview. Clear and Use result as input offer in-memory Undo; refresh clears drafts.

This is not a CSV parser, email validator or URL canonicalizer. Quoted CSV records may span physical lines; use a CSV-specific tool instead. Literal HTML remains text. Clipboard denial leaves selectable output for manual copying. No scheduled jobs, remote processing or persistent storage are used.