Remove duplicate lines
apple, banana, apple, cherry, banana becomes apple, banana, cherry: 2 repeated lines removed and 3 unique lines kept in the order they first appeared. You can also treat Apple and apple as the same line. Free in your browser.
- Free
- Runs in your browser
- No sign-up
- Your data stays on your device
Run — free
Worked example
- Text
- apple banana apple cherry banana
- Text
- apple banana cherry
- Original
- apple banana apple cherry banana
- Original lines
- 5
- Unique lines
- 3
Duplicate lines creep into logs, mailing lists, CSV columns exported as plain text, shell histories, code samples pasted twice, and note apps where the same bullet was copied more than once. Cleaning them by hand is slow and error-prone, especially when case differences hide true duplicates or when order matters and you must keep the first appearance rather than the last. This tool takes the multi-line text you enter, splits it into lines, and returns a deterministic deduplicated string that preserves the original order of first occurrences. You can optionally ignore letter case so Apple and apple count as the same line while the first casing you wrote is kept. Empty input is rejected with a clear validation error instead of silently returning an empty result, which protects clients that forgot to bind a field.
How line deduplication works and what ignore-case means
The engine normalizes carriage-return styles so Windows CRLF and bare CR become plain newline separators, then splits the string on every newline. Each resulting line—including intentionally blank lines—is considered as a candidate. The first time a line is seen it is kept with its original characters; every later line that matches the comparison key is dropped.
By default comparison is exact: whitespace, punctuation, and letter case all matter, so trailing spaces or Apple versus apple are different lines. Removed-line and unique-line counts are returned alongside the cleaned text so dashboards and tests can assert how much changed. Because the path is pure parse arithmetic over strings—a single linear scan with a hash set for seen keys—latency stays sub-millisecond for typical form sizes and the published cost model is analytic rather than model-token based.
There is no fuzzy matching, no edit-distance threshold, and no semantic embedding step; those belong to different tools such as semantic CSV dedupe when rows look alike but are not equal as text.
Where removing duplicate lines shows up in real work
Developers cleaning log snippets and shell history dumps need a first-occurrence unique list before pasting into tickets or bug reports. Data teams exporting a single column as plain text often want unique values without loading a full spreadsheet or writing a one-off awk script. Content editors consolidating tag lists, category names, and glossary entries use case-insensitive mode so spelling variants collapse while the preferred first casing remains.
Students and teachers prepare vocabulary lists and answer-key line sets without accidental repeats from copy-paste. Sysadmins merge blocklists and allowlists from multiple sources and need a deterministic unique set that does not reorder earlier trusted entries. If you later need row-level CSV deduplication with column keys, fuzzy semantic merge of near-duplicate rows, or full-file binary hashing of duplicate images, treat those as separate search intents and landings rather than overloading this remove-duplicate-lines page.
What you can do with it
Clean log and history snippets
Paste multi-line logs or shell history and keep the first occurrence of each line so tickets and reports stay short without losing order.
Unique tag and vocabulary lists
Collapse repeated tags or glossary lines, optionally ignoring case, while preserving the first preferred spelling.
FAQ
How are duplicate lines detected?
The text is split on newlines after normalizing CRLF. The first occurrence is kept; later matches are dropped.
Are blank lines preserved?
Yes. An empty line is a valid line value. The first blank line is kept; additional blank lines are treated as duplicates of that empty line.
What happens if the text is empty?
Provide a non-empty string; empty boxes are treated as client errors and are not charged.
What does it cost?
The browser tool is free and runs locally.
Which copy of a repeated line is kept?
Always the first one. Every later copy is dropped and the remaining lines keep the order in which they first appeared, so entries near the top of a trusted list never move.
Related tools
For developers — API access
Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.
API endpoint
Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, by email and from Telegram — and soon from our app too.
Call it from your stack
curl -X POST https://api.kit.forhosting.com/str/dedupe-lines \
-H "Authorization: Bearer $KIT_KEY" \
-H "Content-Type: application/json" \
-d '{"text":"apple\nbanana\napple\ncherry\nbanana"}'const res = await fetch("https://api.kit.forhosting.com/str/dedupe-lines", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.KIT_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
"text": "apple\nbanana\napple\ncherry\nbanana"
})
});
const { task_id } = await res.json();import os, requests
res = requests.post(
"https://api.kit.forhosting.com/str/dedupe-lines",
headers={"Authorization": f"Bearer {os.environ['KIT_KEY']}"},
json={
"text": "apple\nbanana\napple\ncherry\nbanana"
},
)
task_id = res.json()["task_id"]<?php
$res = file_get_contents("https://api.kit.forhosting.com/str/dedupe-lines", false, stream_context_create([
"http" => [
"method" => "POST",
"header" => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
"content" => '{"text":"apple\\nbanana\\napple\\ncherry\\nbanana"}',
],
]));
$task = json_decode($res, true);body := bytes.NewBufferString(`{"text":"apple\nbanana\napple\ncherry\nbanana"}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/str/dedupe-lines", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)Example request
{
"text": "apple\nbanana\napple\ncherry\nbanana"
}Example response
{
"task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
"type": "str.dedupe_lines",
"status": "queued",
"_links": {
"result": "/tasks/tsk_…/result"
}
}The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.
Pricing
Published price — no tokens, no invented credits. A failed task is never charged.
Errors
| HTTP | Code | Meaning |
|---|---|---|
401 | unauthorized | Missing or invalid API key. |
402 | insufficient_balance | Your balance doesn't cover the task price. |
404 | unknown_type | That task type doesn't exist. |
429 | rate_limited | Too many requests. Use the webhook instead of polling. |
Technical notes
How line deduplication works and what ignore-case means
When ignore_case is true, the membership key is the Unicode lowercased form of the line while the kept string is still the first original spelling you provided, which is the usual expectation for cleaning contact lists and tag dumps without rewriting titles. Extremely long payloads above the documented character cap fail with invalid_input so a single request cannot monopolize Worker memory.
What you send and what you get back
Send a required field named text containing the non-empty multi-line string you want cleaned. Numbers and booleans used as the text body are coerced to their decimal or true/false text forms; objects, arrays, NaN, and Infinity are rejected. Empty text fails validation before any set is built and is not charged. Aliases such as input, value, data, s, message, string, content, and body are accepted for convenience when wiring older clients. Optionally send ignore_case as a boolean or as common string forms true, false, yes, no, one, or zero. On success the response returns text as the deduplicated string, original as the input string for echo and debugging, original_lines, unique_lines, removed_lines, the resolved ignore_case flag, and operation set to dedupe_lines. Classic checks include a fruit list with repeated banana and apple lines collapsing to three unique lines, a mixed-case list that collapses further when ignore_case is true, and a single unique line that passes through unchanged with removed_lines equal to zero. Pricing for successful API calls is the published base of $0.002 per item; validation failures including empty text do not consume prepaid balance.
Where removing duplicate lines shows up in real work
Because this capability is a pure parse engine, API pricing is the published base of $0.002 per successful item and failed validation does not consume balance.
Integration use cases
Normalize export columns as text
Turn a plain-text dump of a single column into a unique line list for fixtures, blocklists, and QA without opening a spreadsheet.
Technical questions
Does ignore_case change the kept spelling?
No. Only the comparison key is lowercased. The line text that is kept is always the first original spelling you provided.