{"id":64582,"date":"2025-05-12T10:34:10","date_gmt":"2025-05-12T07:34:10","guid":{"rendered":"https:\/\/wpcalc.com\/en\/?p=64582"},"modified":"2026-09-28T09:59:21","modified_gmt":"2026-09-28T06:59:21","slug":"duplicate-lines-remover","status":"publish","type":"post","link":"https:\/\/wpcalc.com\/en\/web-tools\/duplicate-lines-remover\/","title":{"rendered":"Duplicate Lines Remover"},"content":{"rendered":"<section class=\"not-prose calculator-box\" aria-labelledby=\"calculator-title\"><div class=\"flex items-baseline sm:items-center justify-between gap-2 sm:gap-3\"><div class=\"flex flex-col sm:flex-row sm:items-center gap-2\"><span class=\"icon icon-calculator text-primary-700! text-base! dark:text-primary-300!\" aria-hidden=\"true\"><\/span><h2 id=\"calculator-title\" class=\"text-lg font-display font-bold\">Remove Repeated Lines from Text Tool<\/h2><\/div><div class=\"calcdash-favorite group\">\n  <button type=\"button\" class=\"favorite\" id=\"favorite\" data-favorite-id=\"64582\" data-favorite-title=\"Duplicate Lines Remover\" data-favorite-url=\"https:\/\/wpcalc.com\/en\/web-tools\/duplicate-lines-remover\/\" data-favorite-excerpt=\"Remove repeated lines from a list while keeping the first occurrence of each line. Paste your list into Text Input to get a trimmed, deduplicated result.\" aria-label=\"Add to Favorites\" data-favorite-icon=\"icon icon-d-edit\">\n    <span class=\"icon icon-shape-star-empty\" aria-hidden=\"true\"><\/span>\n  <\/button>\n  <div class=\"calcdash-favorite__tip\" aria-hidden=\"true\">\n   <span class=\"favorite-tooltip\" id=\"favorite-tooltip\"><\/span>\n  <\/div>\n  <span class=\"sr-only\" role=\"status\" id=\"favorite-status\"><\/span>\n<\/div><\/div><form action=\"https:\/\/wpcalc.com\/en\/web-tools\/duplicate-lines-remover\/\" method=\"POST\" class=\"calculator\" id=\"calculator-64582\" data-post=\"64582\"><fieldset class=\"fieldset-input\"><legend class=\"sr-only\">Input Fields<\/legend><div class=\"field is-single\" id=\"input-1\"><label for=\"field-1\">Text Input<\/label><textarea id=\"field-1\" name=\"text_input\">Apple\nBanana\nApple\nOrange\nBanana<\/textarea><\/div><\/fieldset><div class=\"buttons\"><button type=\"submit\" data-text=\"Re-Calculate\" id=\"calculate-button\" data-post=\"64582\">Calculate<\/button><button type=\"reset\">Reset<\/button><\/div><div class=\"field is-checkbox hidden!\" id=\"field-auto-calc\"><input type=\"checkbox\" id=\"auto-calc\"><label for=\"auto-calc\">Calculate automatically<\/label><small>If enabled, the result will update automatically when you change any value.<\/small><\/div><div class=\"fieldset-result is-hidden\" aria-labelledby=\"result-title\" aria-live=\"polite\" role=\"region\"> <h3 class=\"result-title bg-gradient-to-r from-primary-50 to-gray-50 dark:from-primary-900 dark:to-gray-800\" id=\"result-title\"><span class=\"icon icon-s-pulse\" aria-hidden=\"true\"><\/span> Your Results<\/h3><div class=\"result-box\"><div class=\"field-result is-column\" id=\"output-1\"><span class=\"field-title\"><span>Unique Lines<\/span><\/span><span class=\"field-value\" id=\"result-1\"><\/span><button type=\"button\" class=\"copy-result\" data-tooltip=\"Copy Result\" aria-label=\"Copy Result\"><span class=\"copy-icon icon icon-document-copy\" aria-hidden=\"true\"><\/span><\/button><\/div><\/div><div class=\"result-actions\"><button type=\"button\" class=\"copy-link\"><svg viewBox=\"0 0 20 20\" fill=\"currentColor\" aria-hidden=\"true\"><path d=\"M12.232 4.232a2.5 2.5 0 0 1 3.536 3.536l-1.225 1.224a.75.75 0 0 0 1.061 1.06l1.224-1.224a4 4 0 0 0-5.656-5.656l-3 3a4 4 0 0 0 .225 5.865.75.75 0 0 0 .977-1.138 2.5 2.5 0 0 1-.142-3.667l3-3Z\"\/><path d=\"M11.603 7.963a.75.75 0 0 0-.977 1.138 2.5 2.5 0 0 1 .142 3.667l-3 3a2.5 2.5 0 0 1-3.536-3.536l1.225-1.224a.75.75 0 0 0-1.061-1.06l-1.224 1.224a4 4 0 1 0 5.656 5.656l3-3a4 4 0 0 0-.225-5.865Z\"\/><\/svg><span class=\"copy-link__text\">Copy link to this calculation<\/span><\/button><span class=\"sr-only\" role=\"status\" id=\"copy-link-status\"><\/span><\/div><\/div><a href=\"#respond\" class=\"hidden transition-opacity duration-300 opacity-0 w-50 text-sm justify-center items-center gap-2 px-4 py-2 rounded bg-gray-200 text-gray-700 hover:bg-gray-300\" id=\"leave-comment\"><span class=\"icon icon-comments\" aria-hidden=\"true\"><\/span>Leave a Comment<\/a><\/form><\/section><section id=\"calc-reactions\" class=\"not-prose hidden my-12 bg-gradient-to-r from-primary-50 to-gray-50 border border-indigo-100 rounded-xl px-6 py-4 shadow-sm\" data-post=\"64582\" aria-live=\"polite\" aria-labelledby=\"calc-reactions-title\"><p id=\"calc-reactions-title\" class=\"text-sm text-gray-500 text-center mb-2\">How did this result make you feel?<\/p><div role=\"group\" aria-labelledby=\"calc-reactions-title\" class=\"grid grid-cols-3 sm:flex sm:flex-row sm:justify-around gap-4 sm:items-center sm:flex-wrap\"><button type=\"button\" aria-pressed=\"false\" class=\"reaction-btn flex flex-col items-center gap-1 text-sm text-gray-700 hover:text-primary-600 cursor-pointer transition-transform hover:scale-105\" data-reaction=\"like\"><span class=\"reaction-count\" data-reaction-count=\"like\">0<\/span><span class=\"reaction text-3xl\" aria-hidden=\"true\">\ud83d\ude00 <\/span><span class=\"reaction-description font-medium\">Like <\/span><\/button><button type=\"button\" aria-pressed=\"false\" class=\"reaction-btn flex flex-col items-center gap-1 text-sm text-gray-700 hover:text-primary-600 cursor-pointer transition-transform hover:scale-105\" data-reaction=\"helpful\"><span class=\"reaction-count\" data-reaction-count=\"helpful\">0<\/span><span class=\"reaction text-3xl\" aria-hidden=\"true\">\ud83d\udca1 <\/span><span class=\"reaction-description font-medium\">Helpful <\/span><\/button><button type=\"button\" aria-pressed=\"false\" class=\"reaction-btn flex flex-col items-center gap-1 text-sm text-gray-700 hover:text-primary-600 cursor-pointer transition-transform hover:scale-105\" data-reaction=\"confused\"><span class=\"reaction-count\" data-reaction-count=\"confused\">0<\/span><span class=\"reaction text-3xl\" aria-hidden=\"true\">\ud83d\ude15 <\/span><span class=\"reaction-description font-medium\">Confused <\/span><\/button><button type=\"button\" aria-pressed=\"false\" class=\"reaction-btn flex flex-col items-center gap-1 text-sm text-gray-700 hover:text-primary-600 cursor-pointer transition-transform hover:scale-105\" data-reaction=\"disappointed\"><span class=\"reaction-count\" data-reaction-count=\"disappointed\">0<\/span><span class=\"reaction text-3xl\" aria-hidden=\"true\">\ud83d\ude1e <\/span><span class=\"reaction-description font-medium\">Disappointed <\/span><\/button><button type=\"button\" aria-pressed=\"false\" class=\"reaction-btn flex flex-col items-center gap-1 text-sm text-gray-700 hover:text-primary-600 cursor-pointer transition-transform hover:scale-105\" data-reaction=\"inaccurate\"><span class=\"reaction-count\" data-reaction-count=\"inaccurate\">0<\/span><span class=\"reaction text-3xl\" aria-hidden=\"true\">\u274c <\/span><span class=\"reaction-description font-medium\">Inaccurate <\/span><\/button><\/div><div id=\"reaction-message\" class=\"hidden mt-8 rounded-md border border-primary-100 bg-white\/50 backdrop-blur-sm px-4 py-2 text-sm text-center shadow-sm\"><\/div><\/section><div class=\"calcdash-ad not-prose calcdash-ad--calc_result calcdash-ad--in-article\"><ins class=\"adsbygoogle\" style=\"display:block\" data-ad-client=\"ca-pub-1721569815777345\" data-ad-slot=\"6317458308\" data-ad-layout=\"in-article\" data-ad-format=\"fluid\"><\/ins><script>(adsbygoogle = window.adsbygoogle || []).push({});<\/script><\/div>\n<style id=\"wpcoderpro-inline-1\">.formula-box {\r\n\tborder-bottom: none !important;\r\n}<\/style><section class=\"formula-box\" aria-labelledby=\"formula-title\">\r\n\t<div id=\"formula-title\" class=\"not-prose font-display mb-4\"><span class=\"icon icon-formula\" aria-hidden=\"true\"><\/span><h2>Duplicate Lines Remover Formula<\/h2><\/div>\r\n\t<figure class=\"not-prose formula\">\r\n\t\t<figcaption class=\"formula-title\">Formula<\/figcaption>\r\n\t\t<div class=\"text-base text-gray-800\" id=\"formula\">\n\\[ U = \\operatorname{firstDistinct}\\!\\left(\\left[\\operatorname{trim}(s)\\mid s\\in\\operatorname{lines}(X),\\ \\operatorname{trim}(s)\\ne\\varnothing\\right]\\right) \\]\n<\/div>\r\n\t<\/figure>\r\n\t<\/p>\n<p><strong>Where:<\/strong><\/p>\n<ul>\n<li>\\(X\\) \u2014 the text entered in Text Input.<\/li>\n<li>\\(s\\) \u2014 an individual line of the input.<\/li>\n<li>\\(\\operatorname{trim}(s)\\) \u2014 the line after leading and trailing whitespace is removed.<\/li>\n<li>\\(U\\) \u2014 the nonempty, unique lines in the order they first appear.<\/li>\n<\/ul>\n<p>The tool splits the input into lines, trims each line, skips empty lines, and keeps only the first occurrence of each remaining line. It does not sort the result.<\/p>\n<p>\r\n<\/section>\n<style id=\"wpcoderpro-inline-459\">.calculation-box {\r\n\tborder-bottom: none !important;\r\n}<\/style><section id=\"calculation\" class=\"calculation-box\" aria-labelledby=\"calculation-title\">\r\n\t<div id=\"calculation-title\" class=\"not-prose font-display\"><span class=\"icon icon-unordered-list\" aria-hidden=\"true\"><\/span><h2>Duplicate Lines Remover \u2013 Calculation Example<\/h2><\/div>\r\n\r\n<\/p>\n<p><strong>Example:<\/strong> Enter the default list: Apple, Banana, Apple, Orange, and Banana, with one item on each line.<\/p>\n<p>\\[ [\\text{Apple},\\ \\text{Banana},\\ \\text{Apple},\\ \\text{Orange},\\ \\text{Banana}] \\longrightarrow [\\text{Apple},\\ \\text{Banana},\\ \\text{Orange}] \\]<\/p>\n<p><strong>Result:<\/strong> Unique Lines contains 3 lines: Apple, Banana, and Orange, in that order. The second Apple and second Banana are removed.<\/p>\n<p>\n\r\n<\/section>\n<h2>How to use the Duplicate Lines Remover<\/h2>\n<ol>\n<li>Paste or type your list into <strong>Text Input<\/strong>, placing each item on its own line. The field initially contains a five-line example that you can replace.<\/li>\n<li>Check the <strong>Unique Lines<\/strong> result. Each retained item appears separately, in the same order as its first appearance in your input.<\/li>\n<li>Use the result\u2019s copy control if you need the cleaned list elsewhere. After pasting, check its line breaks in the destination application, especially if you are moving it into a spreadsheet or another editor.<\/li>\n<\/ol>\n<p>You do not need to count or mark duplicates yourself. To test how the tool handles your data, try entering the same item twice with spaces at the beginning or end of one line, then compare the result.<\/p>\n<h2>What counts as a duplicate line?<\/h2>\n<p>A duplicate is a line whose text matches an earlier line after whitespace at both ends has been removed. For example, a line containing <code> Apple <\/code> and a later line containing <code>Apple<\/code> become the same item. Only the first occurrence is kept. Lines containing nothing but whitespace are omitted altogether, so the result can contain fewer lines even when no nonempty item is repeated.<\/p>\n<p>Comparison is case-sensitive. <code>Apple<\/code>, <code>apple<\/code>, and <code>APPLE<\/code> are three different lines and will all remain. Spaces or punctuation <em>within<\/em> a line also matter: <code>New York<\/code> does not match <code>New  York<\/code>. The tool does not correct spelling, standardize formatting, or decide whether two differently written entries mean the same thing.<\/p>\n<p>The result preserves first-seen order. If a name appears on lines two and ten, its position in the output is determined by line two. This is useful when the original sequence conveys priority or context, such as a manually ranked list. If you want alphabetical order afterward, use the <a href=\"https:\/\/wpcalc.com\/en\/web-tools\/list-alphabetizer\/\">List Alphabetizer<\/a>. For differently capitalized entries, a <a href=\"https:\/\/wpcalc.com\/en\/web-tools\/case-converter\/\">Case Converter<\/a> can help standardize the text before you remove duplicates.<\/p>\n<h2>Practical tips and limitations<\/h2>\n<ul>\n<li><strong>Use one record per line.<\/strong> A comma-separated sentence remains one line; the tool does not split it into individual entries. If your items need to be separated first, try the <a href=\"https:\/\/wpcalc.com\/en\/web-tools\/text-separator\/\">Text Separator<\/a>.<\/li>\n<li><strong>Expect exact-text matching after trimming.<\/strong> For an address list, <code>Alex@example.com<\/code> and <code>alex@example.com<\/code> remain separate because their capitalization differs. Review the desired capitalization before treating the output as a definitive list of distinct people.<\/li>\n<li><strong>Check structured data carefully.<\/strong> A CSV row is treated as a whole line, not as parsed columns. Two rows with different spacing or punctuation may remain separate, and the tool cannot identify duplicates based on just one column.<\/li>\n<li><strong>Blank lines are not preserved.<\/strong> Do not use the output when empty lines serve as meaningful separators between groups.<\/li>\n<\/ul>\n<p>For a final check, compare the cleaned result with your original list before deleting the source, particularly when small differences in capitalization, internal spacing, or punctuation may be significant. To measure how much text remains after cleanup, paste the result into the <a href=\"https:\/\/wpcalc.com\/en\/web-tools\/character-counter\/\">Character Counter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Remove repeated lines from a list while keeping the first occurrence of each line. Paste your list into Text Input to get a trimmed, deduplicated result.<\/p>\n","protected":false},"author":70,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[63],"tags":[65],"class_list":["post-64582","post","type-post","status-publish","format-standard","hentry","category-web-tools","tag-text-tools"],"acf":[],"_links":{"self":[{"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/posts\/64582","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/users\/70"}],"replies":[{"embeddable":true,"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/comments?post=64582"}],"version-history":[{"count":3,"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/posts\/64582\/revisions"}],"predecessor-version":[{"id":65332,"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/posts\/64582\/revisions\/65332"}],"wp:attachment":[{"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/media?parent=64582"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/categories?post=64582"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wpcalc.com\/en\/wp-json\/wp\/v2\/tags?post=64582"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}