Invisible characters: what they are and how to get rid of them

Your text looks fine and behaves badly. These are the usual six culprits.

When a search finds nothing, two identical rows refuse to deduplicate, or a CSV import fails on the first column, the cause is usually an invisible character. The common ones are the non-breaking space (U+00A0), the zero-width space (U+200B), the byte order mark (U+FEFF) and the soft hyphen (U+00AD). They arrive by copy-paste and are impossible to see.

The six you will actually meet

The BOM deserves special mention because it produces the single most confusing error message in data work: a CSV where the first column header is id and nothing matches id. In Excel, save as CSV UTF-8 and you get a BOM; save as CSV (Comma delimited) and you get the system code page instead. Neither is plain UTF-8.

How to see them

• Word: press Ctrl+Shift+8 or click the pilcrow button. A non-breaking space shows as a raised degree-like circle instead of a middle dot. Soft hyphens appear as a small hyphen with a hook.
• Notepad++: View → Show Symbol → Show All Characters. Non-breaking spaces are highlighted in orange in recent versions.
• VS Code: highlights unusual invisible characters by default since 2021 and offers to remove them. The setting is editor.unicodeHighlight.invisibleCharacters.
• Any browser: select the suspicious text and count. If the character count is higher than the letters you can see, something is hiding — a character counter that reports with and without spaces makes this a two-second check.
• Command line: cat -A file.txt on Linux, or hexdump -C file.txt | head. A BOM shows as ef bb bf at the very start.

Removing them without damaging the text

Word. Find and Replace, then in the Find field type ^s for a non-breaking space and ^- for an optional hyphen, replacing with a normal space and nothing respectively. Turn off AutoFormat as you type afterwards or Word puts some of them back.

Notepad++. Search mode Extended, find \xA0, replace with a space. For zero-width characters use Regular expression mode and [\x{200B}\x{200C}\x{200D}\x{FEFF}], replace with nothing.

Google Sheets. =CLEAN() only strips the first 32 control characters and misses U+00A0 entirely. Use =SUBSTITUTE(A2,CHAR(160)," ") and then =TRIM(). This is why TRIM alone so often appears not to work.

Excel. Same story: =SUBSTITUTE(A2,CHAR(160)," ") before =TRIM(). Find and Replace also works if you paste an actual non-breaking space into the Find box, which is awkward but reliable.

Python or a script. s.replace(chr(160),' ') for the space, and s.encode().decode('utf-8-sig') to drop a BOM while reading a file. In a shell, sed -i '1s/^\xEF\xBB\xBF//' file.csv removes the BOM from the first line only, which is the safe version — a global substitution would also delete legitimate U+FEFF characters further into the file.

Browser. Paste into a text cleaner that normalises whitespace and strips zero-width characters, then copy back. This is the fastest route when the text came from a PDF or a web page and you do not know what is in it.

What usually goes wrong

• TRIM does nothing. TRIM only removes the ordinary space, U+0020. Substitute CHAR(160) first, always.
• Two identical rows will not deduplicate. One has a trailing non-breaking space. Clean before you remove duplicates, not after.
• Numbers import as text. A thousands separator that is a non-breaking space — standard in French and Russian formatting — makes 1 200 unparseable. Replace it before importing.
• Stripping every zero-width character breaks emoji. Family and profession emoji are built from zero-width joiners. Removing U+200D turns one emoji into three. If the text contains emoji, target U+200B and U+FEFF only.
• Removing the BOM breaks a file that needed it. Some older Windows tools require it to detect UTF-8. Change the consumer, not the file, when you have that choice.
• Word count disagrees with the platform. Zero-width characters inflate character counts without changing what you see. Recount with a word counter after cleaning.
• Sorting puts an entry in the wrong place. A leading invisible character sorts before every letter. Clean, then sort.

A standing rule for pasted text

Anything arriving from a PDF, a web page, a Word document or a chat message gets cleaned before it enters a spreadsheet, a database or a CMS. Three steps, in order: replace non-breaking spaces with ordinary spaces, strip zero-width characters, trim each line and collapse runs of whitespace.

Then count. If the character count still exceeds what you can see by more than the spaces you counted, something else is in there — usually a control character from a database export. At that point look at the bytes rather than guessing.

If you own the pipeline, do it once at the boundary instead of in every sheet: normalise input to NFC, map U+00A0 to a plain space, drop U+200B and U+FEFF, and reject anything still containing control characters below U+0020 other than tab and newline. Four rules, applied at the point data enters the system, remove the problem for everyone downstream.

The habit costs about fifteen seconds and removes an entire category of bug that is otherwise nearly impossible to diagnose, because the evidence is invisible by definition.

Related tools

See all text tools →

Frequently asked questions