AI Prompts for Text Cleaning: The Character You Cannot See

Two strings print the same and the lookup returns zero rows. Most AI prompts for text cleaning start with a rewrite; the ones that work start with a count, because the space in that record is not the space on your keyboard.

Nothing raises. The query returns nothing, the import reports one column more than the file had, the total comes out short. The values disagree below the level the screen shows: different characters drawing the same shape.

A Model Does Not Clean Text, It Retypes It

A model does not hand your string back. It emits new tokens, and you receive a fresh rendering of what it understood. When the register is prose that rendering arrives polished: curly quotes, long dashes, spaces that keep a phrase together. In a paragraph this is typography. In a key, a CSV field or a JSON body it is another character.

The Keyboard Character and the Typographic One

The model has no reason to separate the two contexts unless the prompt says which one you are in. Ask it to polish customer copy and you get U+2019 where you typed U+0027. Ask for a JSON object and the same model switches to plain forms, because the syntax leaves no room for anything else.

Why the Failure Is Silent

Comparing two different codepoints is a legal comparison that returns false. A lookup that misses returns nothing. An import splitting on a space that is not a space produces one field too many and the loader writes it. Failures that never raise are found by inspecting results, so the first move is to dump the value: length, codepoints, and a character-by-character comparison against something you typed.

AI Prompts for Text Cleaning: Inventory Before Rewrite

The useful order is the one you would use on a database before a migration: describe what is there, count what you find, then change it under a rule that logs every change. Three catalog prompts cover the stages.

The Cleaning Prompt That Refuses to Drop Data

Claude Data Cleaning Partner is built around an inventory: the problems in the file with sample rows, numbered rules for the dedupe key, date format, case normalization and outliers, a script that writes a cleaned file plus a quality report, and the edge cases that escape such as missing values and mixed types. Its two written rules are why it belongs here: never silently drop data, prefer explicit rules over guesses.

Notice what it does not ask: what character was in the field beforehand. Say "normalize this column" and you get a rewrite. Say "list the distinct codepoints in this column with a count for each, and modify nothing yet" and you get evidence. Trimming and case normalization are exactly where an unusual character survives, because both work on what the function considers a space or a letter.

The Debugging Prompt That Asks for the Evidence

Debugging Detective runs six steps: reproduce, isolate by bisecting between layers, three hypotheses each with its test, instrument, minimal fix with a regression test, five-line post-mortem. Its rule is ask for missing evidence instead of guessing.

An invisible character is evidence, not a symptom to describe more precisely. The isolating step between network, data, code and environment is where you learn whether it entered through the keyboard, the export or the model.

The Review Where a Diff Hides a Character

Claude Code Review Partner reviews in six blocks with severity P0 to P2 and one rule that reads differently here: only report issues you can justify with the code. In a diff where the only changed line is a quotation mark, a reviewer without the codepoint has nothing to justify, so it passes.

The Characters That Break a Match

Name each one by its codepoint, not by how it looks.

The Space That Is Not a Space

U+00A0 NO-BREAK SPACE exists so a renderer will not break the line at that point. Its narrow sibling, U+202F, does the same in a smaller width. Both draw as a gap, neither is the character your space bar produces, and a strict comparison against a typed value fails. Whether your trimming routine removes them depends on the routine, not the language: in one common runtime I tested, a bare strip removed the no-break space while the same strip given an explicit space character left it in place. Test yours. The reference entry on non-breaking spaces covers the family.

The Quote That Is Not a Quote

The typographic family is U+2018, U+2019, U+201C and U+201D; the keyboard produces U+0027 and U+0022. In prose the first set is an improvement, but in a key or a JSON document it is another character and the parser has no tolerance for it. A model that spent the afternoon polishing copy carries the habit into the next request.

The Accent That Is Two Characters

An accented letter can be stored as one codepoint or as two: base letter plus combining accent. They are canonically equivalent, so Unicode treats them as the same text, but a strict comparison disagrees. Verified on this machine: precomposed e-acute is U+00E9, the decomposed form is U+0065 followed by U+0301, the raw comparison returns false, and normalizing both makes them equal. That is why UAX #15 defines normalization forms: equivalent strings get one binary representation per form.

The Characters That Draw Nothing

Two more, confirmed against the character database: U+200B ZERO WIDTH SPACE, and U+FEFF, the value some tools write at the start of a file as a byte order mark. Neither draws anything in most editors. The second turns the first key of a CSV into a key that never matches.

The Character Card: Five Lines Before You Prompt

Five lines written before the request turn guessing into a specification:

1. Which string has to match, named as the exact field or key and not as "the name".
2. Where it comes from: keyboard, PDF, spreadsheet or model output.
3. Which characters are legal in that field, and which must be substituted.
4. What "equal" means here: exact, case-insensitive, accent-insensitive, or after normalization.
5. Where the check runs and what it reports: how many values changed, and which ones.

Without those lines the model re-quotes whatever looks untidy, and the untidy thing is often the value you needed to see.

Three Tests That Take Five Minutes

The Codepoint Test

Paste the failing value into a codepoint inspector next to one you typed. If the sequences differ, the strings differ, however alike they print.

The Length Test

A field that should hold a known number of characters holds more. The trim did not remove them because they were never spaces.

The Re-run Test

Normalize, run the same query again, compare the count before and after. If the number does not move, the character was not the problem.

What the Model Still Cannot Do Here

It does not know the rule of the destination system: whether the content manager normalizes on save, whether the database compares accents. It rewrites instead of cleaning, and every rewrite makes a new string. It cannot choose between two plausible values, because that is a decision about your data.

FAQ

Why does text from an AI chat break when I paste it into code?

Because the chat rendered it as prose. Curly quotes and non-breaking spaces are correct typography there and invalid syntax in code. Ask for plain ASCII when the destination is a machine.

Can I just delete every non-ASCII character?

No. That removes accents, currency symbols and real names with the problem characters. Substitute deliberately, per field, and count what you substituted.

Does normalizing text lose information?

Composed and decomposed forms are canonically equivalent: the text is the same and only the representation changes. Report it anyway, because a change you cannot see is a change you cannot review.

The habit fits in one sentence: ask what character is there before asking for the text to be fixed. Claude Data Cleaning Partner gives you the inventory and the report, Debugging Detective the dump and the isolation, and Claude Code Review Partner keeps the invisible line from passing review. The model supplies the form; the codepoint and the rule come from whoever knows the system.