Skip to content

What Happens to Watermarks in AI-Generated Code? (Part 2 of 3)

Ask ChatGPT or Claude to write a function, a SQL query, or a JSON document, and much of the output has to follow exact rules. If a text watermark works by gently favouring one valid next word over another, what happens when there are very few valid choices?

This second post answers that question. It shows where code still has room for a watermark, what rewriting and translation can do to the signal, and why a detector score needs to be read in context.

Part 2 of 3 - Explore the complete series
Navigate this post

Imagine a website sends a one-time password to your phone: Enter the one-time password: 482731. You must enter those exact digits. Typing 482732 fails the request completely.

Now imagine a friend asks you to text them that you arrived safely:

  • "I arrived safely."
  • "I am here and all is well."
  • "Reached safely - see you soon!"

All three text messages deliver the correct information. The password request allows only one exact answer, while the text message allows several valid options.

This difference controls two main parts of text watermarking:

  1. Where the AI has room to form a pattern
  2. How much of that pattern remains after someone edits the text

As established in Part 1, the AI builds text one word at a time. A watermarking system gently guides the AI toward preferred word choices. However, it can only use choices that still make sense in the sentence.

How choice room sets the boundary

We call the flexibility in an answer choice room:

Choice room is the availability of multiple good word choices that all produce a correct answer.

---
title: "Choice Room Depends on the Task"
---
flowchart LR
    A["One-time password<br/>One exact path"] --> B["Exact quote<br/>Almost no choice room"]
    B --> C["Ordinary explanation<br/>Many valid wordings"]
    C --> D["Creative story<br/>Plentiful choice room"]

Choice room does not measure creativity. It simply asks how many different words could successfully complete the prompt.

A word-choice watermark relies on that flexibility:

---
title: "Ordinary and Watermarked Generation"
---
flowchart LR
    subgraph Marked["Watermarked Text Generation"]
      direction LR
        B1["Text so far"] --> B2["Several sensible word options"]
        B2 --> B3["Secret recipe gives bonus to one option"]
        B3 --> B4["AI picks a preferred, sensible word"]
        B4 --> B5["Repeated choices build watermark evidence"]
    end
    subgraph Ordinary["Ordinary Text Generation"]
      direction LR
        A1["Text so far"] --> A2["Several sensible word options"]
        A2 --> A3["AI picks one option"]
    end

The main takeaway is simple:

A watermark cannot create word choices. It can only use the choices already allowed by the prompt.

A simple example of choice room

Imagine the AI has written:

"Maya reached home ..."

Here are three valid ways to finish the sentence:

Possible Ending Makes Sense? Secret Bonus?
safely Yes Yes
before dinner Yes No
and called me Yes Yes

The watermarking system can favor safely or and called me. It will never choose a nonsensical ending just to add a watermark.

Now consider a different prompt:

"Copy this exact sentence: 'Maya reached home before dinner.'"

Now, before dinner is the only acceptable ending. Replacing those words fails the request.

Standard Prompt Exact Copy Prompt
Multiple endings make sense. Only one ending is correct.
The watermark has room to prefer specific words. The watermark has no room to change any words.

A prompt can lock certain words while leaving others flexible. Therefore, we must inspect text word by word rather than calling an entire document "fixed" or "flexible."

Text length does not equal choice room

Compare these two prompts:

Prompt 1: "Copy this 500-word passage exactly." Prompt 2: "Explain why sleep matters in about 150 words."

The first response is longer, but every single word is fixed. The second response is shorter, but offers many valid ways to phrase the explanation.

Prompt Document Length Valid Alternatives Watermark Opportunity
Copy 500 words exactly Long Almost none Very little
Write a 150-word explanation Shorter Many High

Length only helps when choice room already exists. A longer passage with flexible word choices gives the detector more data points to check. Increasing text length does not create choices if the prompt forbids them.

Watermark Requirement Purpose
Sufficient text length Allows the detector to combine multiple observations.
Sufficient word choices Gives the watermarking system room to apply its secret preferences.

This explains why short factual answers are hard to watermark. They contain few words and offer very few word choices. Google DeepMind notes that SynthID-Text detector confidence drops on short responses and factual prompts for this reason.

What code leaves room to change

Choice room applies to computer code just as it does to regular writing.

Code consists of instructions written for a computer.

Programming languages use strict writing rules called syntax. Omitting a required bracket or keyword creates an invalid program. Additionally, code must deliver the correct behavior by performing the exact task requested.

---
title: "Acceptable Code Needs Syntax and Behaviour"
---
flowchart TB
    A["Valid syntax"] --> C["Working Code"]
    B["Correct behavior"] --> C

Valid code must fulfill both requirements: it must follow syntax rules and execute the correct task.

However, computer code is not entirely rigid. A prompt might lock function names and specific outputs while leaving comments, variable names, and explanations flexible. Another prompt might allow multiple different ways to solve the problem.

Examining choice room in code

Suppose you submit this prompt:

"Create a Python function named 'welcome' that returns the exact text 'Welcome!'"

Here is a standard solution in Python:

welcome.py
def welcome():
    return "Welcome!"

Here is how each part of the code functions:

Code Fragment Purpose Fixed or Flexible?
def Tells Python you are starting a function. Fixed by Python syntax
welcome The specific name of the function. Fixed by prompt
() and : Required Python punctuation. Fixed by Python syntax
return Tells Python to output a result. Fixed by function logic
"Welcome!" The exact output text. Fixed by prompt

Changing welcome to hello breaks the requested function name. Changing "Welcome!" to "Good morning!" alters the required output.

Now change the prompt slightly:

"Create a Python function named 'welcome' that returns any friendly greeting."

The function name and syntax remain fixed, but the output string is now flexible:

welcome.py
def welcome():
    return "Good to see you!"
Text Only
def welcome():                    ← Fixed syntax and function name
    return "Good to see you!"     ← Fixed instruction + flexible greeting text

This demonstrates why we must analyze code line by line to locate flexible choices.

How prompt constraints affect choice room in code

Prompt Fixed Elements Flexible Elements Choice Room
"Output exactly OK" Every character None Almost none
"Name the function 'welcome' and return any greeting" Syntax and function name Greeting text Medium
"Write a program that welcomes a visitor" Final goal Program structure, names, and text High

When analyzing any prompt, ask two questions:

  1. How much of the output is fixed?
  2. In the flexible sections, how many valid options exist?
---
title: "Inspect Choice Room Position by Position"
---
flowchart TD
    A["Inspect one position in the text"] --> B{"Is an exact word required?"}
    B -->|"Yes"| C["No choice room available"]
    B -->|"No"| D{"Do multiple valid options exist?"}
    D -->|"Yes"| E["Choice room exists"]
    D -->|"No"| C
    C --> F["Move to next position"]
    E --> F

Do not assume a document lacks a watermark simply because it contains computer code or is short. The watermark depends on the available choice room position by position.

How edits change watermark evidence

A watermark depends on the specific word choices made during generation. Editing the text afterwards can alter or remove those choices.

Consider this original message:

"The parcel arrived safely before lunch."

Here are four ways to pass this message along:

Action Resulting Text Meaning Preserved? Original Words Preserved?
Exact copy The parcel arrived safely before lunch. Yes Yes
Word swap The parcel arrived safely before noon. Yes Mostly
Rewrite We received the package safely in the morning. Yes No
Translation Same message written in Spanish Yes No

A human reader recognizes the same meaning in all four versions. However, a watermark detector inspects the exact word choices and surrounding context.

---
title: "Meaning Can Survive While Token Evidence Changes"
---
flowchart LR
    A["Original Watermarked Text"] --> B{"How was the text edited?"}
    B -->|"Exact copy"| C["Preserves original word choices and watermark"]
    B -->|"Rewritten"| D["Replaces original word choices"]
    D --> E["Watermark evidence weakens or disappears"]

The core principle is clear:

A rewrite can preserve the original meaning while completely replacing the word choices that held the watermark.

How detectors combine evidence into a score

Imagine ✓ represents a word choice that matched the secret preference, and · represents a choice that did not:

Text Only
Original watermarked text:   ✓ ✓ · ✓ ✓ ✓ · ✓ ✓ ✓ ✓ ·
Text after minor edit:      ✓ ✓ · ? ? ✓ · ✓ ✓ ✓ ✓ ·

The question marks (?) represent words altered by an edit.

A detector checks many word choices across a document and combines them into an overall score. It then compares that score against a tested threshold to decide if a watermark is present.

---
title: "A Detector Combines Observations Into a Result"
---
flowchart LR
    A["Submitted Text"] --> B["Inspect word choices across the text"]
    B --> C["Combine observations into a total score"]
    C --> D{"Compare score to threshold"}
    D -->|"Above threshold"| E["Watermark detected"]
    D -->|"Below threshold"| F["Result uncertain or no mark detected"]

The SynthID-Text research paper details how these scores and thresholds are calculated.

In watermarking, detector confidence refers strictly to the statistical strength of the pattern evidence. It does not measure:

  • The percentage of the document written by AI
  • The probability that any single sentence is AI-generated
  • The factual truth of the claims inside the text

A small edit removes some matching words. If enough matching words remain across the rest of the text, the detector can still confirm the watermark.

Replacing a few words affects surrounding evidence

Consider this simple sentence:

"The work is complete, so we can finish for today."

Now swap one word:

"The work is complete, so we can stop for today."

Replacing finish with stop removes the watermark evidence for that specific word. It can also impact nearby checks because the secret recipe calculates preferences based on preceding words.

Sentence Section What Happened Effect on Watermark
The work is complete, so we can Unchanged Original evidence remains
finish → stop Replaced Evidence at this position is lost
for today Follows the edit Changed context may alter nearby checks
Later unchanged sentences Context aligns again Original evidence resumes

Google reports that SynthID-Text remains detectable after minor changes, such as modifying a few words or applying light edits. However, detection depends on document length, original choice room, and the location of the edits. See Google DeepMind's announcement for details.

Why word-removal shortcuts do not work

Swapping filler words or editing a fixed percentage of text does not reliably eliminate a watermark.

Common Assumption Why It Fails
"Replacing filler words removes the mark." No single word is permanently marked; context dictates preferences.
"Changing 10% of the words deletes the mark." Detection depends on passage length and remaining pattern strength.
"Keeping the main idea preserves the mark." Meaning and word-choice patterns operate independently.

Without access to the secret recipe and detector, you cannot predict whether altering a specific word removes evidence, changes nothing, or accidentally creates a new matching word.

Rewriting text replaces the watermark pattern

Look at how progressive edits affect watermark evidence:

Version Text Content Original Wording Retained Impact on Watermark
Original Please send the result before lunch. 100% Full watermark pattern present
Word Swap Please share the result before lunch. High Minor loss of evidence
Light Rewrite Please share the result with me before lunch. Medium Moderate loss of evidence
Full Rewrite Could you make sure I receive the outcome by midday? Low Most watermark evidence replaced

The full rewrite conveys the same request, but uses completely different word choices.

Google DeepMind notes that light paraphrasing leaves SynthID-Text detectable, while a thorough rewrite significantly reduces detector confidence. The SynthID-Text research paper lists heavy editing and AI paraphrasing as key limits to detection.

Generating in another language vs. translating marked text

Generating text directly in another language produces a different result than translating watermarked text later.

---
title: "Direct Generation and Later Translation Use Different Choices"
---
flowchart LR
    subgraph Translation["Translating Watermarked Text"]
      direction LR
        B1["Watermarked text in Source Language"] --> B2["Translator picks new words in Target Language"]
        B2 --> B3["Original word-choice pattern is replaced"]
    end
    subgraph Direct["Direct Watermarked Generation"]
      direction LR
        A1["AI generates text in Target Language"] --> A2["Watermark influences word choices in real time"]
        A2 --> A3["Text contains a valid watermark in Target Language"]
    end

The SynthID-Text paper demonstrates consistent performance across multiple supported languages during direct text generation.

However, translating already-generated text replaces the original word choices. As a result, translation significantly lowers the detector's confidence score.

Both facts are true:

  • An AI can generate watermarked text directly across multiple languages.
  • Translating an already-watermarked passage weakens or removes its original mark.

How code reformatting affects watermarks

Reformatting code can alter stored characters without changing how the code runs.

Consider this line of code:

Text Only
total=price+tax

A code formatter might add spacing around the operators:

Text Only
total = price + tax

Both versions execute identically on a computer. However, a watermark detector sees different stored characters. Adding spaces alters how text is split into words and how nearby positions are checked.

Reformatting Action Stored Characters Changed? Watermark Impact
Line wrapping on screen No Detector receives identical text; mark remains intact.
Formatter inserts spaces/lines Yes Alters character sequences; may weaken detection.
Renaming variables/restructuring Yes (substantially) Replaces original word choices; significantly reduces mark.

Code formatting changes do not automatically delete a watermark, nor does working code guarantee the watermark survives. The outcome depends on the programming language, the formatter used, and how the detector splits text.

Mixing marked and unmarked text dilutes detector scores

If someone copies one watermarked AI paragraph into a document containing two human-written paragraphs, the document contains mixed content.

---
title: "Mixed Writing Can Dilute a Whole-Document Signal"
---
flowchart LR
    A["Watermarked Paragraph"] --> D["Combined Document"]
    B["Human Paragraph"] --> D
    C["Human Paragraph"] --> D
    D --> E["Detector checks entire document"]
    E --> F["Matching and non-matching scores combine"]

The human-written paragraphs do not gain a watermark simply by sitting next to AI text. Likewise, the AI paragraph does not instantly lose its word choices.

However, many detectors score an entire document as a single unit. Adding unmarked text dilutes the overall score because matching word choices become a smaller percentage of the total sample. The SynthID-Text paper outlines how detectors calculate these combined scores.

Document Composition Score Result
Mostly watermarked text + short human note Strong overall watermark signal
Short watermarked excerpt inside long human text Weak overall watermark signal
Watermarked excerpt tested separately Strong watermark signal

Key takeaways for mixed documents:

  • A positive watermark result on a mixed document does not mean AI wrote every paragraph.
  • A negative result on a mixed document does not mean the entire document is human-written.

How to read a detector score

Editing Action What Remains? What Changes? Overall Impact on Watermark
Exact Copying Wording and meaning File location or app Watermark remains intact
Minor Word Swaps Most wording and meaning Specific edited words Minor evidence loss; mark usually survives
Thorough Rewrite Core meaning Word choices and structure Significant evidence loss; mark weakens
Translation Core meaning Language and word sequence Significant evidence loss; mark weakens
Screen Line Wrapping Stored text and logic Screen layout only No impact; mark remains intact
Saved Code Formats Program execution Spaces, lines, and syntax Variable impact; may alter evidence
Mixed Documents Original text passages Proportion of marked text Dilutes whole-document score

Interpreting detector results accurately

Detector Result Accurate Interpretation Incorrect Assumption
Strong Watermark Signal The text closely matches the secret pattern. "Every word was written by AI."
Uncertain Result The detector lacks enough evidence for a clear decision. "Exactly half the document is AI-generated."
No Watermark Detected The detector did not find its specific pattern. "A human definitely wrote this text."

A negative result can happen for several reasons: the text was never watermarked, it came from a different AI model, it is too short, it lacks choice room, or it was heavily edited.

A positive result confirms the presence of a specific watermark pattern. It does not prove that an AI invented every idea, wrote every word, or stated true facts.

Conclusion

Code does not make text watermarking impossible, but it changes the conditions. Syntax, exact values, and strict formats leave fewer choices than explanations, comments, or alternative implementations. Editing changes the signal again, and the result depends on what changed - not on the vague label "edited."

When a detector result matters, test the relevant passage with the matching verifier and read the result within its stated scope. Part 3 - Can AI Watermark Images, Audio, and Video? shifts the question from word choices to pixels, sound measurements, and video frames.

References and further reading

Open the complete reference catalog

Primary Sources