A check that could never pass

For months, one of our GEO checks could not return a pass for any site on earth. Not "rarely returned one". Could not. Three independent bugs, each sufficient on its own, and every one of them hid behind an HTTP 200.

The check is [GEO] AI Search Presence. It asks Gemini a question with Google Search grounding turned on, reads the sources the model actually grounded its answer in, and reports whether the audited domain is among them. It is the check most directly about the thing this company exists to measure.

The three, in the order we found them:

1. The field we read as a source URL is a redirect proxy, and never contains the domain.
2. The output token ceiling is shared with thinking tokens, so the response often carried no grounding data at all.
3. The matcher used substring comparison, which is wrong for domains in both directions.

Bug one: the URL is not the URL

Gemini returns its grounding sources as groundingChunks[].web.uri. That reads like a source URL. It is a redirect proxy, and it looks like this:

https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQ...

It never contains the cited domain. The domain is in a different field, web.title, which holds the plain string stripe.com. Our matcher compared the audited host against the URIs, so it could not match for any site, ever. The check had never once returned a pass in its life.

Bug two: thinking tokens come out of the same budget

We had set maxOutputTokens: 400, which is generous for a one-paragraph answer. On a Gemini 3 model that ceiling is shared with thinking tokens. The model spent roughly 380 of them thinking, hit finishReason: MAX_TOKENS mid-sentence, and returned no grounding metadata at all.

So even with the matcher fixed, there was frequently nothing to match against. The response was still a 200. The fix was thinkingConfig: { thinkingBudget: 0 } and a ceiling of 2000, but the lesson is the diagnostic one: check finishReason and usageMetadata.thoughtsTokenCount, never just whether the request succeeded.

Bug three: substring matching, on domains

The matcher used String.includes. Consider what that means for a short domain:

'netflix.com'.includes('x.com')      // true
'microsoft.com'.includes('t.co')     // true

Both x.com and t.co are real, popular domains. So the first two bugs made the check impossible to pass, and this one would have made it pass incorrectly the moment they were fixed, for exactly the domains a ranked sample is full of. Worse: auditing google.com matches vertexaisearch.cloud.google.com, the proxy host from bug one, and passes unconditionally.

Match on domain boundaries instead: equal, or ending in a dot plus the host. And exclude the grounding proxy host explicitly, because it is not a citation of anything.

Why did none of this get noticed?

Production has no Gemini API key, deliberately: grounded search requires billing and the check runs on anonymous audits, so a production key means strangers spend your money. With no key the check returns not-applicable and is excluded from scoring. A defect that only manifests where nobody is looking is invisible for as long as nobody looks.

But that is the small reason. The large one is that a check which can only ever return one value is indistinguishable from a working check. There is no error, no exception, no anomalous log line. There is a verdict, and it is a plausible verdict, and it is the same one every time.

So how do you catch this class of bug?

Run any new or changed check across roughly ten varied real sites and flag anything that comes back 100% pass, 100% fail, or 100% not-applicable. That is it. All three of the above would have been caught in an afternoon by a loop.

Two things make it work in practice. All-pass matters as much as all-fail: a uniform pass is what a too-loose matching rule looks like, and it is the direction nobody checks because it does not feel like a bug. And every external dependency needs a positive control, meaning a subject whose correct answer you already know. Stripe is cited by Google when you ask about Stripe. If the run says otherwise, the run is wrong.

That last point generalizes past APIs. A hardcoded third-party identifier is an external dependency too, and we learned that one separately and later: our AI-crawler check spent months looking for a Google robots.txt token that does not exist, so it had never detected a real Google opt-out either. Same shape, different surface.

What we changed

The matcher compares domain boundaries and reads web.title. The token budget accounts for thinking. The collector will not spend two hours on a run until a positive control comes back correct first. And the three failure modes above are pinned by regression tests, because the thing about a check that can only return one value is that it will happily go on returning it.

The aggregate result looked like a finding about the web. It was a fact about our own code. Any number you are about to publish deserves the question: could this instrument have produced a different answer?

Sources

• Google Search Central: Google crawlers and fetchers, for the Google-Extended robots.txt token and what it controls. developers.google.com
• The behavior of groundingChunks[].web.uri and web.title described above is our own observation against the live API, not a quotation from Google. We are stating it as a measurement because that is what it is.
• The dataset these checks produced, with every raw record and the queries we asked, is published under a DOI alongside the study itself.