Mention Rate Is a Proportion, Not a Fact

The Condition, Not the Person

I ended my last post saying I had been tuning longer than I had been measuring. I want to be clear about what that means. It is not a personal confession of negligence. It is the condition of the field. Almost everyone publishing numbers about AI visibility — how often a brand appears in model outputs, how often a page gets cited, how often a product surfaces in a recommendation — is reporting a sample statistic as a fact. They are taking twenty queries, counting eight mentions, and writing "40%" in a dashboard. The 40% is real. The precision implied by reporting it without an interval is not.

I know this because I once worked somewhere the discipline was enforced. The method was boring and it worked. Then the measurement problem got harder, and the field responded by getting less rigorous. That is the sentence this post is built around.

What the Work Actually Was

In 2008, I worked as a search quality rater for Google through Lionbridge. The programme exists, the guidelines are public, and the method is safe ground to describe. The specifics of my engagement — which queries, which locale, which period — are covered by confidentiality terms that do not expire. What I can describe is the method, because Google publishes it.

The current version of the Search Quality Rater Guidelines is 181 pages, dated January 23, 2025, version 10.1. The structure I worked under in 2008 was simpler but the bones were the same. A fixed set of queries. A published rubric. Multiple raters on the same queries. Periodic reports. The rubric told you what "needs met" meant and what "page quality" meant, and you applied it. You did not apply your opinion. You applied the rubric.

Three disciplines made it work:

You report what the rubric says, not what you think. If the rubric says a page "fully meets" a query and you think the page is terrible, you still mark it "fully meets." The rubric is the measurement instrument. Your opinion is not. This is the first thing the training teaches, and it is the first thing the AI visibility field has forgotten.

Disagreement between raters is data, not noise. Two raters look at the same query-result pair. One says "fully meets," the other says "slightly meets." That disagreement is not a problem to be resolved by averaging. It is a signal that the query is ambiguous or the rubric is unclear. Google tracks inter-rater agreement as a quality metric. If agreement drops, the rubric gets clarified. The disagreement itself is treated as a measurement of the measurement.

The query set is chosen before you look at results. You do not pick queries that you know will produce good results. You do not pick queries after seeing the output and curating the ones that support your hypothesis. The query set is frozen before any rating happens. This is the difference between an experiment and a sales pitch. I will come back to this.

What Changed, and What Got Worse

Then: an index returned ten links. The same ten links for everyone, the same ten links tomorrow. The measurement problem was hard but stable. You could run the same query next week and compare. The result set was deterministic.

Now: a model generates prose that varies by session, by phrasing, by temperature, by version, by who is asking. The same query can return a different answer in the same hour. The measurement problem got harder. The field responded by getting less rigorous. That is the sentence.

The old discipline was designed for a deterministic system and it was still hard to enforce. The new system is non-deterministic, and the discipline most people are applying to it is weaker than what Google applied to ten blue links in 2008. The tools got worse. The rigour collapsed. The dashboards got prettier.

The Arithmetic

Here is a number someone will put in a dashboard: "We appear in 40% of AI answers."

Twenty queries. Eight mentions. Eight divided by twenty is 0.40. That is the sample proportion. It is a real number. It is not a fact about the population.

The 95% Wilson score interval for eight successes out of twenty trials is 21.9% to 61.3%. I will show the calculation so you can check it.

The Wilson interval for a proportion is:

  p̂ + z²/(2n)         z · √(p̂(1-p̀)/n + z²/(4n²))
  ─────────────  ±  ─────────────────────────────
     1 + z²/n                1 + z²/n

Where p̂ = 8/20 = 0.40, n = 20, z = 1.96 (the 95% normal quantile).

  Denominator:  1 + 1.96²/20 = 1 + 0.19208 = 1.19208

  Centre:  (0.40 + 1.96²/40) / 1.19208
         = (0.40 + 0.09604) / 1.19208
         = 0.49604 / 1.19208
         = 0.4162

  Margin:  1.96 · √(0.40 × 0.60 / 20 + 1.96²/(4 × 400)) / 1.19208
         = 1.96 · √(0.012 + 0.002401) / 1.19208
         = 1.96 · √0.014401 / 1.19208
         = 1.96 · 0.12000 / 1.19208
         = 0.23521 / 1.19208
         = 0.1973

  Lower:  0.4162 - 0.1973 = 0.2189 → 21.9%
  Upper:  0.4162 + 0.1973 = 0.6135 → 61.3%

So the honest statement is: "We appear in between 21.9% and 61.3% of AI answers, with 95% confidence, based on twenty queries." That is what the data supports. "40%" is what the data suggests. The difference between "supports" and "suggests" is the difference between a measurement and a guess.

Now the part that lands hardest. To report "40%, plus or minus 5%" — a margin of error that actually narrows the interval to something useful — you need roughly 370 queries, not 20.

The sample size formula for a proportion at a given margin of error:

  n = z² · p(1-p) / E²
    = 1.96² × 0.40 × 0.60 / 0.05²
    = 3.8416 × 0.24 / 0.0025
    = 0.9220 / 0.0025
    ≈ 369

Almost nobody selling AI visibility reporting is running 370 queries. The dashboards I have seen report numbers based on 15 to 30. At n=20, your margin of error at 95% confidence is ±19 percentage points. The number in the dashboard has two digits of precision. The method supports zero.

Why Repeated Sampling Helps Less Than You’d Hope

Because model output is non-deterministic, the obvious fix is to run each query five times. Twenty queries times five runs is 100 observations. Problem solved, right?

No. Those are not 100 independent trials. They are 20 clusters of 5. The five runs of the same query share a cluster — the same query phrasing, the same model version, the same session context. A naïve confidence interval computed on 100 observations will be too narrow because it assumes 100 independent data points. The effective sample size is somewhere between 20 and 100, and without a clustering correction, you do not know where.

The fix is a cluster-robust standard error, or a hierarchical model that accounts for within-cluster correlation. The simpler fix is to report the number of distinct queries, not the number of runs, when stating the interval. The honest fix is to say: "We ran 20 queries, each 5 times, and observed mentions in 8 of 20 query clusters." That brings you back to a Wilson interval on n=20, which is 21.9% to 61.3% — the same wide interval you started with. Repeated sampling buys you stability within a query, not independence across queries.

This is the section that separates this post from most of what is written about AEO measurement. Most people do not know this. The ones who do know it do not mention it, because mentioning it makes their numbers worse. A dashboard that says "40% ± 19 points" does not sell. A dashboard that says "40%" does.

What a Defensible Protocol Looks Like

Here is a protocol that would produce numbers worth defending. Six rules, stated plainly:

1. Fix the query set before you look at any result. Write the queries down. Commit them to a file. Date the file. Do not change the queries after you have seen the outputs. If you add queries, start a new dataset. This is the single most important rule and it is the one most frequently broken.
2. State the model, the version, and the date. "GPT-4o, 2026-08-10" is a measurement. "AI answers" is not. Model behaviour changes between versions. A number measured against one model version does not apply to another. If you do not state the version, your number cannot be reproduced.
3. Repeat each query; report how many times. Non-deterministic output means a single run is a single draw from a distribution. Report the number of runs per query and the number of mentions per query. "8 of 20 queries produced at least one mention in 5 runs" is a different statement from "8 of 100 runs produced a mention." Both are valid. They answer different questions.
4. Report the proportion with an interval, and say which interval. Wilson, Clopper-Pearson, Agresti-Coull — they are all defensible. "40%" with no interval is not. State the method, the confidence level, and the bounds. If the interval is wide, the number is uncertain. Say so.
5. Log the URLs the model cited, not just whether you appeared. A mention without a citation is a statement. A mention with a citation is a signal. The URL the model chose to cite is the thing you can act on — it tells you which source the model is retrieving. If you only track "did we appear," you are measuring visibility without measuring the mechanism that produces it.
6. Never report a number more precise than the method supports. If your sample size is 20, report "between 22% and 61%." If your sample size is 370, report "40% ± 5%." The number of decimal places is a claim about precision. Do not make a claim the method cannot back.

The Uncomfortable Finding

When you run a protocol like this — when you freeze the query set, log the citations, and look at what actually moves the answer — the finding is uncomfortable. What moves the answer is usually which third-party sources get retrieved. Not your page. Not your schema. Not your content. The model cites a source it has already retrieved, and that source mentions you or it does not.

Your own page is rarely the lever. The lever is whether the model retrieves a source that has already mentioned you. This is a retrieval problem, not a content problem. It is also not something you can directly control — you can control your page, you can control your schema, you cannot control which review site or aggregator or directory the model decides to pull from for a given query.

Nobody selling an AEO dashboard wants to say this out loud. The dashboards imply that the lever is your content, your schema, your structured data. If the lever is a third-party source you do not control, the dashboard’s recommended actions — add more JSON-LD, restructure your headings, write more FAQ pages — are not addressing the thing that actually moves the number. Which is a reason to say it.

What Schema Does and Doesn’t Do

Structured data — JSON-LD, microdata, schema.org vocabulary — does two things well. It aids entity resolution: telling a model that "Apple" on this page refers to the company, not the fruit. And it aids disambiguation: telling a model that this page is about a product, this is the price, this is the availability. These are real functions. I use structured data on my own sites. It is not useless.

What structured data does not do is cause citation. Google has been explicit about this. John Mueller stated it publicly: "There’s no generic ranking boost for SD [structured data] usage… SD can make it easier to understand what the page is about, which can make it easier to show where it’s relevant (improves targeting, maybe ranking for the right terms)." Google’s Search Central documentation says structured data "helps Google understand the content of the page" and can make a page "eligible for rich results." Understanding and eligibility are not citation. Understanding is what the model does with the content. Eligibility is whether the content qualifies for a feature. Neither is a mechanism that causes a model to cite your page in a generated answer.

I have not found a published controlled study that links JSON-LD implementation to mention rate in LLM outputs. There are case studies. There are vendor whitepapers. There is no study that holds content constant, varies structured data, and measures the effect on citation frequency in model outputs with a confidence interval. The absence of this study is not proof that schema does not help. It is proof that the claim "schema improves AI visibility" is not supported by the evidence standard the field should be holding itself to.

This is consistent with the uncomfortable finding above. If what moves the answer is which third-party sources get retrieved, then adding schema to your own page addresses a different layer of the stack than the one that determines whether you get cited. Schema helps a model understand your page. Retrieval determines whether the model reads your page. These are different problems.

Close

Rater discipline was boring and it worked. You reported what the rubric said. You treated disagreement as data. You froze the query set before you looked. The method was not exciting. The numbers were defensible.

The measurement problem is harder now. Models are non-deterministic, outputs vary by session, and the population you are sampling from shifts every time a model gets updated. That is precisely why the discipline matters more, not less. A deterministic system forgives sloppy method because you can repeat the query and check. A non-deterministic system does not forgive it because you cannot. The number you measured today may not be the number tomorrow, and without an interval, you cannot tell the difference between sampling noise and a real change.

Report the interval. State the model and the date. Freeze the query set. If your method cannot support the precision you are claiming, your number is not a fact. It is a sample statistic wearing a fact’s clothes.

A full study, with a frozen query set, a published protocol, and per-store results — is coming. This post teaches the method. It does not publish the results.

The discipline I learned as a search quality rater in 2008 was simple: report what the rubric says, treat disagreement as data, freeze the query set before you look. The system we were measuring was deterministic — ten links, same for everyone, same tomorrow. The discipline was still hard to enforce. Now the system is non-deterministic, and the discipline most of the field applies is weaker than what Google used for ten blue links.

The math has not changed. Eight mentions out of twenty queries is a 40% sample proportion with a 95% Wilson interval of 21.9% to 61.3%. To narrow that to ±5%, you need roughly 370 queries. Almost nobody running AI visibility dashboards is doing that. Repeated sampling helps less than you would hope because runs of the same query are clustered, not independent. The honest interval is still wide.

What actually moves the answer is usually which third-party sources the model retrieves, not your own page or your schema. Structured data aids entity resolution and disambiguation — real functions — but Google’s own public statements say it aids understanding and feature eligibility, not ranking. No published controlled study links JSON-LD to mention rate in LLM outputs. The claim is not supported by the evidence standard the field should be holding itself to.

Report the interval. State the model and the date. Freeze the query set. A full study is coming. This post teaches the method.