Search Quality Rater Guidelines<\/a> is 181 pages, dated January 23, 2025, version 10.1. The structure I worked under in 2008 was simpler but the bones were the same. A fixed set of queries. A published rubric. Multiple raters on the same queries. Periodic reports. The rubric told you what "needs met" meant and what "page quality" meant, and you applied it. You did not apply your opinion. You applied the rubric.<\/p>\nThree disciplines made it work:<\/p>\n
You report what the rubric says, not what you think.<\/strong> If the rubric says a page "fully meets" a query and you think the page is terrible, you still mark it "fully meets." The rubric is the measurement instrument. Your opinion is not. This is the first thing the training teaches, and it is the first thing the AI visibility field has forgotten.<\/p>\nDisagreement between raters is data, not noise.<\/strong> Two raters look at the same query-result pair. One says "fully meets," the other says "slightly meets." That disagreement is not a problem to be resolved by averaging. It is a signal that the query is ambiguous or the rubric is unclear. Google tracks inter-rater agreement as a quality metric. If agreement drops, the rubric gets clarified. The disagreement itself is treated as a measurement of the measurement.<\/p>\nThe query set is chosen before you look at results.<\/strong> You do not pick queries that you know will produce good results. You do not pick queries after seeing the output and curating the ones that support your hypothesis. The query set is frozen before any rating happens. This is the difference between an experiment and a sales pitch. I will come back to this.<\/p>\nWhat Changed, and What Got Worse<\/h2>\n Then: an index returned ten links. The same ten links for everyone, the same ten links tomorrow. The measurement problem was hard but stable. You could run the same query next week and compare. The result set was deterministic.<\/p>\n
Now: a model generates prose that varies by session, by phrasing, by temperature, by version, by who is asking. The same query can return a different answer in the same hour. The measurement problem got harder. The field responded by getting less rigorous. That is the sentence.<\/p>\n
The old discipline was designed for a deterministic system and it was still hard to enforce. The new system is non-deterministic, and the discipline most people are applying to it is weaker than what Google applied to ten blue links in 2008. The tools got worse. The rigour collapsed. The dashboards got prettier.<\/p>\nThen: ten links, same for everyone, same tomorrow. Now: generated prose that varies by session. The measurement problem got harder. The field got less rigorous.<\/figcaption><\/figure>\nThe Arithmetic<\/h2>\n Here is a number someone will put in a dashboard: "We appear in 40% of AI answers."<\/p>\n
Twenty queries. Eight mentions. Eight divided by twenty is 0.40. That is the sample proportion. It is a real number. It is not a fact about the population.<\/p>\n
The 95% Wilson score interval for eight successes out of twenty trials is 21.9% to 61.3%. I will show the calculation so you can check it.<\/p>\n
The Wilson interval for a proportion is:<\/p>\n
p̂ + z\u00b2\/(2n) z · √(p̂(1-p̀)\/n + z\u00b2\/(4n\u00b2))\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 ± \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n 1 + z\u00b2\/n 1 + z\u00b2\/n<\/code><\/pre>\nWhere p̂ = 8\/20 = 0.40, n = 20, z = 1.96 (the 95% normal quantile).<\/p>\n
Denominator: 1 + 1.96\u00b2\/20 = 1 + 0.19208 = 1.19208\n\n Centre: (0.40 + 1.96\u00b2\/40) \/ 1.19208\n = (0.40 + 0.09604) \/ 1.19208\n = 0.49604 \/ 1.19208\n = 0.4162\n\n Margin: 1.96 · √(0.40 × 0.60 \/ 20 + 1.96\u00b2\/(4 × 400)) \/ 1.19208\n = 1.96 · √(0.012 + 0.002401) \/ 1.19208\n = 1.96 · √0.014401 \/ 1.19208\n = 1.96 · 0.12000 \/ 1.19208\n = 0.23521 \/ 1.19208\n = 0.1973\n\n Lower: 0.4162 - 0.1973 = 0.2189 → 21.9%\n Upper: 0.4162 + 0.1973 = 0.6135 → 61.3%<\/code><\/pre>\nSo the honest statement is: "We appear in between 21.9% and 61.3% of AI answers, with 95% confidence, based on twenty queries." That is what the data supports. "40%" is what the data suggests. The difference between "supports" and "suggests" is the difference between a measurement and a guess.<\/p>\n
Now the part that lands hardest. To report "40%, plus or minus 5%" \u2014 a margin of error that actually narrows the interval to something useful \u2014 you need roughly 370 queries, not 20.<\/p>\n
The sample size formula for a proportion at a given margin of error:<\/p>\n
n = z\u00b2 · p(1-p) \/ E\u00b2\n = 1.96\u00b2 × 0.40 × 0.60 \/ 0.05\u00b2\n = 3.8416 × 0.24 \/ 0.0025\n = 0.9220 \/ 0.0025\n ≈ 369<\/code><\/pre>\nAlmost nobody selling AI visibility reporting is running 370 queries. The dashboards I have seen report numbers based on 15 to 30. At n=20, your margin of error at 95% confidence is ±19 percentage points. The number in the dashboard has two digits of precision. The method supports zero.<\/p>\n
Why Repeated Sampling Helps Less Than You’d Hope<\/h2>\n Because model output is non-deterministic, the obvious fix is to run each query five times. Twenty queries times five runs is 100 observations. Problem solved, right?<\/p>\n
No. Those are not 100 independent trials. They are 20 clusters of 5. The five runs of the same query share a cluster \u2014 the same query phrasing, the same model version, the same session context. A naïve confidence interval computed on 100 observations will be too narrow because it assumes 100 independent data points. The effective sample size is somewhere between 20 and 100, and without a clustering correction, you do not know where.<\/p>\n
The fix is a cluster-robust standard error, or a hierarchical model that accounts for within-cluster correlation. The simpler fix is to report the number of distinct queries, not the number of runs, when stating the interval. The honest fix is to say: "We ran 20 queries, each 5 times, and observed mentions in 8 of 20 query clusters." That brings you back to a Wilson interval on n=20, which is 21.9% to 61.3% \u2014 the same wide interval you started with. Repeated sampling buys you stability within a query, not independence across queries.<\/p>\n
This is the section that separates this post from most of what is written about AEO measurement. Most people do not know this. The ones who do know it do not mention it, because mentioning it makes their numbers worse. A dashboard that says "40% ± 19 points" does not sell. A dashboard that says "40%" does.<\/p>\n
What a Defensible Protocol Looks Like<\/h2>\n Here is a protocol that would produce numbers worth defending. Six rules, stated plainly:<\/p>\n
\nFix the query set before you look at any result.<\/strong> Write the queries down. Commit them to a file. Date the file. Do not change the queries after you have seen the outputs. If you add queries, start a new dataset. This is the single most important rule and it is the one most frequently broken.<\/li>\nState the model, the version, and the date.<\/strong> "GPT-4o, 2026-08-10" is a measurement. "AI answers" is not. Model behaviour changes between versions. A number measured against one model version does not apply to another. If you do not state the version, your number cannot be reproduced.<\/li>\nRepeat each query; report how many times.<\/strong> Non-deterministic output means a single run is a single draw from a distribution. Report the number of runs per query and the number of mentions per query. "8 of 20 queries produced at least one mention in 5 runs" is a different statement from "8 of 100 runs produced a mention." Both are valid. They answer different questions.<\/li>\nReport the proportion with an interval, and say which interval.<\/strong> Wilson, Clopper-Pearson, Agresti-Coull \u2014 they are all defensible. "40%" with no interval is not. State the method, the confidence level, and the bounds. If the interval is wide, the number is uncertain. Say so.<\/li>\nLog the URLs the model cited, not just whether you appeared.<\/strong> A mention without a citation is a statement. A mention with a citation is a signal. The URL the model chose to cite is the thing you can act on \u2014 it tells you which source the model is retrieving. If you only track "did we appear," you are measuring visibility without measuring the mechanism that produces it.<\/li>\nNever report a number more precise than the method supports.<\/strong> If your sample size is 20, report "between 22% and 61%." If your sample size is 370, report "40% ± 5%." The number of decimal places is a claim about precision. Do not make a claim the method cannot back.<\/li>\n<\/ol>\nThe Uncomfortable Finding<\/h2>\n When you run a protocol like this \u2014 when you freeze the query set, log the citations, and look at what actually moves the answer \u2014 the finding is uncomfortable. What moves the answer is usually which third-party sources get retrieved. Not your page. Not your schema. Not your content. The model cites a source it has already retrieved, and that source mentions you or it does not.<\/p>\n
Your own page is rarely the lever. The lever is whether the model retrieves a source that has already mentioned you. This is a retrieval problem, not a content problem. It is also not something you can directly control \u2014 you can control your page, you can control your schema, you cannot control which review site or aggregator or directory the model decides to pull from for a given query.<\/p>\n
Nobody selling an AEO dashboard wants to say this out loud. The dashboards imply that the lever is your content, your schema, your structured data. If the lever is a third-party source you do not control, the dashboard’s recommended actions \u2014 add more JSON-LD, restructure your headings, write more FAQ pages \u2014 are not addressing the thing that actually moves the number. Which is a reason to say it.<\/p>\n
What Schema Does and Doesn’t Do<\/h2>\n Structured data \u2014 JSON-LD, microdata, schema.org vocabulary \u2014 does two things well. It aids entity resolution: telling a model that "Apple" on this page refers to the company, not the fruit. And it aids disambiguation: telling a model that this page is about a product, this is the price, this is the availability. These are real functions. I use structured data on my own sites. It is not useless.<\/p>\n
What structured data does not do is cause citation. Google has been explicit about this. John Mueller stated it publicly: "There’s no generic ranking boost for SD [structured data] usage… SD can make it easier to understand what the page is about, which can make it easier to show where it’s relevant (improves targeting, maybe ranking for the right terms)." Google’s Search Central documentation says structured data "helps Google understand the content of the page" and can make a page "eligible for rich results." Understanding and eligibility are not citation. Understanding is what the model does with the content. Eligibility is whether the content qualifies for a feature. Neither is a mechanism that causes a model to cite your page in a generated answer.<\/p>\n
I have not found a published controlled study that links JSON-LD implementation to mention rate in LLM outputs. There are case studies. There are vendor whitepapers. There is no study that holds content constant, varies structured data, and measures the effect on citation frequency in model outputs with a confidence interval. The absence of this study is not proof that schema does not help. It is proof that the claim "schema improves AI visibility" is not supported by the evidence standard the field should be holding itself to.<\/p>\n
This is consistent with the uncomfortable finding above. If what moves the answer is which third-party sources get retrieved, then adding schema to your own page addresses a different layer of the stack than the one that determines whether you get cited. Schema helps a model understand your page. Retrieval determines whether the model reads your page. These are different problems.<\/p>\n
Close<\/h2>\n Rater discipline was boring and it worked. You reported what the rubric said. You treated disagreement as data. You froze the query set before you looked. The method was not exciting. The numbers were defensible.<\/p>\n
The measurement problem is harder now. Models are non-deterministic, outputs vary by session, and the population you are sampling from shifts every time a model gets updated. That is precisely why the discipline matters more, not less. A deterministic system forgives sloppy method because you can repeat the query and check. A non-deterministic system does not forgive it because you cannot. The number you measured today may not be the number tomorrow, and without an interval, you cannot tell the difference between sampling noise and a real change.<\/p>\n
Report the interval. State the model and the date. Freeze the query set. If your method cannot support the precision you are claiming, your number is not a fact. It is a sample statistic wearing a fact’s clothes.<\/p>\n
A full study, with a frozen query set, a published protocol, and per-store results \u2014 is coming. This post teaches the method. It does not publish the results.<\/p>\n