A growing commercial question has an awkward statistical shape: how often does a large language model recommend a given brand when asked a category question?
Marketing teams want a number. The obvious approach, ask the model and write down the answer, produces a number that is close to meaningless. Ask again and it changes.
This is a sampling problem wearing a marketing hat, and it is worth treating properly, because the industry that has grown up around it largely does not.
Why a single query tells you almost nothing
Inference in production is non-deterministic in several compounding ways.
Decoding is stochastic. Non-zero temperature and top-p sampling mean the same prompt yields different token sequences.
Retrieval is unstable. Engines with live search retrieve a document set that varies with index state, geography, personalisation and time of day. Different context produces different answers even with identical decoding.
The model behind the endpoint moves. Providers update weights, system prompts and routing without version bumps visible to the caller. A time series collected over months is not measuring a fixed object.
Position bias is real. Order of brands within an answer is not stable, and if your metric weights position, that adds variance on top of presence variance.
The practical consequence is that “does ChatGPT recommend us” has no scalar answer. The measurable quantity is a recommendation rate: the proportion of sampled responses to a defined prompt set in which the brand appears, with a confidence interval attached.
Defining the estimand before measuring it
Most published AI visibility numbers skip this step, which is why they are not comparable across vendors.
Specify, at minimum:
- The prompt set. Category questions, fixed in wording, versioned. Changing a prompt changes the estimand.
- The engine and configuration. Model identifier, temperature, whether live retrieval is enabled, geography.
- The presence criterion. Exact brand string, normalised variants, or an entity-resolution step? “Honeyb”, “honeyb.ai” and “Honeyb AI” are the same brand and different strings.
- The sampling window. Answers drift within a day. A rate is a rate over a stated interval.
Two vendors reporting “38% visibility” for the same brand are almost certainly measuring different things.
How many samples
Treat presence in a single response as a Bernoulli trial with unknown p. The normal-approximation half-width is
half-width ≈ z · sqrt( p(1-p) / n )
At the worst case p = 0.5 and 95% confidence:
| Samples per prompt | Half-width | Practical reading |
| 10 | ±31 pp | Not a measurement |
| 30 | ±18 pp | Detects only enormous changes |
| 100 | ±10 pp | Detects large changes |
| 400 | ±5 pp | Detects a meaningful shift |
| 1,000 | ±3 pp | Detects a campaign effect |
Two things follow immediately.
First, sampling budgets are usually set by price rather than by power. Before buying any tool, work out how many samples per prompt its tier actually gives you per period, then read the table above. Most entry tiers cannot resolve anything smaller than a total repositioning.
The vendors are not wrong about this. The dashboards simply imply a precision the sampling does not support.
Second, sampling budget should be spent on repetition, not breadth. Ten prompts sampled thirty times each supports inference. Three hundred prompts sampled once each does not, however impressive the coverage sounds.
Use the Wilson interval rather than the normal approximation at small n or extreme p. Brand presence rates near zero are exactly where the normal approximation misbehaves. Near zero is where most brands start.
The Honeyb AI visibility checker implements this across eight engines, storing raw responses and reporting rates with intervals rather than point estimates. Both are unusual in commercial tooling, and they are the two components that decide whether a dashboard number can support a decision.
Clustering, and why naive intervals are too narrow
Responses sampled within a short window share retrieval context, so they are not independent. Treating them as independent understates variance.
The correction is standard: treat samples within a prompt-day as a cluster, compute a design effect, and inflate the interval by its square root. In our own runs the design effect sits meaningfully above 1, which means naive intervals are optimistic by a non-trivial factor.
Practically: spread samples across days rather than firing them in a burst. A hundred samples over ten days carries more information than a hundred samples in ten minutes.
What we are not publishing here
It would be easy to close this section with our own variance figures. We are not going to, because the run they came from is not one we can currently point you at, and an unsourced percentage in a methodology article is worth less than nothing.
What we can say is directional and reproducible by anyone: in repeated sampling against fixed prompts, the brand occupying the top recommended position is not stable across runs on the same day, and presence of any given brand moves by more than the width of most published confidence intervals. If you want a number for your own category, the protocol below produces one in about a week.
Report the interval, not the point estimate. A dashboard rendering “42%” with no interval is asserting a precision the data does not contain.
A minimum defensible protocol
- Version the prompt set and treat any edit as a new series.
- Sample each prompt at least thirty times per period, spread across days.
- Record engine, configuration, geography and timestamp with every response.
- Normalise brand strings through an entity-resolution step before counting.
- Report Wilson intervals with a cluster-adjusted design effect.
- Re-baseline when a provider ships a model update, because the series is not continuous across it.
- Store raw responses, not just the derived rate. Every metric you have not thought of yet is recoverable from raw text and from nothing else.
That last point is the one most implementations regret skipping.
Why this is worth the rigour
The commercial stakes are not trivial. A buyer asks an engine for a shortlist and receives three names. Presence is close to binary in its business effect.
The decisions made off these numbers are expensive: budget reallocation, content investment, repositioning. A measurement that cannot separate a real change from sampling noise will reliably produce confident wrong decisions.
The rest of the product follows from the same position. Measuring carefully is only useful if something happens next. So the measurement feeds a technical audit of whether the engines can retrieve the site, recommendations that carry the statistic that produced them, and a content pipeline that ships the resulting article. It is the only tool in the category that goes past the measurement, which is a strange thing to be able to say about a category whose entire purpose is to change an outcome.
Frequently asked
Can I just set temperature to zero? It reduces decoding variance and does not remove retrieval variance, which dominates for engines with live search. It also measures a configuration your buyers are not using.
Is a recommendation rate comparable across engines? Only within a fixed prompt set and presence criterion. Cross-engine comparison of absolute rates is weak; comparison of trends within an engine is much stronger.
How do I detect a model update? Watch for step changes in the rate that coincide across unrelated prompts. A shift affecting every prompt simultaneously is a provider change, not a marketing result.
What sample size for a campaign readout? To detect a 10 percentage point shift with 95% confidence you need roughly 400 samples per arm before clustering adjustment, and more after it.
Where is the implementation? At honeyb.ai, with raw responses stored per run.