AI answers vary because the systems that write them are built to produce a fresh response each time, because the pages they retrieve change, and because location and settings can feed into what is retrieved. Public research shows large variation between runs and between days. Measuring a single answer once tells you almost nothing.
Run to run
The cleanest evidence comes from SparkToro and Gumshoe.ai. Volunteers ran 12 brand-recommendation prompts through ChatGPT, Claude, and Google’s AI features, each prompt 60 to 100 times per tool, for 2,961 runs in total. The authors found a less than 1 in 100 chance that ChatGPT or Google’s AI would return the same list of brands in any 2 responses to the same prompt. Claude was slightly more consistent, but even less likely to repeat the order. The number of brands listed also varied, from 2 or 3 up to 10 or more.
The authors disclose that Gumshoe.ai sells AI tracking tools, and they say their own start was skepticism. They still concluded that how often a brand appears across many runs is informative, while its position in any one list is not worth tracking.
Ahrefs saw the same thing in Google’s AI Overviews. It captured 2 Overviews for the query “renters insurance” two minutes apart in incognito mode, and the phrasing, examples, and structure differed. Ahrefs also says popular Overviews are not cached, so each view can be generated fresh.
Day to day
Ahrefs studied more than 43,000 keywords, each with at least 16 recorded AI Overviews over a month. It reported:
- A 70% chance that an Overview changed from one observation to the next.
- A typical lifespan of 2.15 days before the content changed. Because the checks were not daily, Ahrefs says the real rate is likely higher.
- 54.5% of cited URLs overlapped between consecutive responses, so about 45.5% of citations were new each time.
- A semantic consistency score of 0.95 out of 1.0 between consecutive responses.
That last figure is the important caveat to the first 3. The meaning of the answer was stable. What moved was the wording, the examples, and which pages were cited in support. Ahrefs sells an AI visibility tool built on this kind of data, which is worth keeping in mind, but the method is described and the finding fits SparkToro’s.
For a brand, the consequence is uncomfortable. Being cited today does not mean being cited tomorrow, even when the answer itself did not change.
Engine to engine
Different engines are different systems, and even Google’s own 2 features differ. Google says AI Overviews and AI Mode “may use different models and techniques, so the set of responses and links they show will vary.” Ahrefs compared 540,000 query pairs from September 2025 and found the 2 features cited the same URLs only 13.7% of the time, even though the answers were 86% similar in meaning.
Place to place
Google’s page on how results are ranked says Search uses your location, past search history, and settings to decide what is most relevant. Its example is a search for “football”: someone in Chicago will likely see American football and the Chicago Bears, while someone in London will see soccer and the Premier League. That page describes Search in general and does not say how much of it carries into AI features, so treat it as a reason to test and not as a measured effect.
On the OpenAI side, the developer documentation for its web search tool lets a developer pass an approximate user location, as country, city, region, and timezone, to refine results by geography. That is an input, not a finding about the ChatGPT app. It does show that location is a designed-in variable.
For any business that serves a place, the sensible assumption is that “best plumber” means different things in 2 cities, and that an answer checked from a data center in another state may not be the answer your customer sees.
Device and signed-in state
We could not find a public study that isolates the effect of device or login state on AI answers. Google’s ranking page lists search history and settings as inputs for Search, which suggests signed-in state is a variable worth controlling. Beyond that, the honest position is that device and account state are unmeasured by the public research above. Hold them constant, write them down, and compare like with like.
What to do about it
- Sample, do not snapshot. Run each prompt several times per period and report how often you appear, as a rate with the number of runs beside it.
- Fix the conditions. Same location, same device type, same signed-out or signed-in state, and the same engines every time.
- Compare over weeks. A one-day change in citations is within the normal churn that Ahrefs describes. A change that persists over several weeks is more likely real.
- Track mentions and citations separately. The mention can hold steady while the cited page changes.
- Record the date and place with every result. Without them you cannot tell a real shift from a different day or a different city.
This is how Doily is built. It runs prompts on a schedule, prints the check cadence next to every engine’s numbers, and lets you set the country or city for a prompt. Paid plans check prompts daily, which gives you the repeated samples the research above says you need. Our guide on tracking AI prompts covers setting up the sample, and the pattern in AI answers change weekly shows how to tell churn from a real shift.
What a good report line looks like
Compare these 2 sentences. “Our site was cited by Google AI Overviews yesterday” describes one run, and the next run may disagree. “Our site was cited in 7 of 20 runs this week, 35%, against 5 of 20 last week, 25%” gives a rate, the sample behind it, and a comparison. The second still carries uncertainty, since 20 runs is small, but it is honest about it and it can be repeated.
The numbers in that example are invented to show the format. The habit is the point: every figure you show a client or a boss should come with the count of runs, the dates, and the place.
What this does not mean
Variation does not make AI visibility unmeasurable. It means the measure is a rate over many runs, and it needs enough runs before you act on a change. Weather forecasts are uncertain day to day, and people still read them. The skill is knowing how wide the uncertainty is, and not mistaking one run for a trend.
