The pattern in one line
You ask the same question each week and the AI answer names different brands, in a different order, each time.
AI answers are generated fresh each time, so wording, sources, and brands can change between runs. One check is a sample, not a measurement. Read the trend across repeated checks, treat a steady mention rate as the signal, and act on movement that holds across several checks instead of a single swing.
What this looks like
Check a prompt on Monday and you are the second brand named. Check it on Friday and you are missing. Check it again on Sunday and you are first. Nothing on your site changed.
This is normal behavior for AI answers. The engine writes a new response each time, and it may retrieve a different set of sources, weigh them differently, or phrase the list another way. Engines also update their models and retrieval systems over time. A single answer is a sample.
The trap is reacting to one sample. A good Monday makes a report look strong, and a bad Friday sets off a rewrite. Neither says much. What matters is how often you appear across many checks and whether that rate is moving.
Tracking the same prompt over time is what turns noise into a signal. A rate that stays within a narrow band is stable. A rate that climbs for several checks in a row after you changed something is worth crediting. A single high or low reading belongs in neither group.
A brand that appears in every run is in a different position from a brand that appears in 1 run out of 4, and both can look the same in a screenshot.
Common traits
- The same prompt returns a different list of brands from run to run
- Your name appears in some runs and not in others
- Order and wording shift even when the facts stay the same
- Cited sources rotate
- A single check makes you look better or worse than the trend
Why it happens
-
Answers are generated each time
Wording and ordering can differ on every run, even for the same prompt on the same day.
-
Retrieval picks different sources
Engines that search the web can return different results from one check to the next, so the sources behind the answer shift.
-
Models and systems get updated
Engines change over time, and behavior you measured last month may not match this month.
-
Your position is marginal
A brand at the edge of what an engine considers drops in and out. A brand with strong, repeated support appears in nearly every run.
How to confirm it in Doily
- Open AI visibility and look at the mention rate for 1 prompt across its history instead of the latest response.
- Paid plans check prompts daily, and the product prints the cadence next to each engine's numbers, so you can see how many checks sit behind a figure.
- Open individual responses from a good day and a bad day and compare the cited sources.
- Check whether the swing appears on every engine or only 1.
Fixes to try
| Fix | Impact | Effort |
|---|---|---|
| Report mention rate across many checks instead of single answers | High | Low |
| Track more prompts per topic so 1 odd run does not drive decisions | Medium | Low |
| Wait for a change to hold across several checks before rewriting pages | Medium | Low |
| Note site changes, launches, and campaigns on your reports so movement can be traced to a cause | Medium | Low |
| Build stronger, repeated support for your brand across sources so you appear in most runs | High | High |
What to watch after
- Mention rate over the last several checks
- The spread between your best and worst week
- Which prompts are steady and which swing
- Cited sources that appear in most runs
Questions
How many checks do I need before trusting a number?
There is no fixed count. More checks narrow the uncertainty, and a rate based on a few runs can swing a lot. Compare periods that have similar numbers of checks.
Is a changing answer a sign that something broke?
Usually not. Variation is part of how these answers are generated. Look for changes that persist across checks.
Should I screenshot answers for stakeholders?
A screenshot shows 1 run. Pair it with the rate over time so the reader sees the pattern and not a single sample.
