Why ChatGPT Gives Different Answers to the Same Question

ChatGPT samples its answers, so the same prompt can return a different list each time. In a SparkToro study from November to December 2025, the chance that ChatGPT or Google's AI returned the same brand list in two responses was under 1 in 100. A single answer is one sample, so the reliable measure is how often a brand appears across many runs.

What matters most

• Identical brand lists are rare, and identical lists in the same order are rarer still.
• Temperature 0 does not guarantee identical output, so treat variation as a given.
• Wording and web search add variation on top of sampling. Keep prompts fixed, and report presence and rank separately.
• Run each prompt many times, SparkToro suggests 60 to 100, and report a rate with its sample size, for example the number of runs in which a brand appeared out of the total.

Why does the same prompt return a different list each time?

You ask ChatGPT for recommendations and copy the list. The next day you ask again and the list has changed. Nothing broke. This is how the tool behaves.

SparkToro put the problem in one line:

"there's a <1 in 100 chance that ChatGPT or Google's AI, if asked 100X, will give you the same list of brands" — SparkToro, in its 2025 study

The study covered 2,961 runs. Nearly every response to a recommendation prompt differed in three ways: the brands listed, their order and the number of items on the list. Identical lists in the same order were far rarer, at under 1 in 1,000 runs, whatever the question.

That makes a rank taken from one run unreliable. SparkToro concludes that visibility percentage across many prompts, each run multiple times, is a reasonable metric, while ranking positions from single runs are not. One response is one draw from a distribution of possible answers. It says little about your brand on its own.

The method matters for how you read these results. SparkToro's volunteers ran prompts through their normal, default setups and kept their usual settings, with no temperature changes. The numbers therefore reflect what an everyday user sees.

What to do: treat each answer as one sample and count how often your brand appears across many runs.

Is sampling the main cause?

Sampling is one of the main mechanisms. The model picks each next token from a probability distribution, and temperature shapes that distribution. Lower values bias the model toward the most probable token and produce more consistent output, according to a 2026 arXiv paper on LLMs in qualitative research. The same paper notes that many chat interfaces default to intermediate temperature values, often around 0.5, and do not clearly expose the setting. One vendor says the consumer ChatGPT interface does not expose temperature or top_p, which only the API and Playground provide.

Setting temperature to 0 in a tool that allows it does not close the gap. A 2025 arXiv paper notes that even at temperature 0, the most deterministic setting, hardware factors such as GPU concurrency can prevent determinism.

Matching every parameter does not guarantee a match either. OpenAI's cookbook says there is a small chance that responses differ even when the request parameters and the system fingerprint match. The inherent non-determinism of the models explains it.

The variation can affect substance as well as phrasing. A 2023 arXiv study of ChatGPT on software-testing questions found that for 9.7% of questions, the correctness of the answer changed because of non-determinism. That study looked at testing education, not brand recommendations. It still shows that run-to-run differences can reach the content of an answer.

What to do: accept sampling as a given and measure rates over many runs. Do not look for one "correct" answer to screenshot.

Does your wording change the answer?

Wording is another source of difference, and it affects you most when you compare results with a colleague or a customer. Real users phrase the same intent in very different ways. SparkToro measured an average semantic similarity of 0.081 across the human-written prompts in the same study. Two people with the same goal write prompts that share little.

The brands that come up still hold fairly steady. Across 994 responses to human-written headphone prompts, leading brands such as Bose, Sony, Sennheiser and Apple appeared in 55% to 77% of answers. The wording varied a lot, and the consideration set stayed fairly stable.

Promptly argues that large differences in substance usually point to a vague prompt rather than to sampling itself. SparkToro's controlled prompts do not fully support that claim. Vague wording is one factor among several.

What to do: write a fixed set of prompts that reflect how your buyers actually ask. Keep the wording identical across runs and change only the intent you are testing.

How does web search add a second layer of variation?

When ChatGPT uses web search, it does not run a single query. FTA Global, a vendor blog, explains that web-enabled AI launches several internal sub-searches, pulls different pages and then synthesizes an answer. If the sub-searches or the pages differ between runs, the final answer can differ even before sampling plays a part.

This helps explain a pattern in the SparkToro data. SparkToro attributes how often a brand appears less to its prominence and more to how many potential recommendations the model chooses from in that category. A category with many candidates spreads the answers across more brands.

Presence and rank also diverge. In the SparkToro data, City of Hope, a hospital in Los Angeles, showed up in 69 of 71 ChatGPT answers, a 97% visibility rate. It was the top mention in only 25 responses. A brand can be nearly always present and rarely first.

What to do: report presence and rank as separate numbers. Read a change in rank only when it repeats across many runs.

Do model updates change the results?

Models change over time, so a result you recorded last quarter may not describe today's behavior. SparkToro's authors say that because their collection window ended in late 2025, newer or updated models may show more or less difference than they found over that period.

The same caution applies to your own tests. A drop in your mention rate after a model update does not necessarily mean your content got worse. Compare rates from before and after, with the same prompts and the same number of runs, before you draw a conclusion.

What to do: log the date and the tool version, where you can see it, with every batch of runs. If you use the API, also store the request parameters and the system fingerprint the cookbook describes.

Where do the studies disagree?

Several sources seem to conflict, but they measure different things.

Source	What it looked at	What it found
SparkToro	Full brand lists in ChatGPT and Google's AI, 2025	Under 1 in 100 chance of the same list in two responses
Otterly	Brands mentioned and links cited	Exact or at least very similar results
BrightLocal	Repeat local AI searches	20% to 33% average overlap between repeat result sets

SparkToro looks at the whole list, which almost never repeats. Otterly, a visibility vendor, reports that the brands being mentioned and the links being cited look exact or very similar. Both can be true: the set of brands can hold while the list and its order move.

BrightLocal's local-search data shows the same split from another angle. Overlap between repeat result sets is low, but a business that appears once is seen again in about half of later attempts: 50% persistence for ChatGPT and AI Mode, and 58% for AI Overview.

What to do: decide which question you are asking before you test. "Is my brand in the set?", "Is my link cited?" and "Is the list identical?" are three different measurements, and they have three different levels of stability.

How do you measure it?

SparkToro's own recommendation is to ask the same question many times, usually at least 60 to 100 times, and average the results. The protocol below follows that logic.

• Fix the prompts: pick five to ten prompts that match how your buyers ask, and write each one down exactly.
• Keep default settings: SparkToro's volunteers kept their usual setups, so your results describe what a normal user sees.
• Run each prompt many times: repeat each one on the same tool and record which brands appear and where.
• Report a rate: state how many runs a brand appeared in out of the total, rather than quoting a single rank.
• Compare and trace: compare your rate with the brands that appear most often, then check which pages those brands are cited from.

For a step-by-step version of this process, read how to check if ChatGPT recommends your brand. For the wider question of what to track, see how to measure your brand's visibility in ChatGPT.

Next steps

• Pick five to ten buyer prompts today and write them down word for word, so every later run uses the same text.
• Run each prompt dozens of times on the same tool, since SparkToro suggests at least 60 to 100, and log which brands appear in each response.
• Report a presence rate and a top-mention rate as two separate numbers, each with its sample size.
• Repeat the same batch after any major model update and compare rates, not single answers.
• If you want this measured every week across several AI tools, Citable does it for ChatGPT, Gemini, Perplexity, Claude, Grok and Google AI. To see how one of those engines picks what to cite, read how Perplexity chooses its sources.

Frequently asked questions

Does temperature 0 make ChatGPT give the same answer every time?

No. A 2025 arXiv paper notes that even with temperature set to 0, determinism is not guaranteed because of hardware factors such as GPU concurrency. Lower temperature makes output more consistent, but it does not make it identical.

How many times should I run a prompt to get a reliable answer?

SparkToro recommends asking the same question many times, usually at least 60 to 100, and averaging the results, according to its 2025 study. Report the result as a rate with its sample size.

What is the seed parameter and does it guarantee identical output?

The seed is a request parameter meant to make completions more consistent. OpenAI's cookbook says responses can still differ, with a small chance, even when the seed, the other parameters and the system fingerprint all match.

Does the API give more consistent answers than the ChatGPT app?

The API gives you request parameters to set and record, while many chat interfaces use intermediate temperature settings and do not clearly expose them. More control helps you document a test. OpenAI's cookbook still says identical settings do not guarantee identical responses, so you need repeated runs in both cases.

Sources

• SparkToro: AIs are highly inconsistent when recommending brands or products
• LLMs in Qualitative Research: Opportunities, Limitations, and Practical Considerations (arXiv)
• Validating Formal Specifications with LLM-generated Test Cases (arXiv)
• How to make your completions outputs consistent with the new seed parameter (OpenAI cookbook)
• ChatGPT and Software Testing Education: Promises & Perils (arXiv)
• Why Does AI Give Different Answers to One Prompt? (Promptly)
• Why AI Gives Different Answers To The Same Question? (FTA Global)
• Why does ChatGPT provide different answers to the same question? (Otterly)
• Study: AI local search data (BrightLocal)
• ChatGPT Temperature & Top P Settings: Advanced Guide