Cheaper engine settings cut Claude from 6.8 brands per answer to 3.1 and stopped seven of ten answers mid-sentence. Every check said nothing changed.
· 5 min read
In September we made our AI answers cheaper, and every check we had passed. The score stayed the same. The engines named the customer in exactly the same answers as before. Then we counted the companies in each answer. Claude was naming fewer than half of what it named before, and seven of its ten answers stopped in the middle of a sentence.
We measure which companies AI assistants name when a buyer asks them for a recommendation. That means sending real buyer questions to ChatGPT, Gemini, Perplexity and Claude with live web search turned on, and every one of those answers costs money. A tracked run of ten questions across five engines cost $1.62. We wanted it lower.
The cheaper settings
We tested a candidate setting for each engine against the one we had, on ten real buyer questions per engine:
- ChatGPT: the free-tier model instead of the paid one (gpt-5.4), and replies capped at 700 tokens instead of 1,500.
- Claude: one web search per answer instead of its own choice (two, in the answer we decomposed), and the same 700-token cap.
- Gemini: a smaller search context, and the 700-token cap.
- Perplexity: only the 700-token cap.
A full run on the candidates cost $1.10 against $1.62. That is 1.62 − 1.10 = $0.52 saved, and 0.52 ÷ 1.62 = 32%.
We checked two things. The score was the same. Mention agreement, the share of answers where both settings agreed on whether the customer was named, was 10 of 10 on every engine. Every check passed.
What the answers said
Then we read the answers. We counted the companies each one named, and how many simply stopped mid-sentence.
| Engine | Brands per answer, before | After | Answers cut mid-sentence |
|---|---|---|---|
| ChatGPT | 3.2 | 2.8 | 7 of 10 |
| Claude | 6.8 | 3.1 | 7 of 10 |
| Gemini | 8.4 | 7.0 | 8 of 10 |
| Perplexity | 7.3 | 5.9 | 0 of 10 |
Claude went from 6.8 brands per answer to 3.1. That is 3.1 ÷ 6.8 = 46% of what it named before. More than half the names were gone.
Two of the candidates changed more than the cap. Claude's also cut its searches to one, and Gemini's also narrowed its search context. Both mean the engine reads fewer pages, and an engine that reads fewer pages can know fewer companies. So for Claude and Gemini the drop has two causes, and we did not separate them.
The answers that stopped mid-sentence are the cap alone. A search setting does not end a sentence halfway. The cap stopped seven of ten ChatGPT answers, seven of ten Claude answers and eight of ten Gemini answers.
Perplexity is the clean case, because the cap was its only change. It finished every answer and still went from 7.3 brands to 5.9. One run per setting cannot say how much of that is the cap. Two live runs of the same engine name a different brand first in 4 to 7 of 10 open questions, so answers vary from run to run on their own.
Why the score did not see it
The customer in that test was never named by any engine, under either setting. An answer that leaves you out at 1,500 tokens also leaves you out at 700. So named-or-not agreed 10 times out of 10, and the score, which is built from those answers, did not move.
Take a customer who is named sixth. A Claude answer with room for 6.8 names includes them. A Claude answer that stops after 3.1 does not.
The engine, given room to finish, would have named them. Only our cheaper copy of the answer cut them off. The report would have said "not mentioned".
The check failed because it was built on the case we were testing. Mention agreement only tells two settings apart when the customer is named in some answers and not in others. Brands per answer tells them apart for any customer, because it counts the whole list. Our live cost test now prints it next to the cost, along with the answers cut, per engine.
Where the money in an AI answer goes
The saving was also smaller than it looked, and the reason is in the bill. We expected the cost of an answer to track the length of the reply. We split one answer's bill into its parts with the cost breakdown OpenRouter returns for each call.
A ChatGPT answer on the free-tier model cost $0.036:
- Tokens: $0.006.
- Web searches: three, at a cent each, so $0.03.
The searches are 0.03 ÷ 0.036 = 83% of the answer. The reply cost $0.006.
A Claude answer with two searches cost $0.068. Of that, $0.048 was tokens and the rest was the two search fees. Most of the tokens were not the reply. The search results come back into the conversation and are billed as input: that answer read 19,501 prompt tokens. With one search it read 13,023 tokens and cost $0.046, so each search added about 6,500 tokens of reading.
So the cost of an answer is mostly searching, paid as a fee, as tokens of search results, or both. A reply cap trims the smallest part. The ChatGPT free-tier model is twelve times cheaper than the paid one, and its answers still cost 68% of what they did, because the three searches cost the same on either model.
What we shipped
Our rule is that each engine is measured the way a signed-out or free user gets it, never with the best model. Most buyers are free users.
By that rule, two choices made themselves. The free-tier ChatGPT model stayed: it is cheaper, and it is what a free ChatGPT user actually gets. One search on Claude went: it halved the pages Claude reads, which is not what a free Claude user gets, to save thirty cents a run. The smaller Gemini context went too, for three cents. The 700-token cap saved about ten cents a run and went as well.
Every one of these settings still exists as configuration. Before anyone changes one, a live test prints the cost, the brands per answer and the answers cut, per engine.
If you measure AI answers yourself, look at where each answer stops. An answer that ends mid-sentence was cut by a setting, and whatever came after the cut is missing from your count.
Who AI Knows measures whether AI assistants recommend a business when buyers ask.