Most people costing out an AI assistant budget for the answer. The answer is the cheap part.
A single question in a retrieval system sends the model several thousand tokens and gets a few hundred back. Almost everything you pay for happens before a word is written.
Where the tokens actually go
Here is one answered question in a working RAG pipeline. The numbers are approximate and they move with your settings, but the proportions hold.
The system prefix carries your instructions, tone and rules, and it is byte-identical on every request. The retrieved context is the passages retrieval selected, different every time. History is the recent turns, and the answer is the only part the reader ever sees.
Four thousand four hundred tokens read, three hundred written. The reader sees six percent of what the question cost.
Output is small, and priced like it knows
Output tokens cost about five times input on every model worth using. That sounds like it should dominate, and it does not, because there are fifteen times fewer of them.
The practical consequence is counterintuitive: a long answer over a small context is cheaper than a short answer over a large one. Teams tuning for cost usually start by shortening answers, which is the smallest lever available to them.
The retrieved context is the dial
Look at the numbers again. The largest input component is the one that changes on every request, which means it is the one caching cannot reach.
It is also the one you control directly. Retrieving eight passages instead of five sends roughly 60% more context on every single question, forever.
Cutting retrieval depth is the rare change that reduces cost and improves quality at the same time. Weak matches are noise, and a model handed three sharp passages beats one handed eight where five are marginal.
The second dial is chunk size, because it sets how much text each retrieved passage carries. Retrieval settings are usually discussed as answer-quality knobs. They are the main cost control in the system.
welcomeai.devAI usage and limits: how retrieval settings change what you useWhat caching reaches, and what it cannot
The system prefix is identical on every request, so it can be cached at the provider and billed at a much lower read rate. That is roughly a third of the input handled cheaply.
The retrieved context cannot be cached, by construction. It is assembled per question, and two questions that retrieve different passages share nothing.
This is why the order of a prompt matters for cost and not only for quality. Everything stable goes first; everything per-question goes after it.
Indexing is not where the money is
Turning documents into vectors costs about a hundredth per token of what answering costs, and it happens once per document rather than once per question.
A large manual is cheaper to index than a busy afternoon of questions. Re-indexing after a settings change feels expensive and is not; what is expensive is serving the same question a thousand times with more context than it needed.
The cheapest question is the one never asked
An answer served from a response cache costs nothing at the model, because the model was never called. On a support assistant, where a real share of traffic is the same handful of questions, that is a larger saving than any amount of prompt tuning.
The order to work in, cheapest effort first:
- Cache repeated answers, so common questions stop reaching the model
- Cut retrieval depth until quality starts to drop, then stop
- Keep the stable part of the prompt stable, so it stays cacheable
- Match the model to the job rather than defaulting everything to the largest one
- Only then think about answer length
Questions people ask
If you want one number to watch, make it tokens of retrieved context per answered question. It is the largest thing you pay for, the only large thing you fully control, and the one that quietly grows every time somebody raises a retrieval setting to fix a bad answer.
