An AI feature differs from the rest of your product in one uncomfortable way: every use costs money. A normal page view is close to free. A request to a language model has a price, and that price varies with what the user typed. Teams that ignore this find out at the end of the month.
The good news is that the cost is predictable once you know what drives it.
What are you paying for?
Hosted models charge by the token, which is roughly a piece of a word. You pay for the tokens you send in and the tokens that come back, and output usually costs more per token than input. If you host a model yourself, you pay for capacity by the hour whether it is busy or idle.
The five drivers of cost
| Driver | What it means | Lever you control |
|---|---|---|
| Context size | Everything sent with each request: instructions, history, retrieved documents | Send only what the task needs |
| Output length | How much the model writes back | Ask for short answers, cap the length |
| Model choice | Larger models cost more per token | Use the smallest model that passes your tests |
| Calls per task | Agents and multi-step flows call the model several times | Cut steps, combine where quality allows |
| Volume | How many users, how often | Limits, caching, pricing |
Context size is the one that surprises people. A chat feature that resends the whole conversation each turn gets more expensive with every message. A retrieval feature that attaches ten documents when two would do pays for the other eight on every request.
How do you estimate cost before launch?
- Log input tokens, output tokens and the number of model calls for every request in your prototype.
- Run a realistic set of tasks through it, including long and awkward ones.
- Work out the cost of a typical request and of a heavy one, using your provider’s current price list.
- Multiply by expected requests per user and by users, for a quiet month and a busy one.
- Compare the result with what the feature is worth: the plan price, or the time it saves.
Look at the heavy request as closely as the typical one. A small number of users with very long inputs can account for a large share of the bill.
What brings the cost down?
- Trim the context. Summarise old conversation turns, retrieve fewer and more relevant passages, and keep instructions tight.
- Cache. Identical or near-identical requests can reuse an earlier answer, and many providers discount a repeated prompt prefix.
- Route by difficulty. Send simple requests to a small, fast model and keep the large one for the cases that need it.
- Cap the output. Set a maximum length and ask for the format you need, such as a short list instead of an essay.
- Batch what is not urgent. Work that can wait, such as overnight summaries, is often cheaper to run in bulk.
Check each change against your test set. A cheaper configuration that fails more often is not a saving.
What keeps it predictable?
- A per-user limit by day or month, with a clear message when it is reached.
- A hard spending cap or alert at the provider.
- A dashboard of cost per request and per customer, reviewed weekly at first.
- A limit on input size, so nobody can paste a book into a text box.
- A timeout and a retry limit, so a loop cannot run up charges unnoticed.
Should the price of your product reflect it?
If usage varies a lot between customers, a flat price means light users subsidise heavy ones. Common answers are a generous included allowance with a higher plan above it, or keeping the AI feature for plans whose price covers its cost. Whatever you choose, know your cost per customer before you set it.
Provider prices change often, and new models appear every few months. Re-run the estimate whenever you change model, and keep the layer between your product and the model thin enough to switch.
