What an AI feature costs to run, and how to control it

The five things that drive the running cost of an AI feature, how to estimate cost per request before launch, and the controls that keep the bill steady.

An AI feature differs from the rest of your product in one uncomfortable way: every use costs money. A normal page view is close to free. A request to a language model has a price, and that price varies with what the user typed. Teams that ignore this find out at the end of the month.

The good news is that the cost is predictable once you know what drives it.

What are you paying for?

Hosted models charge by the token, which is roughly a piece of a word. You pay for the tokens you send in and the tokens that come back, and output usually costs more per token than input. If you host a model yourself, you pay for capacity by the hour whether it is busy or idle.

The five drivers of cost

DriverWhat it meansLever you control
Context sizeEverything sent with each request: instructions, history, retrieved documentsSend only what the task needs
Output lengthHow much the model writes backAsk for short answers, cap the length
Model choiceLarger models cost more per tokenUse the smallest model that passes your tests
Calls per taskAgents and multi-step flows call the model several timesCut steps, combine where quality allows
VolumeHow many users, how oftenLimits, caching, pricing

Context size is the one that surprises people. A chat feature that resends the whole conversation each turn gets more expensive with every message. A retrieval feature that attaches ten documents when two would do pays for the other eight on every request.

How do you estimate cost before launch?

  1. Log input tokens, output tokens and the number of model calls for every request in your prototype.
  2. Run a realistic set of tasks through it, including long and awkward ones.
  3. Work out the cost of a typical request and of a heavy one, using your provider’s current price list.
  4. Multiply by expected requests per user and by users, for a quiet month and a busy one.
  5. Compare the result with what the feature is worth: the plan price, or the time it saves.

Look at the heavy request as closely as the typical one. A small number of users with very long inputs can account for a large share of the bill.

What brings the cost down?

  • Trim the context. Summarise old conversation turns, retrieve fewer and more relevant passages, and keep instructions tight.
  • Cache. Identical or near-identical requests can reuse an earlier answer, and many providers discount a repeated prompt prefix.
  • Route by difficulty. Send simple requests to a small, fast model and keep the large one for the cases that need it.
  • Cap the output. Set a maximum length and ask for the format you need, such as a short list instead of an essay.
  • Batch what is not urgent. Work that can wait, such as overnight summaries, is often cheaper to run in bulk.

Check each change against your test set. A cheaper configuration that fails more often is not a saving.

What keeps it predictable?

  • A per-user limit by day or month, with a clear message when it is reached.
  • A hard spending cap or alert at the provider.
  • A dashboard of cost per request and per customer, reviewed weekly at first.
  • A limit on input size, so nobody can paste a book into a text box.
  • A timeout and a retry limit, so a loop cannot run up charges unnoticed.

Should the price of your product reflect it?

If usage varies a lot between customers, a flat price means light users subsidise heavy ones. Common answers are a generous included allowance with a higher plan above it, or keeping the AI feature for plans whose price covers its cost. Whatever you choose, know your cost per customer before you set it.

Provider prices change often, and new models appear every few months. Re-run the estimate whenever you change model, and keep the layer between your product and the model thin enough to switch.

Common questions

How is the cost of an AI feature calculated?

With hosted models, by the number of tokens sent and received on each request, at a rate that depends on the model. Multiply cost per request by how often the feature is used.

What is a token in AI pricing?

A token is a small unit of text, roughly a piece of a word. Providers charge separately for input tokens you send and output tokens the model writes.

How can I reduce the cost of using a language model?

Send less context, cache repeated requests, route simple tasks to a smaller model, cap output length and batch work that is not urgent. Test quality after each change.

How do I stop an AI feature running up an unexpected bill?

Set per-user usage limits, a cap on input size, timeouts and retry limits, and a spending alert at the provider from the first day.

Talk to us

Working through this for your own product?

Send a few lines about what you are building. An engineer replies, usually within one business day with a first opinion.

See how we work with growing companies

We use these details only to reply to this request. No newsletters, no list. We aim to reply within one business day. See our Privacy Policy.

Book a callTalk to our team