Tokens, Rate Limits, and Cost: How AI API Pricing Actually Works

Teams get caught off guard by AI API invoices less because the pricing is secretive and more because a handful of easy-to-miss factors stack up fast once real traffic hits. A setup that looked dirt cheap in a demo can balloon into a five-figure monthly charge the moment actual users, full conversation histories, and retry logic all show up at once. Get the underlying math straight beforehand, and the bill stops being a surprise.

The billing unit is the token, not the request

Tokens are what you're charged for, not individual API calls. In English, a token works out to roughly three-quarters of a word, though the precise ratio depends on which tokenizer your provider runs under the hood. Nearly every provider bills input tokens (your prompt, system instructions, prior conversation) and output tokens (what comes back from the model) at separate rates, and stacking a long prompt with a long reply costs noticeably more on both sides at once.

• Input tokens: whatever goes into the model - the system prompt, the user's message, retrieved documents or tool output, and any earlier turns carried forward.
• Output tokens: whatever the model hands back, including any intermediate reasoning steps some models emit before landing on a final answer.
• Output tends to cost more per token: providers routinely charge a multiple of the input rate for generation, since producing text takes more compute than reading it.

Because generated tokens carry a premium, response length isn't just a style choice - it's a lever with a real dollar figure attached. Nudging the model toward brevity where brevity actually works, or toward a tight structured format instead of a meandering one, is a legitimate optimization wherever the use case allows for it, not just minor polish.

Multi-turn conversations get pricey in a hurry

Here's the part that trips people up: every new call in a multi-turn exchange typically re-sends the entire history - system prompt, every earlier message, every earlier reply - as input tokens, because the model retains nothing between requests on its own. So a conversation that runs long keeps getting more expensive per turn purely from the accumulating input, no matter how short the newest question actually is. A support chatbot left open for twenty back-and-forths can end up billing for those same opening messages twenty times over.

The same dynamic shows up, amplified, in agentic setups - a model calling tools, reading what comes back, and reasoning through several internal steps before it finally produces something the user sees. Each of those internal steps adds to the token total for what looks, from the outside, like a single request.

Rate limits and your budget are two different constraints

Rate limits sit apart from what you're spending entirely - they cap requests or tokens per minute regardless of how much budget you have left. You can be nowhere near your spending ceiling and still get throttled the moment traffic spikes, which is exactly why graceful handling of rate limits (retry with backoff) belongs in the integration from day one, not on a someday list.

• Requests per minute (RPM): caps the number of distinct calls in a given window, no matter how small or large each one is.
• Tokens per minute (TPM): caps total token throughput - one sufficiently large request can chew through this on its own.
• Concurrent request caps: some providers also limit how many calls can run at the same time, which matters for anything parallelized.
• Limits rise with tier: most providers bump these ceilings up automatically as your usage history and account age grow, so getting throttled early isn't a permanent state.

What's actually running up the bill

Keeping the bill predictable

• Cut down conversation history: pass along only what the current turn needs instead of resending a transcript that keeps growing.
• Cap output length explicitly: constrain response size anywhere a shorter answer genuinely does the job.
• Cache what repeats: identical or near-identical requests - a stock FAQ answer, a lookup you've already done - don't need a fresh model call.
• Watch real usage, not guesses: pull actual token numbers from production or a genuine beta rather than extrapolating from a handful of manual test calls.
• Right-size the model per task: a cheaper, smaller model often handles classification or extraction fine, leaving the bigger model for work that actually needs the extra reasoning.

Once you've got the mechanics down, none of this is guesswork. Token-based pricing rewards the same habits any metered resource does: send only what's necessary, ask for only what you actually need back, and treat throttling as a normal part of operating rather than something going wrong.