Tokens Are a Work Meter: The Real Cost of an AI Task

Tokens are the work meter behind an AI request. They show how much text the model must read, write, remember, and pass through tools before a job is accepted. That makes a small-looking question surprisingly expensive when an agent has to inspect, decide, call tools, and try again.

What a token is

A token is a small piece of text that a model reads or writes. It might be a whole short word, part of a long word, punctuation, a space, or a special marker. The exact split depends on the model’s tokenizer, its training data, and the text you send. “One token equals three or four English characters” is a rough teaching example, not a billing law.

Names, code, JSON, emoji, URLs, and words from different scripts can break into very different pieces. One sentence can therefore use a different number of tokens in two models. The official model comparison shows model-specific context and capability details; use the usage field or the model’s tokenizer when you need an exact count.

Tokens are not a score for intelligence or quality. They are a meter of text and model work. A concise answer can be expensive if it required a long history, several tool calls, or many internal reasoning steps. A long answer can be relatively cheap when the context is short and the model can produce it in one pass.

Input and output are different kinds of work

This distinction is called prefill versus decode. During prefill, the model reads the prompt and builds the attention state needed for the reply. During decode, it generates autoregressively: each new token depends on the tokens already generated. A Microsoft Research paper on serving long prompts describes why prefill is highly parallel while decode can leave accelerator hardware underused. Its measured ratios are experimental and depend on hardware and workload, not a universal law.

The service keeps a key-value (KV) cache so it does not recompute every prior key and value from scratch on every generated token. That is a big saving, but decode still has to move model weights and cache data and wait for each next-token decision. The result is a real serving difference even when the prompt and answer contain the same number of tokens.

As a dated example, the standard short-context GPT-5.6 family pricing shown in OpenAI’s pricing documentation on 2026-08-04 listed output at six times input: Sol at $5/$30, Terra at $2.50/$15, and Luna at $1/$6 per million tokens (input/output). Treat that as a dated snapshot, not a timeless rate card. The price is what the provider charges commercially. It is not an audited bill of electricity, memory bandwidth, staff, or profit. The prefill/decode research explains why output can be harder to serve efficiently; it does not prove why a provider chose exactly a six-times ratio.

Five parts of the work meter

Keep these categories separate when you measure. A dashboard that shows only visible output misses the prompt, the hidden work, and the evidence gathered around the answer.

Cached input is a discount on reused work

OpenAI’s prompt-caching guide says cacheable prompts need an exact prefix match. Put stable instructions, examples, and schemas first; put changing user details and tool results later. Reusing that prefix can reduce latency and cost on supported models.

Caching does not mean the model gets a free context window. The service still stores and moves the cached representation, checks the request, and processes the uncached suffix. A changed instruction, reordered example, or extra header can break the prefix match. Record cached and uncached input separately so a good cache design is visible in your ledger.

Why agents make the meter climb

A normal chat can be one request and one answer. An agent usually looks more like this:

1. Read the task and constraints.
2. Inspect the current state.
3. Choose an action.
4. Call a tool.
5. Read the result and decide whether it worked.
6. Repeat until the acceptance check passes.

Every loop may resend original instructions, previous results, tool schemas, and new evidence. A failed attempt can cost more than a successful one because it adds another turn. Tool calls may have their own charges, and their output can be large. In a long-running agent, a single page of logs or a database dump can snowball through the next several prompts.

There is no universal “agents cost 5–30 times more” rule. The multiplier changes with context size, model, tool count, retries, reasoning effort, and how often a human rejects or edits the result. Make the loop smaller: return compact typed tool results, summarize old history, cache stable prefixes, and stop when the acceptance test is true.

The language tax: a token premium for the same meaning

People often call this a “language tax,” but a language is not inherently inefficient. Much of the disparity comes from tokenizer design: how training data shaped the vocabulary, how scripts and whitespace are pre-tokenized, and which words or fragments were common enough to earn a compact token.

Petrov and co-authors compared parallel translations in 2023 and found token-length differences of up to 15× in some cases, including multilingual tokenizers. That is a warning about possible extremes, not a claim that every language or model is 15× different. A 2026 preprint by Churchill and Skiena likewise reports that low-resource languages often need several times more tokens, with higher API and energy cost and less effective context. Both findings are measurements of particular datasets, tokenizers, and models.

That is the compounding effect: more pieces can mean more input cost, less semantic content in the context, more latency, and sometimes more output tokens. The premium is model-specific. Measure the actual language, tokenizer, and workload instead of ranking languages as good or bad.

Reasoning, tools, and retries are part of the bill

A final visible answer is not the whole meter. A reasoning model may spend hidden tokens comparing options. A browser or search tool may return a large page. A failed API call can trigger a retry. A human may reject the first draft and ask for another. Count those events against the accepted task, even when the user sees only the polished final paragraph.

Tool calls can also have separate pricing, quotas, or rate limits. Read the tool’s current documentation. If a tool returns ten thousand log lines, the next model call may pay to read them. Return a small, typed result instead: status, fields needed for the decision, and a compact error.

Measure cost per accepted task

The prompt is not the unit that matters. The accepted task is. Record every request, then group requests by one job or run:

cost per accepted task = model usage + tool usage + human review cost

Use the current price for the model and date of the run. Prices, cached-input rules, and context windows change. Keep the model name, tokenizer, date, and currency beside the number so a future reader knows what the meter meant.

Five ways to lower the meter without lowering quality

• Keep stable rules and examples in a short, cacheable prefix.
• Summarize old tool output instead of appending a full transcript forever.
• Return only the fields the next step needs, with explicit error states.
• Use a fast model for routing and formatting; reserve deeper reasoning for risky decisions.
• Set a stop condition and an attempt limit before the agent starts.
• Test representative languages, code, JSON, and names with the production tokenizer.
• Put human approval gates around publishing, money movement, deletion, and production changes.
• Review accepted-task cost, not just the cheapest single request.

A small example

Suppose a site owner asks, “Is the homepage healthy?” A one-shot answer can guess from a URL. An accepted maintenance task must fetch the page, check the status, inspect cache headers, run a mobile and desktop measurement, compare the result with a baseline, and report a rollback path. The final answer may be a few paragraphs, but the meter includes every check and every result sent back into the loop.

That is not a failure. It is the cost of evidence. The goal is not the lowest token count; it is the lowest reliable cost per accepted task.

Three ways the same request changes size

These are not three grades of the same prompt. They are three different jobs. Comparing their raw token totals without comparing the accepted outcome is like comparing the fuel used by a bicycle, a delivery van, and a fire truck and asking which one is “more efficient.” The right question is whether each job delivered its required result safely.

What to put in a real meter

Start with the request ID and model name. Add input, cached input, output, and reasoning usage from the provider response. Add tool name, tool duration, bytes or rows returned, and any tool charge. Then record whether the result was accepted, retried, abandoned, or rolled back. A short human-review field turns a machine bill into a useful operating measure.

task_id,model,input,cached_input,output,reasoning,tool_cost,attempts,review_minutes,outcome
publish-042,gpt-5.6-terra,18420,12000,2300,4100,0.03,3,12,accepted

This tiny ledger is not a replacement for an invoice. It is a way to answer practical questions: Did caching help? Which tool caused the retry? Did a more capable model reduce review time? Did a language or file format inflate the context? Keep raw provider usage alongside your roll-up so an estimate can be audited later.

Common meter mistakes

• Counting only the final answer. The hidden prompt and prior turns may be much larger.
• Assuming cached means free. A cache hit is discounted reused work, with exact-prefix rules.
• Using one English conversion for every text. Code, names, emoji, and languages can fragment differently.
• Applying a price table forever. Model prices and billing categories change; date your snapshot.
• Ignoring retries and review. An answer rejected by a person was not an accepted task.
• Blaming a language. Tokenizer vocabulary and preprocessing create much of the difference.
• Calling a research ratio a provider invoice. Serving papers explain physical pressure, not commercial pricing.

Questions readers usually ask

Does a longer prompt always cost more?

Usually it uses more input tokens, but caching can make repeated prefixes cheaper. A shorter prompt can also trigger more retries if it leaves out a needed constraint. Measure the accepted run.

Why not make every answer short?

Short output saves decode work, but omitting a test, citation, or rollback step can create expensive rework. Optimize for reliable completion, not tiny text.

Can I compare token counts across models?

Only carefully. Different tokenizers split the same text differently, and models report usage categories differently. Compare cost and outcome for the same representative workload.

Is a high token count bad?

Not by itself. A high count can mean a difficult task was handled thoroughly. It is a warning sign when cost rises while accepted-task rate and review quality stay flat.

Will language-token premiums disappear?

Better multilingual tokenizers may reduce them, but no single tokenizer is best for every script, domain, and model. Re-measure after a model or tokenizer change.

Make the meter useful to a team

Choose one owner for the ledger and one review window, such as a weekly sample of accepted tasks. Look for patterns instead of shaming individual requests. A spike may come from a new tool returning verbose logs, a changed system prompt that broke the cache prefix, a model upgrade with a different tokenizer, or a real increase in task difficulty.

Set simple alerts: a maximum number of attempts, a maximum context size, and a human approval gate before irreversible actions. When an alert fires, preserve the request and tool evidence so someone can see what happened. A safe stop is cheaper than an endless loop.

For multilingual products, sample the actual customer languages and content types. Include short messages, long documents, names, code snippets, punctuation-heavy text, and JSON. Compare tokens per meaning-bearing word, response time, cost, and answer quality. That gives you a fair engineering baseline without treating one language as the “normal” one.

Keep a before-and-after note when you change a prompt, model, tokenizer, or tool. Write down what improved and what got worse. A cheaper request that needs twice as much human correction is not cheaper. A larger request that prevents a production rollback may be the better trade. The meter helps make that trade visible.

Finally, keep the measurement boring and repeatable. Export the provider usage, save the task outcome, and review a small sample on a regular schedule. You do not need a perfect accounting system to notice that a cache stopped hitting, a tool became noisy, or a translation now takes twice as many pieces.

That small habit also gives a team a shared language. “The prompt is short” stops being the end of the discussion; people can ask which tokens were cached, which tool returned them, how many attempts ran, and whether the work was accepted. Those questions turn a mysterious bill into an engineering signal.

Sources and dated caveats