Production AI architecture: optimize cost per accepted task

By Tensor State Group · Published 12 October 2026

Define the unit of useful work

Token cost is convenient to measure but can be a poor proxy for business value. Define an accepted task: an output that meets a task-specific quality threshold and reaches the intended user or downstream system. Count failed generations, retries, review, retrieval, tool execution, and infrastructure in the cost numerator. Keep the acceptance denominator explicit. This article is an architecture analysis, with a synthetic arithmetic example linked below; it does not report customer savings or production benchmark results.

Profile the full request path

Separate queue time, retrieval, prompt construction, prefill, decoding, tool calls, validation, and human escalation. Report end-to-end p50 and p95 latency alongside time to first token where streaming matters. A fast first token does not guarantee fast task completion. Sample realistic prompt and response lengths, concurrency, cache-hit rates, and burst traffic. Include cold starts and dependency failures. A trace that omits reviewer work may hide the dominant cost in a supposedly automated system.

Understand batching and memory trade-offs

Batching can improve device utilization but may increase queueing delay. Long sequences consume cache memory and can reduce available concurrency. The PagedAttention paper explains one approach to KV-cache memory management; its reported comparisons apply to its experimental setup, not every workload. Compare candidate serving configurations on a fixed workload and hardware specification. Distinguish request throughput from token throughput and successful task throughput. An infrastructure change that serves more tokens while reducing accepted quality may be economically worse.

Apply a quality gate to every optimization

Quantization, shorter prompts, smaller models, routing, caching, and retrieval changes can all alter behavior. Run each candidate through a held-out task evaluation. Check important subgroups and low-frequency failure modes as well as the average. Cache keys must include the context that determines the answer; stale or unauthorized content is not a legitimate latency improvement. Routing evaluation should include routing mistakes and fallback costs. Keep rollback conditions measurable and tied to the exact configuration being deployed.

Model capacity and ownership

Compare hosted and self-managed serving with the same demand assumptions. Include idle capacity, engineering operations, monitoring, incident response, data-transfer costs, and model licences where applicable. Use low, expected, and high demand scenarios rather than a single forecast. Account for utilization: an inexpensive accelerator at low occupancy can have an expensive unit cost. Identify who owns on-call response, model updates, dependency changes, and security review. Cost comparisons should disclose what is excluded rather than hiding uncertainty in a precise-looking figure.

Make a bounded deployment decision

Select a candidate only after it meets the quality threshold, tail-latency target, operating budget, and support constraints. Run a progressive rollout and compare observed demand with the benchmark assumptions. Monitor acceptance rate, retries, escalations, and cost per accepted task. Reevaluate when model versions or workload distributions change. The goal is a documented operating point with an owner and rollback path, not a universal promise that one serving technique is fastest or cheapest.

Sources and further reading

PagedAttention: Efficient Memory Management for Large Language Model Serving

References provide background. The proposed methods and decision framing above are Tensor State Group’s technical synthesis; no external publication endorses this service.