Token Budget / Rate Limiting

Token budgets and rate limiting are the controls that govern how much work a large language model endpoint will accept from a given caller over a given window of time. Unlike a traditional API, where every request costs roughly the same, an LLM call's cost scales with the tokens it processes: a 50-token prompt and a 100,000-token prompt each count as one request but consume wildly different compute, memory, and time. Requests-per-minute limits alone therefore cannot protect an LLM service — a handful of maximum-length prompts can consume more GPU time than thousands of short ones. So LLM platforms meter along the token dimension, most commonly tokens per minute (TPM), alongside the familiar requests per minute (RPM), and often concurrent-request caps as well.

How the limits are enforced

Practical rate limiters layer several windows. A short per-minute TPM cap smooths bursts and keeps the serving queue from ballooning; a per-day or per-month token budget enforces an overall ceiling on spend or capacity share; per-model and per-key granularity lets an operator give the big model tighter limits than the small one. Under the hood the usual algorithms apply — token-bucket and sliding-window counters — just denominated in LLM tokens instead of requests. One wrinkle is that a request's true cost isn't fully known until generation finishes, so implementations typically reserve against the prompt size plus the maximum requested output, then reconcile afterward. When a caller exceeds a rate limit, the server answers HTTP 429 (Too Many Requests), often with a retry-after hint; when a hard quota is exhausted, it may return 403. Well-behaved clients read those signals and back off with jitter rather than hammering the endpoint — and well-designed agents surface the pause instead of silently retrying in a tight loop.

Why a self-hoster needs budgets with no bill to fear

It is tempting to think token accounting is a cloud-vendor concern that dies when the model moves onto your own GPU. The opposite is true. On owned hardware the constraint doesn't disappear — it becomes your GPU-seconds, your electricity, and your queue. A single runaway agent stuck in a loop, a misbehaving script, or an overly ambitious batch job can starve every other user of the box for hours. Token budgets give you three things: fairness, because per-user or per-key quotas stop one tenant from monopolizing a shared GPU; predictability, because bounding admitted work bounds queue growth and therefore worst-case latency; and safety, because a budget is the cheapest circuit breaker against both bugs and abuse. If the endpoint is reachable beyond localhost — by family, teammates, or anything on the LAN — unmetered access is an open invitation for one client to become a denial of service.

Budgets inside the pipeline

Rate limiting is the outermost gate in a serving stack: it decides what work is admitted at all. Admitted requests then flow to the scheduler, which decides ordering and batching — see request scheduling — while total capacity is set by the engine-level decisions covered under throughput-optimized serving. The three layers are complementary: capacity determines how much the box can do, scheduling determines who goes next, and budgets determine who was allowed in the door. A related, subtler knob is the per-request output cap (max tokens), which bounds the largest single unit of work any one call can claim.

The sovereign framing

Metering your own infrastructure is not bureaucracy — it is stewardship. The same discipline that keeps a mining operation honest about watts per terahash applies to a local AI box: know what a request costs, decide who may spend it, and enforce that decision in software rather than hope. A quota you set yourself, on hardware you own, answerable to no vendor, is what a rate limit looks like when it serves the operator instead of the platform.