prism-gate

ADR 0001: TCP-Vegas & Netflix Gradient2 Adaptive Concurrency Control

Context

Static semaphore limits per LLM upstream fail under real-world batching and KV-cache contention: an upstream vLLM replica or managed commercial API (gemini-2.5-flash, claude-sonnet-4) experiences dynamic throughput knees depending on prompt sequence lengths and co-tenant load. Unbounded gateway queueing turns transient upstream slowdowns into multi-second p99 tail-latency collapse.

Decision

Implement non-blocking, fail-fast adaptive concurrency limiters (VegasAdaptiveLimiter) governed by TCP-Vegas standing-queue estimation (queue_estimate = inflight * (1 - rtt_min / rtt_sample)) combined with Netflix Gradient2 multiplicative backoff on queue buildup (queue_estimate > beta) and explicit HTTP 429/503 signals, paired with gateway-level progressive priority load shedding (BATCH_BACKGROUND shed at >= 85% utilization, INTERACTIVE shed at >= 95% utilization).

Consequences