PrismGate is a high-throughput, OpenAI-compatible Multi-Tenant LLM Inference Gateway, TCP-Vegas Adaptive Concurrency Controller, Speculative Complexity Cascade Router, Online Claim-Faithfulness Judge, and Statistical Drift Plane.
Designed for production AI infrastructure teams operating heterogeneous fast and frontier model pools (gemini-2.5-flash, vllm-qwen-32b, gemini-2.5-pro, claude-sonnet-4), PrismGate eliminates static rate-limit tuning, cuts inference spend by 57%+ via speculative routing and tenant-isolated signed-hash semantic caching, catches quantitative RAG hallucinations inline, and detects live traffic distribution shifts using Population Stability Index (PSI) and Base-2 Jensen-Shannon Divergence (JSD).

flowchart LR
SDK["OpenAI Client / SSE"] --> Gate["PrismGate FastAPI (/v1/chat/completions)"]
Gate --> Adm["Dual Token-Bucket + ≥85% Priority Shedder"]
Adm --> Cache["Tenant-Isolated Signed-Hash Cache"]
Cache -- "Miss" --> Cascade["Complexity Classifier [0..1]"]
Cascade -- "Simple (<0.48)" --> Fast["Fast Vegas Pool (Flash / Qwen-32B)"]
Cascade -- "Complex (>=0.48)" --> Front["Frontier Vegas Pool (Pro / Sonnet-4)"]
Fast -- "Low Confidence / Faithfulness" --> Front
Fast --> Eval["Atomic Claim-Faithfulness Judge"]
Front --> Eval
Eval --> Drift["PSI + Jensen-Shannon Drift Plane"]
src/prism_gate/concur/vegas_limiter.py):
max_inflight additively when $\text{RTT}{\text{sample}}$ tracks $\text{RTT}{\min}$, holds steady inside the Little’s Law equilibrium band, and backs off multiplicatively on standing queue buildup or HTTP 429/503 throttling—with non-blocking sibling failover across same-tier providers.src/prism_gate/admission/token_bucket.py):
BATCH_BACKGROUND at $\ge 85\%$ and INTERACTIVE at $\ge 95\%$, preserving emergency capacity for CRITICAL tier SLOs without burning shed tenants’ own RPS/TPM budgets.src/prism_gate/routing/cascade_router.py):
src/prism_gate/evals/faithfulness.py):
$12.4M vs $18.9M, 5mg vs 500mg), domain entities, and clinical/financial antonym polarity against retrieved RAG context chunks.src/prism_gate/drift/detectors.py):
PSI) across decile-binned prompt length, complexity, and latency distributions.JSD $\in [0, 1]$ bits) over token and topic distributions, formally verified via Hypothesis for exact identity ($\text{JSD}(P,P)=0$), bit-for-bit symmetry ($\text{JSD}(P,Q)=\text{JSD}(Q,P)$), and unit-interval boundedness.Executed locally on Apple Silicon (Python 3.12.12, uv run prism benchmark --requests 300 --concurrency 32 with mid-run upstream congestion injection):
| Metric | Measured Value | Notes |
|---|---|---|
Gateway Control-Plane Overhead (p50 / p95 / p99) |
0.067 / 0.126 / 0.137 ms |
Admission + Complexity + Signed-Hash Cache + Vegas Lease |
End-to-End Inference Latency (p50 / p95 / p99) |
15.90 / 62.15 / 72.02 ms |
Mixed Fast, Cache Hit, Escalated, and Frontier traffic |
| Sustained Gateway Throughput | 4,666.6 req/s |
300 requests @ 32 concurrent async workers |
| FinOps Net Inference Cost Reduction | 57.16% |
$0.040807 actual vs $0.095247 static claude-sonnet-4 pin |
| Tenant-Isolated Semantic Cache Hit Rate | 17.5% |
Zero cross-tenant leakage; numeric/negation guarded |
CRITICAL Priority Tier Preservation |
97.5% (117 / 120) |
Protected by progressive BATCH_BACKGROUND shedding at $\ge 85\%$ util |
uv sync --python 3.12 --all-extras
# Run interactive routing, Vegas adaptation, RAG hallucination judge & drift demo
uv run prism demo
# Verify PSI & Jensen-Shannon mathematical invariants across traffic regimes
uv run prism drift-check
# Run 300-request concurrent load & FinOps benchmark
uv run prism benchmark --requests 300 --concurrency 32
cd ui && npm install && npm run build && cd ..
uv run prism serve --host 127.0.0.1 --port 8000
Open http://127.0.0.1:8000 to explore the 4-tab PrismGate Control Plane Console:
text/event-stream token streaming + live prism_meta FinOps & complexity breakdown)inflight / max_inflight ceilings, RTT gradients, and one-click 200-request load burst)NO_DRIFT / MODERATE_DRIFT / CRITICAL_DRIFT simulator)curl -N http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "prism-auto",
"stream": true,
"tenant_id": "tenant-enterprise-01",
"priority": "CRITICAL",
"messages": [
{"role": "user", "content": "Summarize Q3 revenue and GAAP operating margin."}
],
"rag_context": [
"Acme Corp reported Q3 2025 revenue of $12.4M with GAAP operating margin of 18.5%."
]
}'
Prometheus scrape endpoint is exposed natively at GET /metrics (text/plain; version=0.0.4). To proxy live external OpenAI-compatible upstreams (vLLM, SGLang, OpenAI, Gemini), set PRISM_UPSTREAM_BASE_URL and PRISM_UPSTREAM_API_KEY.
uv run ruff check src tests
uv run ruff format --check src tests
uv run mypy src
uv run pytest --cov=prism_gate --cov-fail-under=95 --cov-report=term-missing
cd ui && npm ci && npm run build
PrismGate serves as Pillar 3 (High-Throughput Inference Control Plane, Faithfulness & Statistical Drift) in a cohesive 3-system enterprise AI architecture:
aegis-runtime — Zero-Trust Agentic Control Plane, .aegis AST Policy Compiler, & Cryptographic ERP Saga Engine.chronos-mesh — Multi-Echelon Supply-Chain Digital Twin, Exact Branch-and-Bound MILP Solver, & CVaR95 Risk Engine.prism-gate — OpenAI-Compatible Streaming Gateway, TCP-Vegas Concurrency Controller, Speculative Cascade Router, & PSI/JSD Drift Plane.Created and maintained by Nagarjuna Kamireddy (sendemailtonagarjuna@gmail.com).
Licensed under the Apache License, Version 2.0.
Built by Nagarjuna Reddy Kamireddy (Nagarjuna Kamireddy / Nagarjuna K) — Lead AI Architect, GenAI & Agentic AI Lead, and Corporate AI Trainer (20+ years across Enterprise Data, SAP HANA & Production AI; AWS Certified Solutions Architect – Professional & NVIDIA-Certified Professional: Agentic AI).
llms-full.txt · LinkedIn