A context compressor for AI coding agents: order-preserving knapsack pruning of noisy tool logs (C++20 kernel with a pure-Python fallback), Python AST skeletonizing, cross-turn deduplication of repeated file reads, a rule-based model-tier router, and an MCP server.
Coding agents spend a large share of their context window on noisy test-runner output, Docker build logs, unfocused helper code, and the same read_file payload repeated across turns. tokendiet trims that context deterministically before inference. Lines matching error-anchor patterns (tracebacks, error TS…, constraint violations, etc.) are always kept, and skeletonized Python is re-checked with ast.parse. It is a small, focused tool (~2k lines of Python plus a ~220-line C++20 kernel), not a learned compressor.
![]()
flowchart LR
Raw["Raw Agent Context<br/>(pytest/tsc/docker logs · Python files · multi-turn reads)"] --> Split{"Tool / Turn Classifier"}
Split -->|CLI / Test Log| CppKnapsack["C++20 Salience Knapsack<br/>(clang++ -O3 · integer halo DP)"]
Split -->|Python Source| ASTSkel["Python AST Skeletonizer<br/>(ast.parse-verified stub elision)"]
Split -->|Repeated Read| CppDedup["C++20 Cross-Turn Block Dedup<br/>([@tokendiet:ref] back-pointers)"]
CppKnapsack --> Diet["Compressed Context<br/>(100% anchor retention)"]
ASTSkel --> Diet
CppDedup --> Diet
Diet --> Router["FinOps Cascade Router<br/>(Local Flash -> Balanced -> Reasoning Tier)"]
Diet --> MCP["MCP stdio Server (.mcp.json)<br/>& FastAPI + React Workbench (:8110)"]
All checks ran cleanly on local hardware (Python 3.13.9, clang++ -O3 C++20, Ollama qwen3.5:4b):
| Verification Suite | Exact Measured Result |
|---|---|
| Static Quality Gates | 0 Ruff format issues · 0 Ruff lint issues · 0 Mypy --strict errors (21 files) |
| Pytest + Hypothesis Suite | 14 passed · 97.72% statement coverage (differential C++20 vs Python parity across random budgets & turn IDs) |
Live qwen3.5:4b Audit (6 tasks) |
100.0% (6/6) verified accuracy preserved · 6,487 -> 1,930 prompt tokens (70.25% reduction) · 53.98% faster end-to-end latency |
tokendiet demo (Pytest Log Knapsack) |
996 -> 213 tokens (78.61% saved) · 4/4 (100.0%) mandatory error anchors kept |
tokendiet demo (AST Skeletonizer) |
431 -> 275 tokens (36.19% saved) · 3 helper functions elided · ast.parse=True |
tokendiet demo (Multi-Turn Session) |
2,289 -> 1,047 tokens (54.26% saved) · 5 repeated blocks deduplicated |
| FinOps Cascade Routing | Single-unit bugfix routed to local_flash (99.69% cost reduction); retry/formal ledger proof escalated to frontier_reasoning |
qwen3.5:4b) Task BreakdownFull artifact: benchmarks/real_qwen_tokendiet.json
| Task | Mode | Raw Prompt Tok | Diet Prompt Tok | Token Saved | Latency (Raw → Diet) | Verified |
|---|---|---|---|---|---|---|
TD-01 Pytest ZeroDivisionError |
log_prune |
1,621 | 312 | 80.75% | 2970.21ms -> 664.01ms |
PASS → PASS |
TD-02 TypeScript TS2345 Build |
log_prune |
1,328 | 283 | 78.69% | 1002.64ms -> 539.67ms |
PASS → PASS |
TD-03 Multi-Turn Read + TimeoutError |
session_dedup |
823 | 525 | 36.21% | 718.44ms -> 615.84ms |
PASS → PASS |
TD-04 Python AST Focus Diagnosis |
ast_skeleton |
358 | 285 | 20.39% | 624.75ms -> 528.38ms |
PASS → PASS |
TD-05 Docker ModuleNotFoundError |
log_prune |
1,270 | 284 | 77.64% | 1024.04ms -> 552.53ms |
PASS → PASS |
TD-06 Postgres CheckViolationError |
log_prune |
1,087 | 241 | 77.83% | 792.12ms -> 382.16ms |
PASS → PASS |
ceil(len(text) / 4) character heuristic, not a real tokenizer; the live-audit token numbers come from Ollama’s prompt_eval_count for qwen3.5:4b and will differ for other models’ tokenizers.error TS…, E lines, diff hunks, etc.). Log formats the patterns do not recognise can lose useful context; tune the budget conservatively.# From PyPI (once published)
pip install tokendiet
# From source (today)
git clone https://github.com/nagarjunak-pixel/tokendiet.git && cd tokendiet
uv sync --all-extras # or: pip install .
The C++20 kernel source ships inside the package and is compiled on first import with clang++ or g++ (cached in your temp dir). If no compiler is available, tokendiet falls back to an equivalent pure-Python implementation; tokendiet demo prints which backend is active. The React workbench (tokendiet serve) and the recorded benchmark replay need a source checkout.
uv sync --all-extras
uv run tokendiet demo
# Prune noisy pytest/tsc/docker output down to a 180-token salience budget
pytest -v 2>&1 | uv run tokendiet prune --budget 180
# Skeletonize unfocused functions while keeping focus symbols intact
uv run tokendiet skeleton src/billing.py --focus compute_monthly_discount
.mcp.json){
"mcpServers": {
"tokendiet": {
"command": "uv",
"args": ["run", "--directory", "/path/to/tokendiet", "tokendiet", "mcp-stdio"]
}
}
}
Exposed MCP tools:
tokendiet_prune_tool_outputtokendiet_skeletonize_codetokendiet_compress_sessiontokendiet_route_finopscd ui && npm install && npm run build && cd ..
uv run tokendiet eval-ollama --live-only
uv run tokendiet serve --host 127.0.0.1 --port 8110
C++20 Kernel MattersImplemented in src/tokendiet/_csrc/tokendiet_kernels.cpp and automatically compiled via clang++ -O3 (with automatic pure-Python integer parity fallback):
int64_t division raw_score / (1 + distance) to keep traceback frames and nearby failure context intact.Built by Nagarjuna Reddy Kamireddy (Nagarjuna Kamireddy / Nagarjuna K) — Lead AI Architect, GenAI & Agentic AI Lead, and Corporate AI Trainer (20+ years across Enterprise Data, SAP HANA & Production AI; AWS Certified Solutions Architect – Professional & NVIDIA-Certified Professional: Agentic AI).
llms-full.txt · LinkedIn