veritas-rl

VeritasRL — Frontier RLVR Post-Training, DoRA, 2D Parallelism & Step-Level PRM Engine

VeritasRL Frontier CI Release v1.0.0 Python 3.13 C++20 Native Kernels PyTorch Autograd License: Apache-2.0

VeritasRL is a clean-room, zero-mock reference implementation of the four systems pillars behind modern Frontier Reasoning Post-Training and Distributed Scaling (DeepSeek-R1 / DAPO, OpenAI o-Series Process Reward Models, ICML 2024 Oral DoRA, and NVIDIA Megatron-LM + Causal Ring-Attention):

  1. Critic-Free Verifiable-Reward RL (GRPO + DAPO + Schulman $k_3$ KL): Eliminates the value/critic network entirely (50% value-model VRAM saved), scoring $G$ parallel policy rollouts per prompt with a non-hackable deterministic symbolic execution verifier, normalizing group advantages in native C++20, and optimizing a causal Transformer policy via DAPO asymmetric clipping ($[1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}] = [0.80, 1.28]$) and Schulman’s exact non-negative $k_3$ KL estimator ($e^{\log(\pi_{\text{ref}}/\pi_\theta)} - \log(\pi_{\text{ref}}/\pi_\theta) - 1 \ge 0$).
  2. Weight-Decomposed Low-Rank Adaptation (DoRA, Liu et al., ICML 2024 Oral): Decouples fine-tuning updates into a learnable row-magnitude vector $m \in \mathbb{R}^{d_{\text{out}}}$ and normalized directional low-rank matrices $W_0 + \frac{\alpha}{r} B A$, overcoming LoRA’s Eckart-Young rank floor while folding into a zero-overhead nn.Linear at serving time (max |Δ| = 0.00000000).
  3. 2D Distributed Sequence + Tensor Parallelism (Causal Online-Softmax Ring-Attention + Megatron-LM 2D MLP): Proves exact forward and PyTorch autograd.backward gradient equivalence (< 1.5e-6 max |Δ|) between $P$-rank Ring-Attention (circulating $(K_p, V_p)$ shards via associative online log-sum-exp state $(O, m, \ell)$ without materializing the $S \times S$ attention matrix) and $P$-rank Column-Parallel $\to \operatorname{GeLU} \to$ Row-Parallel All-Reduce(SUM) MLPs.
  4. Zero-Mock Step-Level Process Reward Model (PRM) & Test-Time Compute Scaling (qwen3.5:4b): Evaluates live local qwen3.5:4b via Ollama across 8 multi-step Olympiad Combinatorics, Chinese Remainder, Diophantine Optimization, and Number Theory problems—verifying intermediate lhs = rhs AST equations (+ - * / %) and public problem invariants to lift accuracy from 62.50% (5/8 Pass@1) to 100.00% (8/8 PRM Best-of-6, +37.50% lift).

Live Interactive RLVR, DoRA & 2D Parallelism Workbench

VeritasRL Live Workbench — Critic-Free GRPO/DAPO, ICML '24 DoRA, Ring-Attention + Megatron 2D TP & Step-Level PRM


Architecture Overview

flowchart LR
    subgraph CppKernel["Native C++20 Kernel Layer (csrc/veritas_kernels.cpp)"]
        K1["Group-Relative Z-Score\nAdvantage Normalization"]
        K2["Fused Backward GAE-λ +\nSchulman Non-Negative k3 KL"]
        K3["Associative Online-Softmax\n(O, m, l) Ring Tile Merger"]
    end

    subgraph PostTrain["Pillar 1 & 2: RLVR Post-Training & DoRA"]
        PT1["Cold-Start Atomic Primitives\n-> Frozen Reference π_ref"] --> PT2["G=12 Rollouts -> Symbolic\nExecution Verifier R ∈ [0, 1]"]
        PT2 --> PT3["DAPO Asymmetric Clip [0.80, 1.28]\n+ DoRA Magnitude/Direction"]
    end

    subgraph Dist2D["Pillar 3: 2D Sequence + Tensor Parallelism"]
        DP1["Causal Ring-Attention\nO(S²/P) Per-Rank Memory"] --> DP2["Megatron Column -> GeLU -> Row\nSingle All-Reduce Autograd"]
    end

    subgraph TTC["Pillar 4: Test-Time Compute Scaling (Ollama)"]
        T1["Live Qwen-3.5-4B\nN=6 Parallel CoT Rollouts"] --> T2["AST Step-Equation PRM +\nPublic Invariant Consensus"]
    end

    K1 & K2 --> PT3
    K3 --> DP1
    PT3 --> TTC

Measured Reproducible Benchmarks (Zero-Mock Execution)

Every figure below is produced by running the real PyTorch, clang++ -O3 -std=c++20, and Ollama pipelines in this repository:

Subsystem Benchmark Command Measured Result Verified Invariant
GRPO / DAPO Verifiable RL (seed=17, G=12, 32 epochs) uv run veritas train-grpo 6.25% → 50.00% Pass@1 (8.0x lift, +43.75% gain) 50% critic VRAM saved; $k_3 \text{ KL} = 0.86972 \ge 0$; C++20 vs NumPy < 1e-5
DoRA vs. LoRA Adaptation (128×128, r=8, 70 steps) uv run veritas benchmark-dora DoRA MSE 0.00323 vs. LoRA 1.81892 (563x lower) -86.72% trainable params (16,384 -> 2,176); folded nn.Linear max \|Δ\| = 0.00000000
Causal Ring-Attention (P=4, S=64) uv run veritas verify-distributed Forward \|Δ\| = 3.0e-7 · Autograd $\nabla Q,\nabla K$ \|Δ\| = 8.3e-7 4.0x peak attention-matrix reduction; Hypothesis verified vs PyTorch SDPA
Megatron 2D TP MLP (P=4, d_ff=128) uv run veritas verify-distributed Forward \|Δ\| = 1.43e-6 · Autograd $\nabla X,\nabla W_1$ \|Δ\| = 1.19e-6 Zero activation-stage IPC; single forward All-Reduce(SUM)
Live qwen3.5:4b Step-Level PRM Scaling (N=6) uv run veritas eval-ttc --live-only 62.50% (5/8) Pass@1 → 100.00% (8/8) PRM Best-of-6 +37.50% Olympiad lift recorded in benchmarks/real_qwen_ttc.json

Quickstart

1. Install & Run Automated Verification (98% Test Coverage)

Requires Python 3.12+ (3.13 recommended), uv, and a C++20 compiler (clang++ or g++; automatically falls back to exact NumPy if unavailable):

uv sync --python 3.13 --all-extras

# Run Ruff lint, Mypy --strict, and Pytest + Hypothesis property suite (>=95% gate; 98% achieved)
uv run ruff check src tests
uv run mypy src tests
uv run pytest --cov=veritas_rl --cov-report=term-missing --cov-fail-under=95

2. Run CLI Post-Training, DoRA, Parallelism & Ollama PRM Benchmarks

# 1. Critic-free GRPO/DAPO post-training with C++20 group advantages
uv run veritas train-grpo

# 2. DoRA vs. LoRA convergence & zero-latency weight folding proof
uv run veritas benchmark-dora

# 3. Causal Online-Softmax Ring-Attention & Megatron 2D TP forward + backward autograd proof
uv run veritas verify-distributed

# 4. Step-level PRM Test-Time Compute scaling (replays benchmarks/real_qwen_ttc.json if Ollama is offline, or pass --live-only)
uv run veritas eval-ttc

3. Launch the Interactive Frontier RLVR & Parallelism Workbench UI

cd ui && npm install && npm run build && cd ..
uv run veritas serve --host 127.0.0.1 --port 8040
# Open http://127.0.0.1:8040

Repository Structure

veritas-rl/
├── csrc/
│   └── veritas_kernels.cpp              # C++20 (-O3) group advantages, GAE-λ + Schulman k3 KL & online-softmax
├── src/veritas_rl/
│   ├── kernels/native_bridge.py         # SHA-256 cached clang++/g++ compiler & ctypes bridge + NumPy fallback
│   ├── rlvr/grpo_trainer.py             # Critic-free GRPO + DAPO trainer & non-hackable symbolic verifier
│   ├── adapters/dora.py                 # ICML 2024 Oral DoRA vs LoRA & zero-overhead merge_to_linear()
│   ├── distributed/parallel_ops.py      # Causal Online-Softmax Ring-Attention & Megatron 2D TP autograd proofs
│   ├── ttc/prm_ollama.py                # AST step-equation PRM & live Ollama Best-of-N benchmark
│   ├── api/app.py                       # Typed FastAPI control plane + SPA static serving
│   └── cli.py                           # Rich & JSON CLI (`veritas`)
├── ui/                                  # React 18 + TypeScript + Vite Frontier RLVR Workbench
├── benchmarks/real_qwen_ttc.json        # Recorded live qwen3.5:4b Pass@1 vs PRM Best-of-6 telemetry
├── docs/
│   ├── mathematical-proofs.md           # Formal derivations of GRPO/DAPO, k3 KL >= 0, DoRA & Ring-Attention
│   └── adr/                             # Architecture Decision Records (0001..0004)
└── tests/                               # Unit + Hypothesis mathematical property suite (98% coverage)

Documentation & Mathematical Proofs

License

Apache License 2.0 — Copyright (c) 2026 Nagarjuna Kamireddy.


Author & Enterprise AI Architecture Hub

Built by Nagarjuna Reddy Kamireddy (Nagarjuna Kamireddy / Nagarjuna K) — Lead AI Architect, GenAI & Agentic AI Lead, and Corporate AI Trainer (20+ years across Enterprise Data, SAP HANA & Production AI; AWS Certified Solutions Architect – Professional & NVIDIA-Certified Professional: Agentic AI).