Frontier RLVR Post-Training From Scratch: Critic-Free GRPO/DAPO, DoRA Weight Decomposition, 2D Ring-Attention & Step-Level PRMs
Published November 16, 2026 · Reference Codebase: github.com/nagarjunak-pixel/veritas-rl
Modern reasoning models combine verifiable-reward reinforcement learning (RLVR), parameter-efficient directional/magnitude adaptation, distributed sequence-plus-tensor parallelism, and process-verified test-time compute. To provide a clean-room, reproducible engineering reference with 98% automated test coverage, I built VeritasRL in PyTorch 2.x and native C++20.
1. Critic-Free Verifiable-Reward RL (GRPO + DAPO + Schulman k3 KL)
Classical PPO requires maintaining a full-size value/critic network alongside the actor, doubling trainable activation and optimizer VRAM. Group Relative Policy Optimization (GRPO) and DAPO eliminate the critic entirely — saving 50% value-model VRAM — by sampling a group of G rollouts per prompt, scoring each trajectory with a deterministic non-hackable symbolic verifier, normalizing group-relative z-score advantages in native C++20, and applying asymmetric clipping ([0.80, 1.28]) to preserve exploration alongside Schulman's strictly non-negative k3 KL estimator. On our reproducible symbolic reasoning benchmark, critic-free GRPO/DAPO lifts Pass@1 from 6.25% to 50.00% (8.0× lift).
2. Weight-Decomposed Low-Rank Adaptation (DoRA) With Zero Serving Overhead
Standard LoRA couples magnitude and directional weight changes inside a single low-rank product, hitting an Eckart-Young approximation floor on high-rank target transformations. Implementing ICML 2024 Oral DoRA — decoupling row-wise magnitude vectors from unit-normalized directional low-rank matrices — achieved 563× lower MSE than rank-matched LoRA while cutting trainable parameters by 86.72% and folding cleanly back into a standard nn.Linear layer at inference time with exact 0.00000000 maximum numerical deviation.
3. 2D Distributed Causal Ring-Attention & Megatron Tensor Parallelism
Long-context reasoning rollouts bottleneck on quadratic S × S attention memory. VeritasRL pairs associative online-softmax Causal Ring-Attention (circulating KV shards across P ranks while maintaining exact running log-sum-exp accumulators) with Megatron-LM Column-to-Row Parallel MLPs, verifying both forward outputs and backward autograd gradients to within < 1.5e-6 against single-device PyTorch SDPA.
4. Step-Level Process Reward Models (PRM) for Test-Time Compute
Outcome-only voting fails when flawed derivations stumble onto plausible final numbers. Evaluating live local qwen3.5:4b chains-of-thought with an AST step-equation verifier and public problem invariants lifted multi-step Olympiad math accuracy from 62.50% (Pass@1) to 100.00% (PRM Best-of-6).
Explore Companion Frontier Repositories
Inspect the full code, C++20 kernels, and property-based Hypothesis suites in veritas-rl and live documentation, alongside axiom-zero (speculative decoding & Top-K SAEs). Connect with Nagarjuna Reddy Kamireddy via Forward Deployed & Frontier Track or gmailtonagarjuna@gmail.com.