veritas-rl

ADR 0001: Critic-Free GRPO with DAPO Asymmetric Clipping & Deterministic Symbolic Verifiers

Context

Classical RLHF/PPO post-training requires loading four full models simultaneously (Actor $\pi_\theta$, Critic $V_\psi$, Reference $\pi_{\text{ref}}$, and Reward Model $R_\phi$), creating severe HBM bottlenecks and exposing reasoning tasks to neural reward hacking.

Decision

We adopt Group Relative Policy Optimization (GRPO) combined with DAPO Clip-Higher ($\varepsilon_{\text{low}} = 0.20, \varepsilon_{\text{high}} = 0.28$), Schulman’s exact non-negative $k_3$ KL estimator, and a deterministic symbolic execution verifier (verify_symbolic_rollout):

  1. Critic parameters are eliminated entirely (50% actor-critic value network memory saved).
  2. Group-level z-score normalization is offloaded to veritas_group_relative_advantages_f32 in csrc/veritas_kernels.cpp.
  3. Asymmetric upper clipping ($\varepsilon_{\text{high}} = 0.28$) allows low-probability verified reasoning steps to gain probability mass faster without destabilizing policy updates when $A_i < 0$.

Consequences