DART: Dual Adaptive Residual Tracking for Low-Bias Advantage Estimation and Credit Assignment in AI Agents
Shahrad Mohammadzadeh, Amir-massoud Farahmand, Reihaneh Rabbany, Doina Precup
Proceedings of the Reinforcement Learning Conference (RLC)
Abstract
Proximal Policy Optimization (PPO) is the primary workhorse for complex continuous control and Reinforcement Learning from Human Feedback (RLHF). However, the reliability of Generalized Advantage Estimation (GAE) has recently been called into question: critic-free methods like RLOO and GRPO have raised the question of whether value-induced bias is doing more harm than the variance reduction it provides. We show this is the wrong dichotomy. The bottleneck is the dissemination bias of GAE, where value approximation error propagates into advantage estimates and misleads the policy gradient, a bias that can be corrected without discarding the critic. We propose Dual Adaptive Residual Tracking (DART), a versatile two-critic scheme that decouples the bias–variance trade-off. DART pairs a stable base critic learning from low-variance bootstrapped targets with a residual critic that tracks a high-fidelity Monte Carlo correction via a frozen-target mechanism. Theoretically, under linear function approximation assumptions, we prove that DART achieves a strictly smaller steady-state mean-squared error (MSE) upper bound than single-critic methods in high-variance regimes. Empirically, across all dm_control continuous control environments, DART matches or outperforms PPO in aggregate, with clear advantages in sparse and long-horizon tasks. We further provide initial evidence of DART’s effectiveness in LLM fine-tuning.