RESEARCH · RESEARCH · #612
Regularized Emphatic TD (RETD) proposed to address constant-stepsize instability in ETD
The arXiv preprint (arXiv:2609.19170v1) constructs an ergodic two-state counterexample showing that emphatic TD (ETD) can have contracting mean dynamics while sampled trajectories exhibit positive top Lyapunov exponents, and introduces Regularized Emphatic TD (RETD): a normalized post-shock repair that stores the emphatic signal in a leaky scalar and releases a delayed correction. The paper proves almost-sure convergence for harmonic diminishing stepsizes, gives a conditional constant-stepsize moment-contraction bound, shows RETD recovers ETD fixed points under certain readouts, and validates results with large-scale experiments including a Baird point and 10,000-run trials.
KEY POINTS
- The arXiv preprint (arXiv:2609.19170v1) constructs an ergodic two-state counterexample showing that emphatic TD (ETD) can have contracting mean dynamics while sampled trajectories exhibit positive top Lyapunov exponents, and introduces Regularized Emphatic TD (RETD): a normalized post-shock repair that stores the emphatic signal in a leaky scalar and releases a delayed correction.
- The paper proves almost-sure convergence for harmonic diminishing stepsizes, gives a conditional constant-stepsize moment-contraction bound, shows RETD recovers ETD fixed points under certain readouts, and validates results with large-scale experiments including a Baird point and 10,000-run trials.
- Off-policy TD stability with constant stepsizes affects real-world RL training; RETD offers a provable repair that changes post-shock sampled dynamics and recovers ETD fixed points under readout conditions.
WHY IT MATTERS
Off-policy TD stability with constant stepsizes affects real-world RL training; RETD offers a provable repair that changes post-shock sampled dynamics and recovers ETD fixed points under readout conditions.