Hyung Jin Kim  Jung Woo Lee

10.5626/JOK.2026.53.2.109

Reinforcement Learning Soft Actor-Critic TD3 Lyapunov stability

We introduce a lightweight Q-based stability regularizer for Actor–Critic methods, specifically Soft Actor-Critic (SAC) and TD3. Inspired by Lyapunov-style intuition—though without formal guarantees—this regularizer incorporates a one-sided hinge penalty into the policy loss. This penalty discourages updates that reduce the critic value at on-policy states. The training loop, which includes the replay buffer, target networks, and delayed policy updates, remains unchanged; the only additional computation required is extra forward passes through the target critic(s) during policy updates (one for SAC and two for TD3), resulting in a modest wall-clock time increase of approximately 1-2% and negligible memory overhead in our experiments. We evaluate our approach on MuJoCo tasks, including InvertedPendulum, InvertedDoublePendulum, and HumanoidStandup, using an identical environment-step budget. Performance is assessed along two dimensions: MeanRegret@K (lower is better) for early learning speed and MeanCost@K (lower is better) for safety, with an optional balanced composite measure of 0.5. Across tasks, the regularizer generally improves the trade-off between speed and safety, with particularly consistent enhancements when applied to TD3. Results for the HumanoidStandup task show higher variance due to sensitivity to contact; we present aggregate trends in the main text and detailed distributions in the appendix. Overall, this method should be considered a practical regulation mechanism that complements constraint-based approaches such as CPO and PPO-Lag. Limitations include sensitivity to the penalty weight  and reliance on the accuracy of critic estimations.