Fokker-Planck Soft Actor-Critic.

Abstract

Learning expressive and multimodal policies is essential for solving complex continuous control tasks. However, most reinforcement learning (RL) algorithms rely on unimodal or factorized Gaussian policies, limiting their representational flexibility. While soft actor-critic (SAC) encourages stochasticity through the maximum entropy RL framework, its reverse KL-based update biases the policy toward dominant modes, often resulting in mode collapse. In this work, we propose Fokker-Planck SAC (FP-SAC), a principled policy optimization algorithm that formulates soft policy improvement through stochastic differential equations and derives a distribution-level objective from the corresponding Fokker-Planck (FP) equation. By minimizing the residual of this partial differential equationusing physics-informed learning and leveraging the expressiveness of normalizing flows, FP-SAC enables stable and accurate learning of multimodal policies. Experiments on multigoal environments, MuJoCo and Meta-World benchmarks, demonstrate that FP-SAC captures multimodal behaviors more effectively and achieves higher task performance with greater stability compared to existing approaches.