Koopman-Driven Linearized Model-Based Offline Planning With Application to Freeway Ramp Metering.

Abstract

This article proposes a novel model-based planning framework for freeway ramp metering (RM), denoted as Koopman-driven linearized model-based offline planning (KLMOP). This framework integrates the model predictive control (MPC) and offline reinforcement learning (RL) under assumptions of a linear Markov decision process (MDP) with the Koopman operator. KLMOP introduces a fully linearized control framework by learning and modeling the dynamics, reward function, and value function in a latent space through a Koopman-based latent dynamical model (KLDM) and a pessimistic value iteration (PEVI) algorithm. This formulation builds upon the connection between Koopman operator theory and linear MDP. Contrastive learning is employed to ensure the expressiveness and structural conditions of the latent representation in linear MDP, enabling accurate reward prediction and efficient policy optimization. The MPC-based planning policy, then, leverages these components to solve a linear MPC problem efficiently in the latent space. Extensive simulation studies demonstrate that KLMOP significantly improves computational efficiency and control performance as compared with existing baseline methods for RM control. This framework provides a theoretically grounded and computationally efficient approach to linearizing nonlinear control problems, and its learning-based design makes it adaptable to broader applications.