In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics.

Abstract

Synthesizing human motions from textual descriptions is a pivotal task for immersive digital applications, whereas existing methods inevitably face a critical trade-off. Large Language Model (LLM) based approaches typically excel at interpreting diverse, open-vocabulary text, however, they usually reproduce physically implausible motions. On the other hand, physics-aware models produce realistic movements but struggle with semantic complexity and generalization to novel open-vocabulary concepts. This division highlights a significant challenge to reconcile high-level semantic fidelity with low-level physical realism in motion generation. In order to obtain the best of both worlds, this work introduces the In-Context Model Predictive Generation (ICMPG) framework that attempts to integrate the principles from control theory into the motion sequence generation. Concretely, ICMPG reframes the motion synthesis into a Model Predictive Control (MPC)-like framework composed of two primary modules: a Context-Aware Motion Generation (CAMG) module that utilizes an LLM as a planner to decompose instructions and generate a set of candidate motion sequences from the input text, and a Model Predictive Generation (MPG) module. The MPG module iteratively refines the output by evaluating these candidates through a physical simulation to sequentially infer the composite reward based on both physically-constrained realism and semantic consistency with the textual description. By selecting the sequence with the highest reward to inform the sequential synthesis steps, ICMPG performs closed-loop, inference-time optimization to generate motions and improves their adaptability and robustness for a wide range of open-vocabulary commands, overcoming a key limitation of existing techniques. Extensive experimental evaluations demonstrate that our method enables robust zero-shot generalization to open-vocabulary commands, producing motions significantly more realistic and semantically faithful than state-of-the-art alternatives. The ICMPG framework not only bridges the divide between semantic interpretation and physical simulation but also offers the flexibility to incorporate various LLMs, paving the way to synthesize more versatile and controllable motions.