OneActor++: Consistent Multi-Subject Generation via Cluster-Conditioned Guidance.

Abstract

Text-to-image diffusion models benefit artists with high-quality image generation. Yet their stochastic nature hinders artists from creating consistent images of the same subject. To address this challenge, we previously proposed a cluster-guided tuning paradigm, termed OneActor, which efficiently performs consistent subject generation via a learned semantic cluster guidance to bypass the laborious backbone tuning. However, along with the rapid development, there is a growing demand for the capability to synthesize images containing multiple consistent subjects from user-provided image references, which exceeds the competence of OneActor. For this issue, we present a novel consistent multi-subject generation paradigm, termed OneActor++, to render multiple subjects from both textual prompts and image references. To start with, we pioneer a new multi-target formulation of the cluster guidance theory, paving the way to multi-subject identity preservation. Leveraging this foundation, we reinvent multi-subject modulization with a Mixture of Projectors (MoP) architecture. This design orchestrates a holistic representation of multiple subjects aggregating their commonality and individualities. To refine the cluster guidance predictions, we introduce an enhanced tuning recipe featuring an anchored noising strategy to rectify guidance misalignment, coupled with a drift regulation strategy to prevent optimization collapse. To unlock the new ability of image-referenced generation, we devise a one-step inversion mechanism as the final piece of the overall paradigm. Comprehensive experiments demonstrate that OneActor++ outperforms a variety of baselines in generating multiple consistent subjects from both prompt and image inputs. With excellent subject consistency, superior prompt alignment, and high image quality, OneActor++ showcases powerful practical possibilities across a wide range of application scenarios.