JointConn-v2: Learning a joint vector field with diffusion transformers for cross-modal connectivity and dual-timestep modeling.

Abstract

This work revisits diffusion Transformers for relative-depth-conditioned and joint image-depth synthesis, focusing on two bottlenecks: (1) cross-modal attention degrades around edges and structural regions, causing geometric distortions; (2) the depth branch can be overly influenced by the semantic branch, leading to over-coupling and instability. To address these issues, we propose JointConn-v2 with GCM-WFM (Gated Cross-Modal Weighted Flow Matching), a unified framework for joint and depth-conditioned image synthesis. JointConn-v2 strengthens bidirectional guidance via Swap-Q cross-attention, explicitly injects 2D relative positions and edge energy into the attention logits through a Geometric Mask Bias, performs token-level spatial selection of where and in which direction to fuse cross-modal features via Regional Routing, and controls whether and how strongly cross-branch injection occurs through a sample-level Content Gate with residual fusion. On the training side, we introduce GCM-WFM, which regresses a joint vector field in the packed sequence space and incorporates temporal, geometric, gating, and routing terms into the objective with diagonal weights. Our approach achieves a better balance between edge controllability and cross-modal consistency. The current formulation treats depth as a normalized relative geometric signal and is not intended to preserve absolute metric scale. Code is available at https://github.com/haizhu12/JointConn-v2.