Instruction-Driven Multi-Weather Image Translation Based on a Large-Scale Image Editing Model.

Abstract

Weather image translation aims to convert sunny images into diverse weather scenes, addressing the challenge of the costly collections of multi-weather samples. Existing weather translation methods based on generative adversarial networks (GANs) suffer from limited generalization, often producing images lacking authenticity and diversity. In contrast, the emerging diffusion-based has surpassed GANs-across various visual tasks. This work pioneers diffusion models for weather translation with a novel Instruction-driven Multi-Weather Translation (InstructWT), built on the large image editing model, InstructPix2Pix and its zero-shot generalization capacities. We develop a user-friendly instruction set via prompt engineering and introduce a weather intensity factor for precise weather effect control well enhancing translation authenticity and diversity. A weather correlation-based blended editing preserves the original scene layout while physically based rendering of rain and snow incorporated further improve realism. Experiments on a public dataset Cityscapes demonstrate that InstructWT outperforms existing methods in authenticity and fidelity achieving Contrastive Language-Image Pre-Training (CLIP) image embedding cosine similarity of 0.8302 and directional CLIP similarity of 0.1598. Furthermore, several semantic segmentation algorithms fine-tuned using InsturctWT-augmented multi-weather datasets show significant performance gains under all complex weather conditions.