Video to Video Generative Adversarial Network for Few-Shot Learning Based on Policy Gradient.

Abstract

The development of sophisticated models for video-to-video synthesis has been facilitated by recent advances in deep reinforcement learning (RL) and generative adversarial networks (GANs). In this article, we propose RL-V2V-GAN, a new deep neural network approach based on RL for unsupervised conditional video-to-video synthesis. While preserving the unique style of the source video domain, our approach aims to learn a mapping from a source video domain to a target video domain. We train the model using policy gradient and employ convolutional long short-term memory (ConvLSTM) layers to capture the spatial and temporal information by designing a fine-grained GAN architecture and incorporating spatiotemporal adversarial goals. The adversarial losses aid in content translation while preserving style. Unlike traditional video-to-video synthesis methods requiring paired inputs, our proposed approach is more general because it does not require paired inputs. Thus, when dealing with limited videos in the target domain, that is, few-shot learning, it is particularly effective. Our experiments show that RL-V2V-GAN can produce temporally coherent video results. These results highlight the potential of our approach for further advances in video-to-video synthesis.