Learning Long-form Movie Prior via Large Language Models.

Abstract

Concepts in movies like characters, objects, and their positions and interactions, can be viewed as following an implicit prior. They are notably too complex to be comprehensively learned. Recent large language models have exhibited capacities in modeling text content even visual locations. Can these models work for learning movie prior? Instead of operating on pixel space, it is possible to represent movie prior using texts and visual locations like object bounding boxes and keypoints, which can be tokenized for consumption by language models. Due to the scarcity of suitable data, we curate the Storyboard20K from movies to serve as a representative. It includes scripts, shot-by-shot keyframes, and fine-grained annotations of film sets and characters with consistent IDs, bounding boxes, and whole body keypoints. Hence, movies can be represented by a set of tokens and learned via next-token prediction. We validate that our approach can learn movie prior and sample professional storyboards, serving as guidance for visual story generation and even inspiration for movie creation.