Movie Summarization Based on Emotion Dynamics and Multimodal Information

Myungji Lee, Hongseok Kwon, WonKee Lee, Jong-Hyeok Lee

http://doi.org/10.5626/JOK.2022.49.9.735

Movie summarization is the task of summarizing a full-length movie by creating a short video summary containing its most informative scenes. This paper proposes an automatic movie summarization model that comprehensively considers the three main elements of the movie: characters, plot, and video information for movie summary. To accurately identify major events on the movie plot, we propose a Transformer-based architecture that uses the movie script"s dialogue information and the main characters" emotion dynamics information as model training features, and then combines the script and video information. Through experiments, the proposed method is shown to be helpful in increasing the accuracy of identifying major events in movies and consequently improves the quality of movie summaries.