Attention Map-Based Automatic Masking for Object Swapping in Diffusion Models

Soohyun Lee, Jongyoul Park

http://doi.org/10.5626/JOK.2025.52.4.284

latent diffusion model, stable diffusion, text-to-image model, object swapping, automatic masking AbstractDiffusion models have gained significant traction in the realm of text-to-image generation. The advent of Null-Text Inversion techniques has opened up new avenues for image editing by inverting real images into noise and applying modifications. However, most image editing methods, particularly those involving object manipulation, require user-defined masks, necessitating incorporation of an additional masking model into the pipeline. This complicates the inference process, which ideally should be streamlined within a single model. This paper proposed AutoMask, an attention-based automatic object masking method utilizing attention maps inherent in diffusion models to generate masks during the inference process. Unlike conventional approaches, AutoMask could leverage information obtained from the inversion step, eliminating the need for user intervention in masking. Experiments demonstrated the effectiveness of AutoMask in generating novel objects.