Automatic Controllable Colorization via Imagination

Xiaoyan Cong3, Yue Wu4, Qifeng Chen2, Chenyang Lei1,
1CAIR, HKISI-CAS, 2HKUST, 3Zhejiang University, 4Huawei Noah's Ark Lab
Paper arXiv Video Code
Interpolation end reference image.

Given a black-and-white input, our framework first synthesizes a semantically similar, spatially aligned, and instance-aware reference by mimicking the imagination process of human experts. Then the colorization module colorizes the black-and-white image with the guidance of reference.

Abstract

We propose a framework for automatic colorization that allows for iterative editing and modifications. The core of our framework lies in an imagination module: by understanding the content within a grayscale image, we utilize a pre-trained image generation model to generate multiple images that contain the same content. These images serve as references for coloring, mimicking the process of human experts. As the synthesized images can be imperfect or different from the original grayscale image, we propose a Reference Refinement Module to select the optimal reference composition. Unlike most previous end-to-end automatic colorization algorithms, our framework allows for iterative and localized modifications of the colorization results because we explicitly model the coloring samples. Extensive experiments demonstrate the superiority of our framework over existing automatic colorization algorithms in editability and flexibility.

Interpolation end reference image.

TL;DR

  • We propose a novel automatic colorization framework that leverages the pre-trained diffusion models. We introduce an imagination module that emulates human experts to synthesize semantically similar, structurally aligned, and instance-aware colorful references, with potential applications beyond colorization.
  • We demonstrate our novel automatic colorization framework exhibits remarkable controllable and user-interactive capabilities. We can also present diverse colorization results.
  • Compared to previous automatic colorization methods, our framework achieves state-of-the-art performance and generalization.

Diverse Results

We can synthesize diverse colorful references from Imagination Module, yielding diverse colorzation results.

Diverse Colorization

User Interaction & Editability

The top panel of (a) shows the black-and-white input, the reference, the colorization result, and the area marked by the user's mouse click indicating dissatisfaction with the colorization. The bottom panel of (a) displays multiple reference candidates generated by our Imagination Module. (b) Users select a preferred segment from the reference candidates to replace the unsatisfactory part of the reference, consisting of a new reference. (c) The colorization result after the adjustment.

User Interaction & Editability Effects.

Colorization Gallery

Comparison with the state-of-the-art automatic colorization methods on the COCO-Stuff vaidation dataset. Our method can generate more natural and photo-realistic colors. Please zoom in for the best view.

COCO-Stuff comparison.

Comparison with the state-of-the-art automatic colorization methods on the in-the-wild vaidation dataset which was collected randomly on the Internet. Our method can generate more natural and photo-realistic colors. Please zoom in for the best view.

COCO-Stuff comparison.

BibTeX

@article{cong2024imaginecolorization,
  author    = {Cong, Xiaoyan and Wu, Yue and Chen, Qifeng and Lei, Chenyang},
  title     = {Automatic Controllable Colorization via Imagination},
  journal   = {CVPR},
  year      = {2024},
}