Abstract
A persistent challenge in conditional image synthesis has been to generate diverse output images from the same input image despite only one output image being observed per input image. GAN-based methods are prone to mode collapse, which leads to low diversity. To get around this, we leverage Implicit Maximum Likelihood Estimation (IMLE) which can overcome mode collapse fundamentally. IMLE uses the same generator as GANs but trains it with a different, non-adversarial objective which ensures each observed image has a generated sample nearby. Unfortunately, to generate high-fidelity images, prior IMLE-based methods require a large number of samples, which is expensive. In this paper, we propose a new method to get around this limitation, which we dub Conditional Hierarchical IMLE (CHIMLE), which can generate high-fidelity images without requiring many samples. We show CHIMLE significantly outperforms the prior best IMLE, GAN and diffusion-based methods in terms of image fidelity and mode coverage across four tasks, namely night-to-day, 16x single image super-resolution, image colourization and image decompression. Quantitatively, our method improves Fréchet Inception Distance (FID) by 36.9% on average compared to the prior best IMLE-based method, and by 27.5% on average compared to the best non-IMLE-based general-purpose methods. More results and code are available on the project website.
If you find our work useful in your research, please consider citing:
@inproceedings{peng2022chimle,
title={CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis},
author={Shichong Peng and Alireza Moazeni and Ke Li},
booktitle={Advances in Neural Information Processing Systems},
year={2022}
}
Shichong Peng
Ph.D. Student
My research focuses on generative model and 3D neural rendering. My goal is to develop methods that can better aid 3D object generation and animation.
Ke Li
Professor of Computer Science
My research interests include Machine Learning, Computer Vision and Algorithms.