Semantic Scholar Open Access 2023 6592 sitasi

Adding Conditional Control to Text-to-Image Diffusion Models

Lvmin Zhang Anyi Rao Maneesh Agrawala

Lihat Sumber DOI

Abstrak

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.

Topik & Kata Kunci

Computer Science

Penulis (3)

Lvmin Zhang

Anyi Rao

Maneesh Agrawala

Format Sitasi

APA MLA BibTeX

Zhang, L., Rao, A., Agrawala, M. (2023). Adding Conditional Control to Text-to-Image Diffusion Models. https://doi.org/10.1109/ICCV51070.2023.00355

Akses Cepat

Lihat di Sumber doi.org/10.1109/ICCV51070.2023.00355

Informasi Jurnal

Tahun Terbit: 2023
Bahasa: en
Total Sitasi: 6592×
Sumber Database: Semantic Scholar
DOI: 10.1109/ICCV51070.2023.00355
Akses: Open Access ✓