Learning to generate line drawings that convey geometry and semantics

Caroline Chan
Frédo Durand
Phillip Isola
MIT CSAIL

To appear in CVPR 2022
[Paper]
[Code]
[Video]
[Demo]


Abstract


This paper presents an unpaired method for creating line drawings from photographs. Current methods often rely on high quality paired datasets to generate line drawings. However, these datasets often have limitations due to the subjects of the drawings belonging to a specific domain, or in the amount of data collected. Although recent work in unsupervised image-to-image translation has shown progress on tasks with shape deformation and style transfer, the latest methods still struggle to generate compelling line drawings. To solve this problem, we observe that line drawings are encodings of scene information and seek to convey 3D shape and semantic meaning. We build these observations into a set of objectives and train an image translation network to map photographs into line drawings. We introduce a geometry loss which predicts depth information from the image features of a line drawing, and a semantic loss which matches the CLIP features of a line drawing with its corresponding photograph. Our approach outperforms state-of-the-art unpaired image translation and line drawing generation methods on creating line drawings both from arbitrary photographs and portraits.


Summary

Our method approaches line drawing generation as an unsupervised image translation problem which uses various losses to assess the information communicated in a line drawing. Our key idea is to view the problem as an encoding through a line drawing and to maximize the quality of this encoding through explicit geometry, semantic, and appearance decoding objectives. This evaluation is performed by deep learning methods which decode depth, semantics, and appearance from line drawings. The aim is for the extracted depth and semantic information to match the scene geometry and semantics of the input photographs.


Video



Bibtex


	      @inproceedings{chan2022drawings,
	      title={Learning to generate line drawings that convey geometry and semantics},
	      author={Chan, Caroline and Durand, Frédo and Isola, Phillip},
	      booktitle={CVPR},
	      year={2022}
	      }
      

Acknowledgements

Thank you to Hyojin Bahng for proofreading the paper. This webpage template was recycled from here.