[Paper + Experimentation] CartoonGAN applied to Hong Kong landscapes using Streamlit
[Paper + Experimentation] CartoonGAN applied to Hong Kong landscapes using Streamlit
Material discussed in this blog:
- CartoonGAN paper: CartoonGAN: Generative Adversarial Networks for Photo Cartoonization
- CartoonGAN-TensorFlow2: A GitHub repo containing code for training a CartoonGAN model
- Streamlit: An open-source framework to create ML/DS web apps
- www.cartoongan.ai: Upload a picture for cartoonization, and apply one of the CartoonGAN models available
From the original paper:
CartoonGAN is able to produce high-quality cartoon stylization using the data of individual artists for training, which are easily obtained from cartoon videos, since our method does not require paired images. Different artists have their unique cartoon styles, which can be effectively learned by CartoonGAN.
Our method takes a set of photos and a set of cartoon images for training.
Two novel losses suitable for cartoonization are proposed: (1) a semantic content loss, which is formulated as a sparse regularization in the high-level feature maps ofthe VGG network to cope with substantial style variation between photos and cartoons, and (2) an edge-promoting adversarial loss for preserving clear edges.
Architecture of CartoonGAN (essentially CNNs; note that the Generator Network is somewhat similar to an auto-encoder but using a different loss, i.e. it doesn’t try to reconstruct the input but something similar in content which cannot be distinguished from a real cartoon image):
A great GitHub repo has re-implemented the model described in the paper, and allows one to train a CartoonGAN model with his own pictures (cartoons and photos), which is what I did:
- Training photos: Selfie Data Set
- Training cartoons: animeface-character-dataset
Application of the obtained CartoonGAN models (different set of parameters, models frozen at different epochs during the training) on Hong Kong landscapes (pictures taken while hiking around):
(Top) Original picture; (Bottom) The same, cartoonized.
One can explore more on my CartoonGAN Explorer - www.cartoongan.ai, a small web app I wrote with Streamlit (thanks Tomas Thornquist for recommending it to me). There, you can upload a picture and apply one of the models available to cartoonize your picture!
Beware that I was quite cheap on the EC2 instance on which this small web app is running, and that there are no supervisor or queuing mechanism, which means that if at some point there are too many users the Streamlit server can crash due to lack of memory.