UCLA

We have just witnessed an unprecedented booming in the research area of artistic style transfer ever since Gatys et. al. introduced the neural method. One of the remaining challenges is to balance a trade-off among three critical aspects---speed, flexibility, and quality: (i) the vanilla optimization-based algorithm produces impressive results for arbitrary styles, but is unsatisfyingly slow due to its iterative nature, (ii) the fast approximation methods based on feed-forward neural networks generate satisfactory artistic effects but bound to only a limited number of styles, and (iii) feature-matching methods like AdaIN achieve arbitrary style transfer in a real-time manner but at a cost of the compromised quality. We find it considerably difficult to balance the trade-off well by merely using a single feed-forward step and ask, instead, whether there exists an algorithm that could adapt quickly to any style, while the adapted model maintains high efficiency and good image quality. Motivated by this idea, we propose a novel method, coined MetaStyle, which formulates the neural style transfer as a bilevel optimization problem and combines learning with only a few post-processing update steps to adapt to a fast approximation model. The qualitative and quantitative analysis in the experiments demonstrates that the proposed approach achieves high-quality arbitrary artistic style transfer effectively, with a good trade-off among speed, flexibility, and quality. We also investigate the style-free representation learned by MetaStyle. Apart from style interpolation and video style transfer, we also implement well-known style transfer methods and examine the style transfer results after substituting the original content image inputs with their style-free representation learned by MetaStyle. This could be thought of as inserting a preprocessing step to the content transformation branch. We show in the experiments that models trained using the MetaStyle preprocessing step produce consistently lower style loss and total loss, with a slightly higher content loss, compared to its counterparts without MetaStyle processing. And therefore, the stylized results achieve a better balance in appropriately combining semantics and styles. This shows that MetaStyle also learns a more general content representation in terms of adapting different artistic styles.