Selected Papers

A curated list of research papers we are reading.
June 2021
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, Dani Lischinski
Manipulate StyleGAN images with text. Multimodal transformers such as CLIP open up so many possibilities for text-based media editing. A new paradigm in creative tools that relies less on precise manipulation of sliders and anchor points and more on imaginative descriptions and prompts. Hello, post-slider interfaces?
View Paper
Enhancing Photorealism Enhancement
Stephan R. Richter, Hassan Abu AlHaija, Vladlen Koltun
We saw NVIDIA make the first steps towards adding ConvNets as a rendering pass in video games to perform real-time super-resolution with DLSS. This paper takes that approach to a next level by using image-to-image GANs applied to G-buffers from the game engine to generate temporally consistent photorealistic GTA V frames.
View Paper
Skip-Convolutions for Efficient Video Processing
Amirhossein Habibian, Davide Abati, Taco S. Cohen, Babak Ehteshami Bejnordi
There are decades of work in image and video compression taking advantage of insights on the human perceptual system and the redundancies in videos to reduce bandwidth with techniques such as DCT coding, chroma subsampling, and motion compensation. This is one of a few recent papers that uses an idea analogous to motion compensation in the context of DNN inference on video by only operating on the residuals between frames, significantly saving compute.
View Paper
Growing 3D Artefacts and Functional Machines with Neural Cellular Automata
Shyam Sudhakaran, Djordje Grbic, Siyan Li, Adam Katona, Elias Najarro, Claire Glanois, Sebastian Risi
Biology has been a consistent source of inspiration for new architectures and techniques in machine learning, from neural networks to genetic algorithms. Neural Cell Automata (NCAs) bring ideas from morphogenesis, the process by which biological organisms self-assemble from a single cell, to the world of deep neural networks and differentiable computing. In this paper, the authors apply NCAs to Minecraft to evolve complex buildings, and even machines, from a single block!
View Paper