A Breakthrough in Deep Image Inpainting

A Breakthrough in Deep Image Inpainting

In this post, I would like to focus on the proposed contextual attention mechanism. Therefore, I briefly cover the coarse-to-fine network architecture, the WGAN adversarial loss, and the weighted L1 loss in above.

Welcome back guys! Happy to see you guys:) Last time, we realized that how copy-and-paste is embedded in CNNs for deep image inpainting. Can you get the main idea? If yes, Good! If no, Don’t worry! Today, we are going to dive into a breakthrough in deep image inpainting, for which contextual attention is proposed. By using contextual attention, we can effectively borrow information from distant spatial locations for reconstructing the local missing pixels. This idea is actually more or less the same as copy-and-paste. Let’s see how they can do that together!


In my previous post, I have introduced the shift-connection layer in which features from known regions act as references to the generated features inside missing regions to allow us to further refine the generated features for better inpainting results. Here, we assume that the generated features are reasonable estimations of the ground truth and suitable references are determined according to the similarity between features from known regions and the generated features inside missing regions.


For the task of image inpainting, the structure of CNNs cannot effectively model the long-term correlations between the missing regions and information given by distant spatial locations. If you are familiar with CNNs, you should know that the kernel size and the dilation rate control the receptive field at a convolutional layer and the network has to go deeper and deeper so as to see the entire input image. This means that if we want to capture the context of an image, we have to rely on deeper layers but we lose the spatial information as deeper layers always have smaller spatial size of features. So, we have to find a way to borrow information from distant spatial locations (i.e. understanding the context of an image) without going too deep into a network.

If you remember what dilated convolution is (We have covered in previous post), you will know that dilated convolution is one way to increase the receptive field at early convolutional layers without adding additional parameters. However, dilated convolution has its limitations. It skips consecutive spatial locations in order to enlarge the receptive field. Note that the skipped consecutive spatial locations are also crucial for filling in the missing regions.


This work shares similar network architecture, loss function and relevant techniques that we have covered before. For the architecture, the proposed framework consists of two generator networks and two discriminator networks. The two generators follow the fully convolutional networks with dilated convolutions. One generator is for coarse reconstruction and another one is for refinement. This is called standard coarse-to-fine network structure. The two discriminators also look at the completed images both globally and locally. The global discriminator takes the entire image as input while the local discriminator takes the filled region as input.

For the loss function, simply speaking, they also employ adversarial loss (GAN loss) and L1 loss (for pixel-wise reconstruction accuracy). For the L1 loss, they use a spatially discounted L1 loss in which a weight is assigned to each pixel difference and the weight is based on the distance of a pixel to its nearest known pixel. For GAN loss, they use a WGAN-GP loss instead of the most standard adversarial loss we have introduced. They claim that this WGAN adversarial loss is also based on L1 distance measure, hence the network is easier to train and the training process is more stable.

In this post, I would like to focus on the proposed contextual attention mechanism. Therefore, I briefly cover the coarse-to-fine network architecture, the WGAN adversarial loss, and the weighted L1 loss in above. Interested readers can refer to my previous posts and the paper of this work for further details.

image-inpainting image-processing deep-learning convolutional-network machine-learning

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Convolution Neural Network for Image Processing — Using Keras

A comprehensive guide towards working with CNN in Python. Image classification is the process of segmenting images into different categories based on their features

Preprocessing your images for machine learning (image recognition)

During my studies at JKU there was a task for preprocessing images for a machine learning project. It is necessary to clean the raw images…

Convolutional Neural Network (CNN) in the Easiest Way

CNN is an easy way to deal with complex image data, it plays a significant role in our lives, from facial recognition and analyzing docs to object detection.

Hire Machine Learning Developers in India

We supply you with world class machine learning experts / ML Developers with years of domain experience who can add more value to your business.

Changing Image Backgrounds Using Image Segmentation & Deep Learning

Changing Image Backgrounds Using Image Segmentation. Make the world your green screen. Hello readers! In this article I’ll be discussing another cool trick we can do with deep learning.