Helloooooo guys! In the previous post, we have gone through the introduction to image inpainting and the first GAN-based inpainting algorithm, Context Encoders. If you have not read the previous post, I highly recommend you to have a quick look of it first! This time, we will dive into another inpainting method which can be regarded as an improved version of Context Encoders. Let’s start!

Recall

Here, I briefly recall what we have learnt in the previous post.

• Deep semantic understanding of an image or the context of an image is important to the task of inpainting, and (channel-wise) fully-connected layer is one way to capture the context of an image.
• For image inpainting, visual quality of the filled images is much more important than the pixel-wise reconstruction accuracy. More specifically, as there is no model answer to generated pixels (we do not have the ground truth in real-world situations), we just want look-realistic filled images.

Motivation

• Existing inpainting algorithms can only handle low-resolution images because of the memory limitations and the training difficulty in high-resolution images.
• Although state-of-the-art inpainting method, Context Encoders, can successfully regress (predict) the missing parts with certain degree of semantic correctness, there is still room for improvement in the textures and details of the predicted pixels as shown in Figure 1.

Introduction

• Context Encoder is not perfect. i) texture details of the generated pixels can be further improved. ii) not able to handle high-resolution images.
• At the same time, Neural Style Transfer is a hot topic in which we would like to transfer the style of an image (style image) to another image with its same content (content image) as shown in Figure 2 below.

Figure 2. Example to illustrate the task of style transfer [2]

• Note that textures and colors can be regarded as a kind of styles. The authors of this paper employ the style transfer algorithm to enhance the texture details of the generated pixels.

#convolutional-network #image-inpainting #generative-adversarial #deep-learning