Deep Portrait Image Completion and Extrapolation

Xian Wu 1 Ruilong Li 1 Fang-Lue Zhang 2
Jian-Cheng Liu 1 Jue Wang 3 Ariel Shamir 4 Shi-Min Hu 1
1. Tsinghua Unviersity 2. Victoria University of Wellington 3. Megvii (Face++) Research 4. Interdisciplinary Center
IEEE Transactions on Image Processing 2019


General image completion and extrapolation methods often fail on portrait images where parts of the human body need to be recovered - a task that requires accurate human body structure and appearance synthesis. We present a two stage deep learning framework for tacking this problem. In the first stage, given a portrait image with an incomplete human body, we extract a complete, coherent human body structure through a human parsing network, which focuses on structure recovery inside the unknown region with the help of pose estimation. In the second stage, we use an image completion network to fill the unknown region, guided by the structure map recovered in the first stage. For realistic synthesis the completion network is trained with both perceptual loss and conditional adversarial loss. We evaluate our method on public portrait image datasets, and show that it outperforms other state-of-art general image completion methods. Our method enables new portrait image editing applications such as occlusion removal and portrait extrapolation. We further show that the proposed general learning framework can be applied to other types of images, e.g. animal images.



title={Deep Portrait Image Completion and Extrapolation},
author={Wu, Xian and Li, Rui-Long and Zhang, Fang-Lue and Liu, Jian-Cheng and Wang, Jue and Shamir, Ariel and Hu, Shi-Min},
journal={IEEE Transactions on Image Processing},