Person Remover v2
Would you like to travel to a touristic spot and yet appear alone in the photos?
Person remover is a project that uses partial convolutions to remove people or other objects from photos. For partial convolutions, the code by naoto0804 has been adapted, whereas for segmentation models, either torch hub models or the code by MIT CSAIL Computer Vision, have been used.
This project is capable of removing objects in images and video.
Python 3.7 and Pytorch 1.7.0 a have been used in this project.
How does it work?
A model with partial convolutions has been trained to fill holes in images. These instructions will you train a model in your local machine. However, the training dataset that has been used for the model are not publicly available. This dataset consists of 14900, 256x256x3 images. The code handles the creation of a hole in the center of the images and learns how to fill it with the surrounding data.
By default the segmentation model is deeplab, which is included in PyTorch by default.
Next steps
The quality of the filling heavily relies on the segmentator, a bigger mask that covers the whole person is better than leaving out small parts such as one hand or one foot because the network tends to use these pixels to inpaint the rest. Working on a way to expand the mask by pixel offsetting would very likely lead to better results.
Partial convolutions at times modify the lightning conditions of the image, which then results in big borders around the filled areas. A smoother mix between inpainted and original images would be a quantitative step towards visual reality.
With the available weights, big artifacts are present in some cases due to the training methodology. A retraining will be carried out to tackle this issue.
Author
License
This project is under Apache license. See LICENSE.md for more details.
Acknowledgments
- naoto0804 for partial convolutions initial code
- CSAIL Vision for segmentation models