javirk/object-removal-partial-convolutions

Public
Removes specified objects from image
63 runs
Readme

Person Remover v2

Would you like to travel to a touristic spot and yet appear alone in the photos?

Person remover is a project that uses partial convolutions to remove people or other objects from
photos. For partial convolutions, the code by naoto0804 has been adapted,
whereas for segmentation models, either torch hub models or the code by MIT CSAIL Computer Vision,
have been used.

This project is capable of removing objects in images and video.

Python 3.7 and Pytorch 1.7.0 a have been used in this project.

How does it work?

A model with partial convolutions has been trained to fill holes in images. These instructions will you train a model in
your local machine. However, the training dataset that has been used for the model are not
publicly available. This dataset consists of 14900,
256x256x3 images. The code handles the creation of a hole in the center of the images and learns how to fill it with the
surrounding data.

By default the segmentation model is deeplab, which is
included in PyTorch by default.

Next steps

The quality of the filling heavily relies on the segmentator, a bigger mask that covers the whole person
is better than leaving out small parts such as one hand or one foot because the network tends to use these pixels to
inpaint the rest. Working on a way to expand the mask by pixel offsetting would very likely lead to better results.

Partial convolutions at times modify the lightning conditions of the image, which then results in big borders around the
filled areas. A smoother mix between inpainted and original images would be a quantitative step towards visual reality.

With the available weights, big artifacts are present in some cases due to the training methodology. A retraining will
be carried out to tackle this issue.

Author

License

This project is under Apache license. See LICENSE.md for more details.

Acknowledgments

Replicate Reproducible machine learning