zsxkib / st-mfnet

📽️ Increase Framerate 🎬 ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

  • Public
  • 50.8K runs
  • GitHub
  • Paper
  • License
Run with an API
Playground API Examples README Versions

Examples

View more

Run time and cost

This model costs approximately $0.021 to run on Replicate, or 47 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A40 GPU hardware. Predictions typically complete within 38 seconds. The predict time for this model varies significantly based on the inputs.

Readme

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

📽️ Increase Framerate 🎬

Duolikun Danier, Fan Zhang, David Bull

Accepted in CVPR 2022.

Project | Paper | arXiv | Video | Poster

Replicate

Welcome to the Video Frame Rate Enhancer on Replicate’s WebUI! 🎥 If you’ve ever wanted your videos to play back smoother, you’re in the right place. Let’s dive in and make some movie magic! 🌠

Model description 🧠

Our Video Frame Rate Enhancer works behind the scenes with a piece of advanced tech named STMFNet. Think of it as a smart tool that studies short segments of your video (sets of 4 frames) and predicts additional frames to slip in between. The result? Your video plays back at a higher frame rate, making movements look silky smooth, almost like those high-end slow-motion clips you adore on professional broadcasts. 🍿

Intended use 💡

Who is this for?

  1. Content Creators 🎨: Elevate your vlogs, tutorials, or any content by making playback smoother, giving it that premium feel.
  2. Hobby Videographers 📹: Transition your standard videos into breathtaking slow-motion works of art, minus the usual stutter.
  3. Retro Video Fans 🎞️: Revitalize older, jittery clips by enhancing the frame rate.

WebUI Controls: A Deep Dive

  • mp4: Upload your video magic here! We support .mp4 formats. 📥
  • keep_original_video_length:
  • Description: Do you want your enhanced video to keep its original length? This option controls that. If you turn this on (True), the model will adjust the frame rate so that the video remains the same duration while making it smoother. If turned off (False), you’ll then get to set a specific frame rate with custom_fps.

  • Default: True (on) ⏱️

  • custom_fps:

  • Description: This is where you dial in your desired video smoothness! FPS stands for “frames per second.” The higher the number, the smoother the video. But remember, this only kicks in if you’ve turned off the keep_original_video_length.
  • Range: 1 (minimal smoothness) to 240 (ultra-smooth playback)

  • Default: 24 FPS ⚙️

  • number_of_framerate_doubles:

  • Description: Here’s where the magic really happens! This control lets you decide how many times you want to multiply the frame rate. For instance, setting it to 1 will roughly double a 100-frame video to about 200 frames. Going for 2? Expect it to jump to around 400 frames, and so on.
  • Range: 1 (double the frames) to 4 (multiply by a whopping 16 times!)
  • Default: 1 (double it up!) 🔄

Ethical considerations 🌟

Using tools like ours is super fun, but let’s keep a few things in mind:

  1. Play Fair 🤝: Enhance videos you own or have rights to.
  2. Be Honest 🧐: If you’re sharing the enhanced videos, give viewers a heads-up that it’s been tweaked.
  3. Respect Privacy 🕶️: Respect boundaries. Not every moment needs to be in HD.

Caveats and recommendations ⚠️

  1. Quality vs. Smoothness: We make videos smoother, not clearer. Keep expectations in check! 📺
  2. Best for Mid to High Frame Rate Videos: Super slow originals might not get the full VIP treatment.
  3. Storage Space 📦: We use an interim .avi format before the final .mp4. A tad more storage is needed while processing.
  4. Using Custom FPS: If you’re setting a custom frame rate, remember to toggle off “keep_original_video_length”, or it’ll be overlooked. 🚫

We hope you love using our Video Frame Rate Enhancer! 🌈 If things get confusing or if you bump into snags, feel free to contact us! Happy enhancing! 🚀

Note: your video will lose its sound!

Example results

Paper

Citation

@InProceedings{Danier_2022_CVPR,
    author    = {Danier, Duolikun and Zhang, Fan and Bull, David},
    title     = {ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {3521-3531}
}

Acknowledgement from Danier et al.

Lots of code in this repository are adapted/taken from the following repositories:

We would like to thank the authors for sharing their code.