camenduru / emage

EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling (Updated 1 year, 2 months ago)

  • Public
  • 107 runs
  • L40S
  • GitHub
  • Paper
  • License
Iterate in playground

Input

Video Player is loading.
Current Time 00:00:000
Duration 00:00:000
Loaded: 0%
Stream Type LIVE
Remaining Time 00:00:000
 
1x
*file

Input Audio

Output

Generated in

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

🐣 Please follow me for new updates https://twitter.com/camenduru
🔥 Please join our discord server https://discord.gg/k5BwmmvJJU
🥳 Please join my patreon community https://patreon.com/camenduru

🕸 Replicate

https://replicate.com/camenduru/minigpt4-video

📋 Tutorial

  • Register at https://smpl-x.is.tue.mpg.de and download the SMPL-X for Blender add-on. The ZIP release (https://drive.google.com/drive/folders/1ukbifhHc85qWTzspEgvAxCXwn9mK4ifr) file (smplx_blender_addon_20230921.zip) will include the required SMPL-X+FLAME model which is not included in the code repository.
  • Blender>Edit>Preferences>Add-ons>Install
  • Select downloaded SMPL-X for Blender add-on ZIP file (smplx_blender_addon-YYYYMMDD.zip) and install
  • Enable SMPL-X for Blender add-on
  • Enable sidebar in 3D Viewport>View>Sidebar
  • SMPL-X tool will show up in sidebar

🧬 Code

https://github.com/PantoMatrix/PantoMatrix/tree/main/scripts/EMAGE_2024

📄 Paper

https://arxiv.org/abs/2401.00374

🌐 Page

https://pantomatrix.github.io/EMAGE/

https://replicate.com