wanjon/ardy2vrm

Generate text-to-motion with NVIDIA ARDY and export animated previews, NPZ data, and VRM Animation files for Three.js and Three-VRM.

Public
18 runs
License
GitHub
Paper
Playground API Examples README Versions

ARDY to VRM Animation

Generate 3D character motion from a natural-language prompt with NVIDIA ARDY. This Replicate wrapper exports animation data that can be used directly with Three.js, @pixiv/three-vrm, and @pixiv/three-vrm-animation.

Outputs

Each prediction returns three files:

  • preview — an animated GIF preview of the generated skeleton motion.
  • vrma — a binary VRM Animation (.vrma) file for VRM 1.0 avatars.
  • npz — pickle-free numeric motion data for custom pipelines and analysis.

The generated animation is avatar-independent: apply the .vrma file to your own compatible VRM avatar in Three.js.

Inputs

  • prompt — a motion description between 1 and 500 characters.
  • duration — requested motion duration in seconds. The current published version accepts values from 1 to 10 seconds.
  • seed — use the same seed and inputs to reproduce a result.
  • skeleton — core for the standard humanoid model, or experimental g1 robot retargeting.

Prompt examples

  • A person walks forward confidently, then waves with their right hand.
  • A tired person slowly sits down and looks toward the floor.
  • A dancer takes two light steps to the left and spins.
  • A person jogs forward, stops, and raises both arms in celebration.

Describe the action, direction, speed, mood, and any important ending pose. Short, concrete prompts usually produce the most controllable results.

Three.js and Three-VRM

Load the returned .vrma file with @pixiv/three-vrm-animation, create a clip for your avatar, and play it with a Three.js AnimationMixer:

import { AnimationMixer } from "three";
import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader.js";
import {
  VRMAnimationLoaderPlugin,
  createVRMAnimationClip,
} from "@pixiv/three-vrm-animation";

const loader = new GLTFLoader();
loader.register((parser) => new VRMAnimationLoaderPlugin(parser));

const gltf = await loader.loadAsync(vrmaUrl);
const vrmAnimation = gltf.userData.vrmAnimations[0];
const clip = createVRMAnimationClip(vrmAnimation, vrm);
const mixer = new AnimationMixer(vrm.scene);
mixer.clipAction(clip).play();

The export uses a right-handed, Y-up coordinate system. Humanoid local rotations and hips translation are animated. Horizontal root motion starts at the origin.

Limitations

  • Body motion only: no facial animation, gaze animation, or full finger tracking.
  • The g1 skeleton is experimental and has lower humanoid-joint coverage than core.
  • Results may require foot-contact cleanup for production use, especially after strong root-motion edits or retargeting to avatars with very different proportions.

Credits and licenses

This model is based on NVIDIA ARDY and uses the ARDY Core/G1 checkpoints. ARDY code is Apache-2.0; checkpoint use remains subject to the NVIDIA Open Model License.

The text encoder is built with Meta Llama 3 and LLM2Vec. Their respective upstream license and acceptable-use terms also apply. Review those terms before redistributing model weights.

Model created
Model updated