Marigold Normals + Intrinsics v1.1 — Unified Surface Geometry & Lighting Decomposition

Replicate Wrapper · Single-Image Scene Understanding

This model combines two official Marigold pipelines from ETH Zürich:

Marigold Normals v1.1 – Diffusion-based dense surface normal estimation

Marigold IID Lighting v1.1 – Intrinsic image decomposition (Albedo, Shading, Residual)

It exposes a simple image-to-multi-image API on Replicate. Upload one RGB image, and the model outputs both geometry (normals) and intrinsics (albedo, shading, residual).

What this unified model does

Given a single RGB input image, the model performs:

1. Surface Normal Estimation

Powered by MarigoldNormalsPipeline, it predicts dense per-pixel surface normals in camera space and returns them as an RGB-encoded normal map:

normals ∈ [-1, 1]³ → mapped to [0, 255] RGB

suitable for 3D reconstruction, relighting, segmentation, material replacement, etc.

1. Intrinsic Image Decomposition

Using Marigold IID Lighting, it produces:

albedo.png — lighting-independent surface color

shading.png — illumination and geometric shading

residual.png — remaining signal the IID model cannot fully explain

These outputs are physically meaningful and compatible with workflows such as relighting, room visualization, material replacement, and AR/VR scene understanding.

Typical Use Cases 🏠 Room / Interior Understanding

Derive wall/floor/ceiling normals for 3D alignment

Replace materials realistically using albedo + normals

Estimate lighting direction from shading maps

🏗️ 3D Reconstruction / Calibration

Use normals to detect plane orientation & geometry consistency

Feed normals + albedo into inverse rendering pipelines

🖌️ Relighting & Material Editing

Modify albedo while keeping shading intact

Relight scenes using shading maps

Use normals for environment-light estimation

👓 AR/VR Perception

Extract geometry + lighting cues from a single image

Build lightweight perception for robotics or XR applications

Filename Description normals.png RGB visualization of estimated surface normals. albedo.png Reflectance-only surface color (lighting removed). shading.png Illumination component only. residual.png Remaining unexplained signal.

Implementation Details Upstream Models

The wrapper loads both official models:

Component Upstream Source Normals prs-eth/marigold-normals-v1-1 Intrinsics prs-eth/marigold-iid-lighting-v1-1

Both are part of the ETH Zürich Marigold project.

Device & Precision

Uses CUDA + float16 if available

Falls back to CPU + float32

Ensures all outputs are clean RGB PNGs

Robust Output Extraction

The wrapper gracefully handles differences in Diffusers API:

tries .prediction,

then .pred_normals,

then .images[0].

Intrinsic Layers

The model returns:

albedo = surface reflectance

shading = lighting × geometry

residual = leftover information

These three layers recombine approximately to the original image.

Limitations

Works best on natural interior/exterior scenes with real lighting.

Noisy, low-light, or stylized images may degrade prediction quality.

Normal estimation may be unstable on texture-less regions (white walls, glass).

Intrinsics are not physically perfect but strongly useful for editing workflows.

License

Both upstream models are released under:

CreativeML Open RAIL++-M License (See the exact terms linked on the Hugging Face model pages.)

By using this unified Replicate model, you agree to comply with the upstream license, including restrictions on:

redistribution

commercial use

content limitations

Please review the license for your use case.

Liscence https://huggingface.co/prs-eth/marigold-normals-v1-1/blob/main/README.md https://huggingface.co/prs-eth/marigold-iid-lighting-v1-1/blob/main/README.md