[FEATURE] Integrating Madrona Batch Renderer #1416

yuhongyi · 2025-07-15T02:08:42Z

This PR migrates all Madrona batch rendering changes from https://github.com/genesis-company/Genesis, which is temporary to the official repo. All the code has already been reviewed before except for a few changes I commented in this PR to resolve merge conflicts. See this link for previous PRs.

Description

This PR add support of batch rendering for Genesis.

Batch Renderer

[New API] Enable batch renderer
```
    scene = gs.Scene(
				...
        renderer = gs.options.renderers.BatchRenderer(
            use_rasterizer=True,
        )
    )
```
- use_rasterizer - If true, use rasterizer for batch renderer, otherwise raytracer will be used.
  Default to false
[New API] Batch render - Scene.render_all_cameras(). A tuple of 4-d arrays will be returned by this function, with len() of the tuple being the number of cameras, first dimension of each tensor being environment/world. Right now only rgb and depth is support. Normal and segmentation output will be supported later
[New Behavior] Single-camera render - Camera.render() still works, and it will return a tensor with output only from the camera from which render() is called. In the case of n_envs >0, the first dimension is the environment. When Camera.render() is called, actually Scene.batch_render() will be invoked internally, with outputs from all cameras cached, which means if Camera.render() is later called from another camera, the output will be returned immediately without any additional rendering. This behavior may change later to only render for the requesting camera.

Scene

When n_envs == 0 or is not specified, treat it as n_envs == 1 for batch renderer. The first dimension of outputs from camera.render() or scene.batch_render() is the environment.

Lights

[New API] Add a second Scene.add_light() with a new signature to add non-mesh light. This add_light() can be used to add point lights, spot lights and directional lights, which are supported by the lighting model of the batch renderer.
The signature of the new add_light() function

    def add_light(
        self,
        pos,
        dir,
        intensity,
        directional,
        castshadow,
        cutoff,
    ):

Light attenuation and light color will be supported in future updates.
There are 2 add_light() functions now, with different signatures. If BatchRenderer is used, only the new add_light() can be used, otherwise, there will be a warning and lights won’t be added.
If BatchRenderer is not used, only the old add_light() can be used, otherwise, there will be a warning and lights won’t be added.
The old add_light() might be renamed to add_mesh_light() in future and be supported in the ray tracing pipeline in the batch renderer, but right now only directional lights, point lights and spot lights are supported.
Non-batch rasterizer uses the lights defined in VisOptions. Now we have 3 different places adding lights for different pipelines, which is confusing and hard to maintain. The APIs to add lights will be unified in future.

Shadow

Shadow will be casted from lights added with castshadow=true. The raytracer pipeline of batch rendering can cast shadow from all lights, while the rasterizer pipeline can only cast shadow from the first light added with shadow casting enabled.

Camera

Cameras can be added by the existing Scene.add_camera() function, but aperture and focus_dis is not supported right now.
[New Benavior] The resolution of batch camera now uses the resolution of the first camera. Batch render with cameras with different resolutions will be supported in future.
[New Feature] Attach cameras to links - Attaching camera to links is now supported with batch rendering. Each camera can only be attached to one link, but the camera positions will be updated for each environment.
[New Feature] Camera follows entity - Following entity is now supported with batch rendering. Each camera an only follow one entity, but camera positions will be updated for each environment.
[New API] Support multi-environment pose update for cameras by adding a new parameter env_idx to Camera.set_pose() to specify the indices of environments to update camera transforms.
Transform updates from attaching cameras to links supersedes updates from following entities.

Related Issue

Resolves /issues/1232
Resolves /issues/21

How Has This Been / Can This Be Tested?

Tests have been done with example script single_franka.py and single_franka_batch_render.py, to cover combinations of

batch rasterizer / batch raytracer / non-batch rasterizer / non-batch raytracer
n_envs > 0 / n_envs == 0
camera attached to link / camera not attached to link
rgb / depth
[Performance benchmark] (https://www.notion.so/Madrona-benchmark-results-2070c06194e181b58753d2362c174bce)

Screenshots (if appropriate):

Checklist:

I read the CONTRIBUTING document.
I followed the Submitting Code Changes section of CONTRIBUTING document.
I tagged the title correctly (including BUG FIX/FEATURE/MISC/BREAKING)
I updated the documentation accordingly or no change is needed.
I tested my changes and added instructions on how to test it for reviewers.
I have added tests to cover my changes.
All new and existing tests passed.

Create tuples directly in BatchRender.render() Check input_types in Camera.set_pose() with set Update comments in batch_render.py and scene.py

* Improve math formulas and comments * Simplify the math to do y-forward quaternion transform * Black format scene.py and camera.py * Refactor ImageExporter to handle RGB and Depth separately with partial function * Black format image_exporter.py and batch_renderer.py

* Genesis now passes render_options to Madrona to specify the which outputs to generate, including RGB and Depth for now * Black format batch_renderer.py

…tiple environments (#4) * Fix transform issue with camera.set_pose by supporting batch transform with z_up_to_R Optimize quaternion transform of camera with Taichi kernel * Format files with black * Optimize R_to_quat

Create tuples directly in BatchRender.render() Check input_types in Camera.set_pose() with set Update comments in batch_render.py and scene.py

* Improve math formulas and comments * Simplify the math to do y-forward quaternion transform * Black format scene.py and camera.py * Refactor ImageExporter to handle RGB and Depth separately with partial function * Black format image_exporter.py and batch_renderer.py

* Genesis now passes render_options to Madrona to specify the which outputs to generate, including RGB and Depth for now * Black format batch_renderer.py

… wrong. It uses nenvs as index to be passed to attach. With the new implementation, index is not required. With batch rendering, cameras attached to links can be updated for each environment.

genesis/vis/rasterizer_context.py

yuhongyi · 2025-07-16T06:28:07Z

tests/test_render.py

-    for nenv, cam in enumerate(cams):
-        cam.attach(robot.get_link("hand"), T, nenv)
+    for cam in cams:
+        cam.attach(robot.get_link("hand"), T)


Passing camera index as env index is wrong.
The new camera attach implementation in this PR allows attaching a camera to the same link and update the camera location to follow the position of the link in each environment independently, so env index is not needed any more.

Did you add new tests for madrona batched renderer?

maybe we can start with some some tests for API, not necessarily the correctness test of rendering results

I tried, but it's more work than I thought. The test failed probably because the container needs to be updated due to extra dependencies. I think it's better to do it in a separate PR.

Added one test case for batch renderer

I think it's better to do it in a separate PR.

Note that we don't merge any new feature without accompanying unit tests. No exceptions.

But it seems you added one. Even if it only runs locally on your machine, this is fine for now.

The new camera attach implementation in this PR allows attaching a camera to the same link and update the camera location to follow the position of the link in each environment independently.

This was not the case before? From what I understand, it does not make sense anymore to create multiple cameras or is this still necessary?

This is not about multiple cameras.
My implementation could, for each camera, update the camera location to follow the link it is attaching to in all environments, but the current implementation added by some one 2 weeks ago only support one environment, by specifying an env_idx.
For example, the position of a link could be at different location in different environment. My implementation can move camera to different locations in different environments. The camera transforms are updated in batch.
I didn't know how the current implementation got approved. It is not a good implementation.
I added this feature per request from @YilingQiao

yeah, I merged that one as a temporay solution for pyrender. You can replace it with a better solution.

yuhongyi · 2025-07-16T06:30:32Z

genesis/vis/camera.py

        """
        self._attached_link = rigid_link
-        self._attached_offset_T = offset_T


Someone implemented a attaching camera 2 weeks ago, but the implementation is limited to only attach to the location of one environment, so this implementation is removed with the implementation in this PR, which allows attaching cameras to the same rigid link and update their location independently in each environment.

yuhongyi · 2025-07-16T06:32:24Z

genesis/vis/visualizer.py

-                self._rasterizer.update_scene()
-                self._rasterizer.render_camera(self._cameras[0])
+                if not self._use_batch_renderer:
+                    self._rasterizer.update_scene()


@AnisB I moved the update_scene() you added in a previous PR under the if-statement, since when batch renderer is enabled, _rasterizer, which is the legacy pyrender renderer, will not be used

Yes, this is the way to go.

Skipped _rasterizer.render_camera() when batch renderer is enabled, as we discussed.

yuhongyi · 2025-07-16T06:40:10Z

genesis/utils/geom.py

+        return _np_z_up_to_R(z, up, out)
+
+
+def _tc_z_up_to_R(z, up=None, out=None):


Added back the torch implementation.
Will do a thorough benchmark to find out which is faster.

Numba + CPU->GPU memory copy

Torch tensor + all GPU memory

Did you check whether the 2 implementation were matching?

Did you do the benchmark that you suggested?

If you are asking the whether the implementation of tc version matches the implementation of your new np version, the answer is no, but the tc version is based on the previous implementation and expanded it into the 2-dimension case. I didn't rewrite the algorithm.
I think we discussed this earlier and agreed that I can use the previously implemented torch version for this PR. That implementation was already tested and approved in the PR under genesis-company/Genesis.

The benchmarking will be done later on.

…is used.

YilingQiao · 2025-07-18T16:22:07Z

README.md

@@ -172,6 +172,8 @@ Genesis's development has been made possible thanks to these open-source project
 - [libccd](https://github.com/danfis/libccd): Reference for collision detection.
 - [PyRender](https://github.com/mmatl/pyrender): Rasterization-based renderer.
 - [LuisaCompute](https://github.com/LuisaGroup/LuisaCompute) and [LuisaRender](https://github.com/LuisaGroup/LuisaRender): Ray-tracing DSL.
+- [Madrona](https://github.com/shacklettbp/madrona): Madrona batch renderer
+- [Madrona-mjx](https://github.com/shacklettbp/madrona_mjx): Madrona MJX integration


can we put the in the same line?

Madrona and Madrona-mjx: Batch renderer backend

Yes, I could

examples/rigid/single_franka_batch_render.py

genesis/__init__.py

YilingQiao · 2025-07-18T16:36:35Z

tests/test_render.py

-    for nenv, cam in enumerate(cams):
-        cam.attach(robot.get_link("hand"), T, nenv)
+    for cam in cams:
+        cam.attach(robot.get_link("hand"), T)


Did you add new tests for madrona batched renderer?

YilingQiao · 2025-07-18T16:38:22Z

genesis/utils/image_exporter.py

+from genesis.utils.misc import tensor_to_array
+
+
+class FrameImageExporter:


when you add batched rendering test, can you also test this function?

Yes, and a fairly detailed documentation (I would expect at least 4-5 lines).

YilingQiao · 2025-07-18T16:39:08Z

tests/test_render.py

-    for nenv, cam in enumerate(cams):
-        cam.attach(robot.get_link("hand"), T, nenv)
+    for cam in cams:
+        cam.attach(robot.get_link("hand"), T)


maybe we can start with some some tests for API, not necessarily the correctness test of rendering results

duburcqa

You should write a unit test in this PR. It will only pass locally since Madrona must be installed manually, so you need add the decorator @pytest.mark.xfail(reason="gs-madrona must be built and installed manually for now.").

This way, once the python wheels are ready, I will just have to enable it systematically!

duburcqa · 2025-07-18T16:59:05Z

.gitmodules

@@ -7,3 +7,6 @@
 [submodule "genesis/ext/ParticleMesher"]
 	path = genesis/ext/ParticleMesher
 	url = https://github.com/ACMLCZH/ParticleMesher
+[submodule "genesis/ext/gs-madrona"]


I don't think gs-madrona should be a submodules. Python wheels will be compiled and distributed on PyPI completely independently from Genesis. gs-madrona will then be added to the Python requirements of Genesis simulator.

@YilingQiao Do you agree to remove gs-madrona as a submodule?

duburcqa · 2025-07-18T17:02:49Z

examples/rigid/single_franka_batch_render.py

+    scene.build(n_envs=N_ENVS)
+
+    # Create an image exporter
+    exporter = FrameImageExporter(OUTPUT_DIR)


Could you add a comment to explain what this frame image exporter does and why you are using it instead of the usual start|stop_recording public API? This would be helpful for practitioners.

I could. Basically this does batch exporting

Yes, but why it works differently for batch rendering?

Added comments to explain the reason to use FrameImageExporter for batch exporting

yuhongyi · 2025-07-18T17:03:03Z

@pytest.mark.xfail(reason="gs-madrona must be built and installed manually for now.").

Good idea. I'll add one

duburcqa · 2025-07-18T17:07:31Z

genesis/engine/scene.py

+            Whether to force render the scene.
+
+        Returns:
+            A tuple of tensors of shape (n_envs, H, W, 3) if rgb is not None, otherwise a list of tensors of shape (n_envs, H, W, 1) if depth is not None.


120 chars linewidth limit.

Split into 2 lines

duburcqa · 2025-07-18T17:08:56Z

genesis/ext/gs-madrona

Remove this submodule. It will be managed differently. On the long run we should get rid of all these submodules, it is not a good approach.

Removed. Yes, we should remove LuisaRender as well.

yuhongyi · 2025-07-18T20:47:44Z

@pytest.mark.xfail(reason="gs-madrona must be built and installed manually for now.").

Good idea. I'll add one

Added a test case using batch renderer and marked xfail. Tested locally

.gitmodules

duburcqa · 2025-07-19T05:58:54Z

.gitignore

@@ -148,3 +148,6 @@ dmypy.json

 # Pyre type checker
 .pyre/
+
+# Batch renderer output


Remove this change. I don't think it should be the default. It is the responsibility of the user to choose a folder outside the repository if desired.

it's not removed?

I'll push the change

tests/test_render.py

duburcqa · 2025-07-19T06:15:13Z

tests/test_render.py

+@pytest.mark.parametrize("n_steps", [1, 1000])
+@pytest.mark.required
+@pytest.mark.xfail(reason="gs-madrona must be built and installed manually for now.")
+def test_madrona_batch_rendering(use_rasterizer, render_all_cameras, n_envs, n_steps):


What about just update test_batched_offscreen_rendering to run for both pyrender and gs-madrona, and add 2 fixed cameras in the scene to make the test more comprehensive?

Good point. Let me see how I could do it.
I also found that test_batched_offscreen_rendering should be updated. I just went through the test case. It was not written using cameras and environments properly.

If I update test_batched_offscreen_rendering() , I'll have to make it temporarily xfail. How about we have 2 separate test functions for now. When the wheels are ready, and the test case for batched camera can pass, we merge them into one.

tests/test_render.py

duburcqa · 2025-07-19T06:23:45Z

tests/test_render.py

+    exporter = FrameImageExporter(export_dir)
+
+    # Only verify functionality works without crashes and output dimension matches for now
+    # To verify whether the output is correct pixel-wise, we need to use a more sophisticated test


This is not enough. See unit test test_batched_offscreen_rendering. You should check that the standard deviation for each frame is large enough, that 2 frames that are supposed to match are truly matching, and that 2 frames that are supposed to be different are truly different (easy if you have 2 cameras at different locations).

Having good extensive unit tests is more important than adding new features. We do not merge PRs if their unit tests are not comprehensive enough.

I was hesitant of adding something to compare mean and std for an image because I don't think this is the right way to compare rendering results. It might work as some point, but it is far from ideal, because rendering result may change very frequently for a system under development.

A rendering test system should not be part of CI but not part of the unit test framework. It should compare images pixel by pixel and show the percentage of differences, and allow users to update golden images when necessary.

But anyway, since we don't have such a system yet, I'm OK to just compare mean and std here.

tests/test_render.py

yuhongyi and others added 5 commits June 26, 2025 12:51

Integrate Madrona batch renderer to Genesis

1ad60a1

Create tuples directly in BatchRender.render() Check input_types in Camera.set_pose() with set Update comments in batch_render.py and scene.py

[Feature] Support Depth-only render (#5)

4da5b3e

* Genesis now passes render_options to Madrona to specify the which outputs to generate, including RGB and Depth for now * Black format batch_renderer.py

Genesis side option for aa (#6)

27939a9

yuhongyi force-pushed the support-batch-renderer branch from 97306db to 8964649 Compare July 15, 2025 18:01

yuhongyi and others added 10 commits July 15, 2025 19:08

Integrate Madrona batch renderer to Genesis

5d46c3a

Create tuples directly in BatchRender.render() Check input_types in Camera.set_pose() with set Update comments in batch_render.py and scene.py

[Feature] Support Depth-only render (#5)

a141b0c

* Genesis now passes render_options to Madrona to specify the which outputs to generate, including RGB and Depth for now * Black format batch_renderer.py

Merge branch 'main' into main-rebased-company-main

dd8e545

Fix issues after merge with main

40be6e5

Update submodule gs-madrona to latest

9c7193d

Minor bug fix and remove unnecessary changes

fedf301

black format

7f886a6

Use rasterizer for batch rendering in demo script. Fix a crash bug

04a5eee

Fix test_batched_mounted_camera_rendering. The previous test case was…

ca30205

… wrong. It uses nenvs as index to be passed to attach. With the new implementation, index is not required. With batch rendering, cameras attached to links can be updated for each environment.

yuhongyi force-pushed the support-batch-renderer branch from 8964649 to ca30205 Compare July 16, 2025 06:23

yuhongyi commented Jul 16, 2025

View reviewed changes

genesis/vis/rasterizer_context.py Outdated Show resolved Hide resolved

yuhongyi commented Jul 16, 2025

View reviewed changes

yuhongyi requested a review from AnisB July 16, 2025 06:34

yuhongyi commented Jul 16, 2025

View reviewed changes

yuhongyi marked this pull request as ready for review July 16, 2025 16:34

yuhongyi requested review from YilingQiao and duburcqa as code owners July 16, 2025 16:34

yuhongyi and others added 2 commits July 16, 2025 10:42

Merge remote-tracking branch 'company/main' into support-batch-renderer

bdb1642

Fix crash calling self._rasterizer.render_camera() when batch render …

3e30137

…is used.

yuhongyi changed the title ~~Merge from https://github.com/genesis-company/Genesis~~ [FEATURE] Merge from https://github.com/genesis-company/Genesis Jul 17, 2025

Fixed the crashin self._rasterizer.render_camera(camera)

b3de3a9

yuhongyi force-pushed the support-batch-renderer branch from 59a534f to b3de3a9 Compare July 17, 2025 02:09

Skip _rasterizer.render_camera() when batch renderer is enabled

861a07e

YilingQiao reviewed Jul 18, 2025

View reviewed changes

duburcqa requested changes Jul 18, 2025

View reviewed changes

duburcqa reviewed Jul 18, 2025

View reviewed changes

Fix comments and add test case

eb13cc0

yuhongyi added 2 commits July 18, 2025 13:55

Address PR comments

1e43116

Merge branch 'main' into support-batch-renderer

c3a2c55