Question: How to correctly generate NOCS maps for BOP datasets (YCBV/HOPE)?

[YufengJin]

Describe the issue

Hi BlenderProc team,

I'm trying to generate NOCS maps using objects from BOP datasets like YCBV and HOPE. While the rendering pipeline for RGB and depth works well, the NOCS maps I get look incorrect or inconsistent (see image below).

I have a few questions:

1. Is NOCS rendering in BlenderProc only intended for ShapeNetCore objects?
2. Do I need to normalize the BOP object meshes (e.g., center at origin, scale to unit cube)?
3. Could the quality of the NOCS maps be affected by the mesh resolution or number of vertices?

Any advice on generating meaningful NOCS maps for BOP objects would be greatly appreciated.

Thanks in advance!

Minimal code example

mport blenderproc as bproc
import argparse
import os
import numpy as np

parser = argparse.ArgumentParser()
parser.add_argument('bop_parent_path', nargs='?', help="Path to the bop datasets parent directory")
parser.add_argument('bop_dataset_name', nargs='?', help="Main BOP dataset")
parser.add_argument('output_dir', nargs='?', help="Path to where the final files will be saved ")
args = parser.parse_args()

bproc.init()

# load specified bop objects into the scene
bop_objs = bproc.loader.load_bop_objs(bop_dataset_path=os.path.join(args.bop_parent_path, args.bop_dataset_name),
                                      mm2m=True,
                                      obj_ids=[1, 1, 3])

# load BOP datset intrinsics
bproc.loader.load_bop_intrinsics(bop_dataset_path=os.path.join(args.bop_parent_path, args.bop_dataset_name))
...
...
...
for _ in range(5):

    # Sample object poses and check collisions 
    bproc.object.sample_poses(objects_to_sample=bop_objs,
                              sample_pose_func=sample_pose_func,
                              max_tries=1000)

    # BVH tree used for camera obstacle checks
    bop_bvh_tree = bproc.object.create_bvh_tree_multi_objects(bop_objs)

    poses = 0
    # Render two camera poses
    while poses < 2:
        # Sample location
        location = bproc.sampler.shell(center=[0, 0, 0],
                                       radius_min=1,
                                       radius_max=1.2,
                                       elevation_min=1,
                                       elevation_max=89,
                                       uniform_volume=False)
        # Determine point of interest in scene as the object closest to the mean of a subset of objects
        poi = bproc.object.compute_poi(bop_objs)
        # Compute rotation based on vector going from location towards poi
        rotation_matrix = bproc.camera.rotation_from_forward_vec(poi - location,
                                                                 inplane_rot=np.random.uniform(-0.7854, 0.7854))
        # Add homog cam pose based on location an rotation
        cam2world_matrix = bproc.math.build_transformation_mat(location, rotation_matrix)

        # Check that obstacles are at least 0.3 meter away from the camera and make sure the view interesting enough
        if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.3}, bop_bvh_tree):
            # Persist camera pose
            bproc.camera.add_camera_pose(cam2world_matrix,
                                         frame=poses)
            poses += 1

    # render the cameras of the current scene
    data = bproc.renderer.render()
    data.update(bproc.renderer.render_nocs())

bproc.writer.write_hdf5(os.path.join(args.output_dir, 'noc_data'), data)

Files required to run the code

No response

Expected behavior

BlenderProc version

v2.8

[cornerfarmer]

Hey @YufengJin

see the docs of render_nocs():

    Colors each object based on its local coordinates.
    The coordinates [-1, 1] are mapped into the [0, 1] colorspace.
    It is therefore, recommended that all local vertex coordinates are in range [-1, 1].

However the bop objects are in mm, which means the local coordinates are way outside of [-1, 1].
One thing you could do in the beginning is to apply the mm -> m scaling directly to the local coordinates:

for obj in bop_objs:
    obj.persist_transformation_into_mesh(location=False, rotation=False, scale=True)

Afterwards, the nocs should be more meaningful.