Semantics Introduction
In this case you will get familiar with strategies for semantic information handling. You will get to know, how to
define, use, organize and then utilize gathered semantic information data.
Agenda:
- Scene semantic description
- Object Labeling in SkyRenderer
- Advanced semantic information
Scene setup
Let's use custom scene composer to set up the scene.
from skyrenderer.cases.utils import RugbyPlayerSceneComposer scene_composer = RugbyPlayerSceneComposer(antialiasing_level=2048)
scene_composer.setup_scene()
scene_composer.log_info(scene_composer.renderer_context.layout())2025-02-04 14:45:17,429 | skyrenderer.scene.renderer_context | INFO: Root paths:
- root path: /dli/skyenvironment/skyrenderer/skyrenderer
- assets path: /dli/mount/assets
- config path: /dli/skyenvironment/skyrenderer/skyrenderer/config
- gpu sources path: /dli/skyenvironment/skyrenderer/skyrenderer/optix_sources/sources
- cache path: /dli/mount/cache
- ptx cache path: compiled_ptx/ptx
- ocio path: ocio_configs/aces_1.2/config.ocio
2025-02-04 14:45:17,785 | skyrenderer.scene.renderer_context | WARNING: Light with light_id=light_LIGHT_NUL already exists in the scene and will be replace with a new one.There can only be a single light for a single node.
2025-02-04 14:45:17,787 | skyrenderer | INFO:
top_node
|-- player_GEO_NUL
| +-- player_GEO
|-- camera_target_NUL
|-- camera_CAM_NUL
| +-- camera_CAM
+-- light_LIGHT_NUL
Scene semantics description
SkyRenderer is meticulously crafted for data science applications, featuring an integrated pixel-perfect
semantic ground truth tool. This ensures precise and accurate semantic labeling within your datasets, enhancing
the quality and reliability of your data analysis.
Scene Layout and Object Instances
Within the scene layout, each object node has the capability to generate its own instances. Managing these
instances is the role of Instancer, which oversees instance management, positioning, and population support.
Instance Management with Instancer
Each object that follows the same generation procedure is handled by a single Instancer. This management approach
is deeply connected to the scene's semantics, ensuring that typically, each Instancer is responsible for a
distinct class of objects. This organization enhances consistency and efficiency in handling various object
classes within the scene.
Object Labeling in SkyRenderer
In SkyRenderer, object labeling is defined by a tuple consisting of (class_id, instance_id). To effectively
label objects in the current scene:
- Assign Class IDs: Allocate specific class IDs to the instances you wish to label.
- Automatic Instance ID Assignment: Instance IDs are generated automatically, simplifying the labeling process
and reducing the potential for errors.
This structured approach to labeling ensures that each object is accurately identified and managed, facilitating
seamless data science workflows. To simply label an object you have only to perform first step of a process with
a single line.
Visualization rendered below is just a debug eye-candy. Real semantic information is being returned in
another part of render result - 'MASKS' - it is being returned as an integer numpy array of shape same as image
shape and pixel values equal to labels. We have one object of one class in scene, so masks contains only one
image.
scene_composer.renderer_context.set_semantic_class("player_GEO", 1)
scene_composer.visualize_grid(
[scene_composer.get_render(), scene_composer.get_semantic()], shape=(500, 1000, 4), n_cols=2
)
raw_render = scene_composer.renderer_context.render_to_numpy()
scene_composer.log_info(raw_render.keys())
scene_composer.log_info(raw_render["MASKS"].shape)2025-02-04 14:45:17,897 | skyrenderer.utils.time_measurement | INFO: Setup time: 102 ms
2025-02-04 14:45:18,101 | skyrenderer.utils.time_measurement | INFO: Context update time: 203 ms
2025-02-04 14:45:19,623 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 26 ms
2025-02-04 14:45:19,623 | skyrenderer.utils.time_measurement | INFO: Render time: 1.52 seconds
2025-02-04 14:45:19,723 | skyrenderer.utils.time_measurement | INFO: Setup time: 64 ms
2025-02-04 14:45:19,853 | skyrenderer.utils.time_measurement | INFO: Context update time: 129 ms
2025-02-04 14:45:20,969 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 4 ms
2025-02-04 14:45:20,970 | skyrenderer.utils.time_measurement | INFO: Render time: 1.12 seconds
2025-02-04 14:45:21,118 | skyrenderer.utils.time_measurement | INFO: Context update time: 41 ms
2025-02-04 14:45:22,214 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 4 ms
2025-02-04 14:45:22,214 | skyrenderer.utils.time_measurement | INFO: Render time: 1.09 seconds
2025-02-04 14:45:22,236 | skyrenderer | INFO: dict_keys(['RENDER', 'SEMANTIC', 'NORMAL_MAP', 'POSITION_MAP', 'RAW_RENDER', 'MASKS', 'INDIRECT_MASKS', 'TEXTURE_MASKS', 'INTERNAL_IDS', 'DEPTH_MAP', 'ALBEDO_MAP', 'EXTENDED_NORMAL_MAP', 'LIGHT_SOURCE_MAP', 'SHADER_TYPE', 'RENDER_STEP_MAPPING', 'INTRINSIC_CAMERA_PARAMETERS', 'EXTRINSIC_CAMERA_PARAMETERS', 'SCENE_TREE_INFO', 'KEYPOINTS', 'LIGHT_POWER'])
2025-02-04 14:45:22,236 | skyrenderer | INFO: (1000, 1000)
Advanced semantic information
Multiclass setup
Let's use another custom scene composer to generate scene with multiple instances grouped into two sets.
from skyrenderer.cases.utils import MultiPlayerSceneComposer scene_composer = MultiPlayerSceneComposer(antialiasing_level=2048)
scene_composer.setup_scene()
scene_composer.log_info(scene_composer.renderer_context.layout())
scene_composer.renderer_context.set_semantic_class("player_GEO", 1)
scene_composer.renderer_context.set_semantic_class("player_GEO_ANOTHER", 2)
scene_composer.visualize_grid(
[scene_composer.get_render(), scene_composer.get_semantic()], shape=(500, 1000, 4), n_cols=2
)2025-02-04 14:45:22,248 | skyrenderer.scene.renderer_context | INFO: Root paths:
- root path: /dli/skyenvironment/skyrenderer/skyrenderer
- assets path: /dli/mount/assets
- config path: /dli/skyenvironment/skyrenderer/skyrenderer/config
- gpu sources path: /dli/skyenvironment/skyrenderer/skyrenderer/optix_sources/sources
- cache path: /dli/mount/cache
- ptx cache path: compiled_ptx/ptx
- ocio path: ocio_configs/aces_1.2/config.ocio
2025-02-04 14:45:22,279 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'node_name': 'player_GEO_NUL'}
2025-02-04 14:45:22,280 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'node_name': 'camera_target_NUL'}
2025-02-04 14:45:22,280 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'node_name': 'camera_CAM_NUL'}
2025-02-04 14:45:22,281 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'node_name': 'light_LIGHT_NUL'}
2025-02-04 14:45:22,281 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'node_name': 'player_GEO'}
2025-02-04 14:45:22,282 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'path': PosixPath('player_kp_SCN.abc'), 'mesh_name': 'player_GEO', 'forbidden_frames': []}
2025-02-04 14:45:22,283 | skyrenderer.basic_types.provider.iprovider | WARNING: Provider for this config has been already initialized! There should be just one Provider created per unit source, consider fixing the scene.
Config:
{'node_name': 'camera_CAM'}
2025-02-04 14:45:22,285 | skyrenderer.scene.renderer_context | WARNING: Light with light_id=light_LIGHT_NUL already exists in the scene and will be replace with a new one.There can only be a single light for a single node.
2025-02-04 14:45:22,286 | skyrenderer | INFO:
top_node
|-- camera_target_NUL
|-- camera_CAM_NUL
| +-- camera_CAM
|-- light_LIGHT_NUL
|-- instances
| +-- player_GEO_NUL
| +-- player_GEO
+-- instances_another
+-- player_GEO_NUL_ANOTHER
+-- player_GEO_ANOTHER
2025-02-04 14:45:22,442 | skyrenderer.utils.time_measurement | INFO: Setup time: 156 ms
2025-02-04 14:45:22,598 | skyrenderer.utils.time_measurement | INFO: Context update time: 155 ms
2025-02-04 14:45:25,140 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 29 ms
2025-02-04 14:45:25,141 | skyrenderer.utils.time_measurement | INFO: Render time: 2.54 seconds
2025-02-04 14:45:25,234 | skyrenderer.utils.time_measurement | INFO: Setup time: 76 ms
2025-02-04 14:45:25,394 | skyrenderer.utils.time_measurement | INFO: Context update time: 160 ms
2025-02-04 14:45:28,497 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 32 ms
2025-02-04 14:45:28,497 | skyrenderer.utils.time_measurement | INFO: Render time: 3.10 seconds
Semantic metadata
The semantic metadata is mapped and stored in SceneSemanticReference and easily accessible.
As you can see, it's simply a dict mapping a semantic class to the instance numbers of its instances.
WARNING: Instances numbers are ordered by tree search order, not by class number.
semantic_metadata = scene_composer.renderer_context.current_scene.semantic_reference.classes_instances
scene_composer.log_info(semantic_metadata)2025-02-04 14:45:28,623 | skyrenderer | INFO: {1: [1, 2, 3, 4, 5], 2: [6, 7, 8, 9, 10]}
Visibility percentage
Objects in rendered scene are not always fully visible. In SkyRenderer there is possibility to calculate and
return ratio of visibility (in range (0,1)) associated with semantic class instances.
scene_composer.renderer_context.enable_occlusion_aware_semantics()
scene_composer.renderer_context.setup()
raw_render = scene_composer.renderer_context.render_to_numpy()
scene_composer.log_info(raw_render["VISIBILITY"])2025-02-04 14:45:28,702 | skyrenderer.utils.time_measurement | INFO: Setup time: 74 ms
2025-02-04 14:45:28,909 | skyrenderer.utils.time_measurement | INFO: Context update time: 205 ms
2025-02-04 14:45:32,239 | skyrenderer.utils.time_measurement | INFO: Key points calculation time: 30 ms
2025-02-04 14:45:32,240 | skyrenderer.utils.time_measurement | INFO: Render time: 3.33 seconds
2025-02-04 14:45:32,256 | skyrenderer | INFO: {1: 1.0, 2: 1.0, 3: 0.38123219373219375, 4: 1.0, 5: 0.9971143756558237, 6: 0.10261987147800297, 7: 1.0, 8: 1.0, 9: 1.0, 10: 0.9901003344481606}
Summary
In this section you have learnt:
- SkyRenderer by default return pixel perfect semantic info.
- Semantic output format is generic, so it can be converted to any format required by model.
- All semantic data is used in datasource to map and prepare ground truth.
- You can access semantic metadata and visibility from
renderer_context.current_scene.semantic_reference
and render['VISIBILITY'] respectively.