3 Solution

What if we could generate automatically the images suited perfectly for the task at hand with the complete and always correct ground truth built-in?

We would like to show our attempt to achieve exactly this on the example of football players pose recognition. The goal is to train the model to accurately recognize the football players and their poses as human keypoints in 3D space on the real-life match footage, like below, having been trained exclusively on artificial, synthetic data. The images are rendered scenes, that are fully controlled by our renderer, so all kinds of ground truths can be provided, depending on the model's requirements.

[4]:

[5]:

[6]:

3.1 Agenda

• Dependences
• Context Configuration
• SKYENGINE RENDERER CONFIGURATION
  • The graphic assets
  • Assets configuration
  • Scene Tree Structure
  • Scene
  • Renderer Scenario
  • Renderer Datasource
• TRAINING
• EVALUATION
  • On Synthetic Data
  • On Real Data

4 Sky Engine renderer configuration

4.0.1 Context configuration

It is required to set the path where the assets (images, meshes, animations etc.) are stored. For convenience, the example assistant is configured. It will help with visualizations.

4.1 The graphic assets

In the Sky Engine pipeline the graphic assets, the building blocks for the scene, are prepared by a CG Artist using third-party software tools. Assets prepared for this scene:

1. Geometries

   The main format used in Sky Engine for carrying information about scene definition: models and their relative positions (or position ranges for randomization) is Alembic (.abc). Alembic exchange format developed by Sony Pictures Imageworks and Lucasfilm is widely used in the industry and is supported by most of the modern CG tools.

   For this scene an artist prepared:

   • Model of a rugby stadium,
   • Animation of a rugby player with keypoints,
   • Scene definition Alembic file - locators specifying positions of all the geometries, lights and camera. The player does not have a fixed position, it has a position range instead.

2. Materials

   Sky Engine by default uses a metallic-roughness PBR shader. The input maps for the shader can come from files or from the Substance archive (.sbsar). Sky Engine built-in support for Substance allows for parameter randomization and texture rendering on the fly in background.

   For this scene an artist prepared:

   • Substance archive for rugby players,
   • Substance archive for parts of the stadium: base, grass, logos, crowd,
   • Files with maps for banners, bumpers and screen.

3. Environmental mapping

   The background for this scene is a simple cloudy sky HDR.

4.1.1 Context configuration

For the Alembic assets prepared according to Sky Engine guidelines, the whole scene can be loaded and visualized without further configuration.

[17]:

4.1.2 Materials

Each loaded object needs to have a material assigned.

[28]:

Let's replace the gray background with a sky.

[31]:

4.1.3 Scene configuration

The Sky Engine renderer provides virtually endless possibilities to shuffle, multiply, randomize and organize the assets.

From one Alembic animation we are creating two teams of 20 players each.

[33]:

By default, all materials are drawn randomly. To create two proper teams we need to ensure that each team has the same shirt color which is different than the other team's color, while keeping all the other inputs (hair, skin color, socks color, shirt number etc.) random.

To achieve it, we need to put the the players into separate randomization groups and define their drawing strategy. The Substance archive input that controls shirt color is called "Colors_select". It needs to be the same (synchronized) inside the randomization group and different between groups. All the other inputs are kept randomized by default.

[36]:

If you looked closer on the picture above, you might notice that each player is in the exact same pose. By default, Sky Engine plays animations from Alembic files frame by frame, so we need to randomize this parameter.

[39]:

During the rugby match, players are not distributed roughly uniformly - they tend to gather in a group closer together. To make the scene look more natural, we can change the way the players' positions are drawn. Instead of drawing them uniformly, we can use random Gaussian random distribution. It is double-random, because first 𝜇 and 𝜎 are drawn, and then the positions for players are drawn also randomly with these parameters.

[42]:

[42]:

[42]:

[42]:

[42]:

This concludes configuration of materials, geometries and their positions.

4.1.4 Lights

The artist defined light positions in the Alembic scene definition. By default they have a constant intensity. We will randomize them.

4.1.5 Camera

Now we must improve the camera and its filters. The rendering process in Sky Engine is defined by a chain of render steps. The output of a step is an input of the next one. We define four render steps: * PinholeRenderStep - simple pinhole camera with randomized horizontal field of view (hfov, in degrees), which simulates random zoom, * Denoiser - AI-Accelerated Optix Denoiser, * Tonemapper - Optix tonemapper with randomized parameters - gamma and exposure, * GaussianBlurPostprocess - the train images should not be perfect. Images with different degree of blurring help the model to generalize better. Here we're using random Gaussian blur.

Additionally, we're reducing the output size to match the deep learning model required input size.

[52]:

[52]:

[52]:

[52]:

[52]:

4.1.6 Scene semantics

Last but not least, we must provide ground truth - information about scene semantics. This setup is designed for player detection with keypoints, so we must assign the semantic class only to players.

As mentioned before, the keypoints are already present in the player animation. Sky Engine by default calculates all the information about keypoints, if it receives them in the input assets, we just need to visualize them to be sure everything is configure correctly. Green keypoints are visible, red - hidden.

[55]:

[55]:

[55]:

[55]:

[55]:

[55]:

[55]:

[55]:

[55]:

[55]:

Everything is OK, so we can create a renderer datasource for training.

5 Training

Training configuration.

You've probably noticed that we set a tiny dataset: just 20 images. Normally it would naturally be at least a few thousands of images, but we can't wait now a few hours for the training to finish and we want to present the full Sky Engine workflow.

[epoch 1]: 100%|  | 18/18 [00:14<00:00, 1.26it/s, str=pose_head:7.064 loss_sum:7.064]
[epoch 1]: 100%|  | 2/2 [00:05<00:00, 2.71s/it, str=pose_head:6.501 loss_sum:6.501]
[inf]: 100%|  | 2/2 [00:00<00:00, 6.32it/s]

After each epoch we save a checkpoint and produce some inference example on inference data to be able to see the training progress. Images generated during longer training process as above on bigger datasets are presented as follows:

[77]:

[78]:

5.1 On real data

Lets load model for 3D pose estimation pretrained on large synthetic data and run inference on real rugby match.

We will need also player detection model which also was trained on the same artificial data with bouding boxes provided by the same renderer datasource.

Lets detect players and vizualize results

[85]:

After bounding boxes were generated, we can crop target objects and estimate the pose in 3D space. for data preprocessing we will use the same datasource we have used during training

[91]: