Pinhole Lens

In this tutorial you will get familiar with Pinhole Lens, which is a sensor that projects 3D points on the scene
to the image plane with perspective transformation. It is the most used type of lens, as its mechanism is the same
as in the human eye or commonly used cameras.

Agenda:

• Pinhole lens model details
• PinholeLens - render and undistortion
• PinholeLens - parameters change

Scene setup

Let's use custom scene composer to set up the scene.

Pinhole lens model details

In generalized Lens projection model (INTRO_Lens), following re-projection steps are done:

1. (u,v) are reprojected onto projection plane (z=1).
2. Points are being undistorted, to match original ray directions (using distortion back mapping and
   precalculated distortion map).
3. Based on re-projection radial distance and projection equation ray angle (theta) is calculated.
4. Projection sphere coordinates are reconstructed (unit vector with angle theta between incident plane pi and
   sensor optical axis).
5. Ray made from O and P' is generated.

Pinhole camera distortion (2nd step)

In real cameras, lenses introduce distortion, especially radial and tangential distortion, which can affect the
image. These distortions are most apparent at the edges of the image and arise due to the physical properties of
the lens used in the camera.
Distorted coordinates of a point are transformed with radial based and tangential distortion as follows:
\begin{equation}
\begin{bmatrix}
u \
v
\end{bmatrix} = \begin{bmatrix}
f_x x' + c_x \
f_y y' + c_y
\end{bmatrix}
\end{equation}

where

\begin{align}
\begin{bmatrix}
x' \
y'
\end{bmatrix} &= \begin{bmatrix}
x \cdot r_f + 2p_1 \cdot x \cdot y + p_2 \cdot (r^2+2x^2) + s_1 \cdot r^2 + s_2 \cdot r^4 \
y \cdot r_f + p_1 \cdot (r^2+2y^2) + 2p_2 \cdot x \cdot y + s_3 \cdot r^2 + s_4 \cdot r^4 \
\end{bmatrix} \
r_f &= \dfrac{1+k_1\cdot r^2+k_2\cdot r^4+k_3\cdot r^6}{1+k_4\cdot r^2+k_5\cdot r^4+k_6\cdot r^6} \
r &= x^2+y^2 \
\begin{bmatrix}
x\
y
\end{bmatrix} &= \begin{bmatrix}
X_c/Z_c \
Y_c/Z_c
\end{bmatrix} \
\end{align}

Pinhole re-projection (3rd step)

In pinhole model incident angle and sensor matrix radius are connected with each other according to equation:

\begin{align}
r &= f \cdot \tan{\theta} \to r = \tan \theta, & \text{when focal length is normalized} \
\end{align}

Camera extrinsic parameters

Extrinsic parameters describe the camera's position and orientation in the 3D world. They consist of:

• Rotation ($R$): The orientation of the camera relative to the world.
• Translation ($T$): The position of the camera center relative to the world origin.

The extrinsic parameters form the 3x4 camera extrinsic matrix $[R|T]$, which transforms world coordinates into
camera coordinates. They are stored in SceneLayoutNode/SceneNode transformations.

Camera intrinsic parameters

Intrinsic parameters stores the properties of the camera itself that affect how 3D points are mapped to
2D coordinates. They include:

• Focal length ($f$) - the distance from the pinhole to the image plane.
• Principal point ($c_x, c_y$) - the point where the principal axis intersects the image plane (often the image
  center).

Intrinsic parameters are passed in the lens parameters. More about the influence of intrinsic parameters on ray
generation can be found in INTRO_Lens.py tutorial.

PinholeLens - render and undistortion

In our implementation of the PinholeLens, we can adjust parameters presented in the section above.
In this example, we will adjust camera focal length and principal point, as well as all distortions parameters.

PinholeLens setup and visualization

We can pass the above parameters to the PinholeLens through parameter provider, and pass parametrized lens to
VisibleLightRenderStep as in SENSOR_PinholeLens tutorial.

Image undistortion

Our Pinhole model implementation is compatible with the state-of-the-art solutions (e.g. OpenCV), therefore we can
perform undistortion on the rendered image with 3rd party library:

PinholeLens - parameters change

fx, fy - focal length

cx_relative, cy_relative - center of projection

aperture_jitter and focus_distance

apperture_jitter is the parameter responsible for aperture related to focal length. The higher value, the smaller
DOF of a camera. If 0, infinite DOF is present. On the other hand, focus_distance is parameter holding a value of
plane distance in meters. It is used, when aperture jitter is nonzero.

Summary

In this section you have learnt:

• All parameters from pinhole lens mathematical theory can be set in PinholeLens's create_parameter_provider
  method.
• Pinhole lens model follows generic projection model described in INTRO_Lens.
• Model-based parameters of pinhole lens are compatible with popular CV libraries like OpenCV, and renders can be
  undistorted using 3rd party library.