• lens

Pinhole lens parameters

By: SKY ENGINE AI
scroll down ↓to find out morepinhole-lens-parameters_4_resourcesTutorial

Pinhole Lens

In this tutorial you will get familiar with Pinhole Lens, which is a sensor that projects 3D points on the scene
to the image plane with perspective transformation. It is the most used type of lens, as its mechanism is the same
as in the human eye or commonly used cameras.

Agenda:

  • Pinhole lens model details
  • PinholeLens - render and undistortion
  • PinholeLens - parameters change

Scene setup

Let's use custom scene composer to set up the scene.

    from skyrenderer.cases.utils import SensorSceneComposer
    scene_composer = SensorSceneComposer(antialiasing_level=1)
    scene_composer.setup_scene()
    scene_composer.reset_camera_position([0, 0, 15])
    renderer_context = scene_composer.renderer_context
2025-02-24 15:57:34,297 | skyrenderer.scene.renderer_context |  INFO: Root paths:
- root path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer
- assets path: /dli/mount/assets
- config path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/config
- gpu sources path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/optix_sources/sources
- cache path: /dli/mount/cache
- ptx cache path: compiled_ptx/ptx
- ocio path: ocio_configs/aces_1.2/config.ocio

Pinhole lens model details

In generalized Lens projection model (INTRO_Lens), following re-projection steps are done:

  1. (u,v) are reprojected onto projection plane (z=1).
  2. Points are being undistorted, to match original ray directions (using distortion back mapping and
    precalculated distortion map).
  3. Based on re-projection radial distance and projection equation ray angle (theta) is calculated.
  4. Projection sphere coordinates are reconstructed (unit vector with angle theta between incident plane pi and
    sensor optical axis).
  5. Ray made from O and P' is generated.

Pinhole camera distortion (2nd step)

In real cameras, lenses introduce distortion, especially radial and tangential distortion, which can affect the
image. These distortions are most apparent at the edges of the image and arise due to the physical properties of
the lens used in the camera.
Distorted coordinates of a point are transformed with radial based and tangential distortion as follows:

    from skyrenderer.example_assistant.markdown_helpers import show_jupyter_picture
    from skyrenderer.utils import get_presentation_folder_from_assets
    get_presentation_folder_from_assets(scene_composer.renderer_context.assets_path, "TUT_se_lenses")
    show_jupyter_picture("TUT_se_lenses/pinhole-lens-distorted-coodrinates.png")
Asset is missing or invalid
2025-02-24 15:57:34,339 | skyrenderer.scene.renderer_context |  WARNING: assets_path() is deprecated and will be no longer supported, please consider using get_abs_path() instead

where

    show_jupyter_picture("TUT_se_lenses/pinhole-lens-distorted-coordinates-explication.png")
Asset is missing or invalid

Pinhole re-projection (3rd step)

In pinhole model incident angle and sensor matrix radius are connected with each other according to equation:

    show_jupyter_picture("TUT_se_lenses/pinhole-lens-re-projection.png")
Asset is missing or invalid

Camera extrinsic parameters

Extrinsic parameters describe the camera's position and orientation in the 3D world. They consist of:

  • Rotation ($R$): The orientation of the camera relative to the world.
  • Translation ($T$): The position of the camera center relative to the world origin.

The extrinsic parameters form the 3x4 camera extrinsic matrix $[R|T]$, which transforms world coordinates into
camera coordinates. They are stored in SceneLayoutNode/SceneNode transformations.

Camera intrinsic parameters

Intrinsic parameters stores the properties of the camera itself that affect how 3D points are mapped to
2D coordinates. They include:

  • Focal length ($f$) - the distance from the pinhole to the image plane.
  • Principal point ($c_x, c_y$) - the point where the principal axis intersects the image plane (often the image
    center).

Intrinsic parameters are passed in the lens parameters. More about the influence of intrinsic parameters on ray
generation can be found in INTRO_Lens.py tutorial.

PinholeLens - render and undistortion

In our implementation of the PinholeLens, we can adjust parameters presented in the section above.
In this example, we will adjust camera focal length and principal point, as well as all distortions parameters.

PinholeLens setup and visualization

    # Image resolution
    width = 1000
    height = 1000
    # Camera intrinsic parameters
    camera_fx = 750
    camera_fy = 750
    camera_cx = width / 2
    camera_cy = height / 2
    # Camera distortion parameters
    dist_k1 = -0.2
    dist_k2 = 0.1660696363841201
    dist_k3 = -0.1046838971475154
    dist_k4 = -0.05589875277472606
    dist_k5 = 0
    dist_k6 = 0
    dist_p1 = -0.00104636649317819
    dist_p2 = -0.00168027284332261
    dist_s1 = 0.01253
    dist_s2 = 0.05434
    dist_s3 = 0.08452
    dist_s4 = 0.01137
    dist_sampling_factor = 4
    dist_range_extension = 0

We can pass the above parameters to the PinholeLens through parameter provider, and pass parametrized lens to
VisibleLightRenderStep as in SENSOR_PinholeLens tutorial.

    from skyrenderer.render_chain import RenderChain, VisibleLightRenderStep, Denoiser
    from skyrenderer.render_chain.camera_steps.Lens.pinhole_lens import PinholeLens
    pinhole_params = PinholeLens.create_parameter_provider(
        renderer_context,
        fx=camera_fx,
        fy=camera_fy,
        cx_relative=0.5,
        cy_relative=0.5,
        dist_k1=dist_k1,
        dist_k2=dist_k2,
        dist_k3=dist_k3,
        dist_k4=dist_k4,
        dist_k5=dist_k5,
        dist_k6=dist_k6,
        dist_p1=dist_p1,
        dist_p2=dist_p2,
        dist_s1=dist_s1,
        dist_s2=dist_s2,
        dist_s3=dist_s3,
        dist_s4=dist_s4,
        dist_sampling_factor=dist_sampling_factor,
        dist_range_extension=dist_range_extension,
    )
    lens = PinholeLens(renderer_context, parameter_provider=pinhole_params)
    rs = VisibleLightRenderStep(
        renderer_context,
        lens=lens,
        origin_name="camera_CAM_NUL",
        target_name="top_node",
    )
    renderer_context.define_render_chain(
        RenderChain(render_steps=[rs, Denoiser(renderer_context)], width=width, height=height)
    )
    distorted_image = scene_composer.get_render()
    scene_composer.visualize(distorted_image)
Asset is missing or invalid
2025-02-24 15:57:38,125 | skyrenderer.utils.time_measurement |  INFO: Setup time: 3.75 seconds

2025-02-24 15:57:38,584 | skyrenderer.utils.time_measurement |  INFO: Context update time: 458 ms

2025-02-24 15:57:39,329 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:39,330 | skyrenderer.utils.time_measurement |  INFO: Render time: 745 ms

Image undistortion

Our Pinhole model implementation is compatible with the state-of-the-art solutions (e.g. OpenCV), therefore we can
perform undistortion on the rendered image with 3rd party library:

    import cv2 as cv
    import numpy as np
    camera_matrix = np.array([[camera_fx, 0, camera_cx], [0, camera_fy, camera_cy], [0, 0, 1]], np.float)
    dist_coeffs = np.array(
        [dist_k1, dist_k2, dist_p1, dist_p2, dist_k3, dist_k4, dist_k5, dist_k6, dist_s1, dist_s2, dist_s3, dist_s4],
        np.float,
    )
    img_undistorted = cv.undistort(distorted_image, camera_matrix, dist_coeffs)
    scene_composer.visualize(img_undistorted)
Asset is missing or invalid

PinholeLens - parameters change

fx, fy - focal length

    width = 450
    height = 450
    renders = {}
    scene_composer.setup_pinhole_lens(width, height, fx=200)
    renders["fx=200"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fx=915)
    renders["fx=915(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fx=1500)
    renders["fx=1500"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fx=450, fy=200)
    renders["fy=200"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fy=915)
    renders["fy=915(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fy=1500)
    renders["fy=1500"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fx=200, fy=200)
    renders["fx=fy=200"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fx=915, fy=915)
    renders["fx=fy=915(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, fx=1500, fy=1500)
    renders["fx=fy=1500"] = scene_composer.get_render()
    scene_composer.visualize_grid_desc(renders, shape=(3 * height, 3 * width), n_cols=3, font_scale=2)
Asset is missing or invalid
2025-02-24 15:57:40,090 | skyrenderer.utils.time_measurement |  INFO: Setup time: 127 ms

2025-02-24 15:57:40,540 | skyrenderer.utils.time_measurement |  INFO: Context update time: 449 ms

2025-02-24 15:57:41,610 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:41,611 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.07 seconds

2025-02-24 15:57:41,664 | skyrenderer.utils.time_measurement |  INFO: Setup time: 48 ms

2025-02-24 15:57:42,110 | skyrenderer.utils.time_measurement |  INFO: Context update time: 445 ms

2025-02-24 15:57:42,303 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:42,304 | skyrenderer.utils.time_measurement |  INFO: Render time: 193 ms

2025-02-24 15:57:42,355 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:42,776 | skyrenderer.utils.time_measurement |  INFO: Context update time: 420 ms

2025-02-24 15:57:42,962 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:42,962 | skyrenderer.utils.time_measurement |  INFO: Render time: 186 ms

2025-02-24 15:57:43,010 | skyrenderer.utils.time_measurement |  INFO: Setup time: 43 ms

2025-02-24 15:57:43,426 | skyrenderer.utils.time_measurement |  INFO: Context update time: 415 ms

2025-02-24 15:57:44,477 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:44,478 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.05 seconds

2025-02-24 15:57:44,527 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:44,945 | skyrenderer.utils.time_measurement |  INFO: Context update time: 417 ms

2025-02-24 15:57:46,013 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:46,014 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.07 seconds

2025-02-24 15:57:46,065 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-24 15:57:46,478 | skyrenderer.utils.time_measurement |  INFO: Context update time: 413 ms

2025-02-24 15:57:47,501 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:47,503 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.02 seconds

2025-02-24 15:57:47,553 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:47,972 | skyrenderer.utils.time_measurement |  INFO: Context update time: 418 ms

2025-02-24 15:57:48,147 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:48,148 | skyrenderer.utils.time_measurement |  INFO: Render time: 175 ms

2025-02-24 15:57:48,198 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:48,622 | skyrenderer.utils.time_measurement |  INFO: Context update time: 423 ms

2025-02-24 15:57:48,808 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:48,809 | skyrenderer.utils.time_measurement |  INFO: Render time: 186 ms

2025-02-24 15:57:48,857 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-24 15:57:49,298 | skyrenderer.utils.time_measurement |  INFO: Context update time: 440 ms

2025-02-24 15:57:49,482 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:49,483 | skyrenderer.utils.time_measurement |  INFO: Render time: 184 ms

cx_relative, cy_relative - center of projection

    renders = {}
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.2)
    renders["cx=0.2"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5)
    renders["cx=0.5(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.8)
    renders["cx=0.8"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.2)
    renders["cy=0.2"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cy_relative=0.5)
    renders["cy=0.5(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cy_relative=0.8)
    renders["cy=0.8"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.2, cy_relative=0.2)
    renders["cx=cy=0.2"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.5)
    renders["cx=cy=0.5(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.8, cy_relative=0.8)
    renders["cx=cy=0.8"] = scene_composer.get_render()
    scene_composer.visualize_grid_desc(renders, shape=(3 * height, 3 * width), n_cols=3, font_scale=2)
Asset is missing or invalid
2025-02-24 15:57:50,155 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:50,596 | skyrenderer.utils.time_measurement |  INFO: Context update time: 440 ms

2025-02-24 15:57:50,780 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:50,781 | skyrenderer.utils.time_measurement |  INFO: Render time: 184 ms

2025-02-24 15:57:50,830 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:51,268 | skyrenderer.utils.time_measurement |  INFO: Context update time: 437 ms

2025-02-24 15:57:51,449 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:51,450 | skyrenderer.utils.time_measurement |  INFO: Render time: 181 ms

2025-02-24 15:57:51,499 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:57:51,919 | skyrenderer.utils.time_measurement |  INFO: Context update time: 419 ms

2025-02-24 15:57:52,914 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:52,915 | skyrenderer.utils.time_measurement |  INFO: Render time: 995 ms

2025-02-24 15:57:52,963 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-24 15:57:53,371 | skyrenderer.utils.time_measurement |  INFO: Context update time: 407 ms

2025-02-24 15:57:54,390 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:54,391 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.02 seconds

2025-02-24 15:57:54,440 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-24 15:57:54,856 | skyrenderer.utils.time_measurement |  INFO: Context update time: 415 ms

2025-02-24 15:57:55,041 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:55,041 | skyrenderer.utils.time_measurement |  INFO: Render time: 184 ms

2025-02-24 15:57:55,098 | skyrenderer.utils.time_measurement |  INFO: Setup time: 51 ms

2025-02-24 15:57:55,509 | skyrenderer.utils.time_measurement |  INFO: Context update time: 410 ms

2025-02-24 15:57:55,690 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:55,691 | skyrenderer.utils.time_measurement |  INFO: Render time: 181 ms

2025-02-24 15:57:55,739 | skyrenderer.utils.time_measurement |  INFO: Setup time: 43 ms

2025-02-24 15:57:56,154 | skyrenderer.utils.time_measurement |  INFO: Context update time: 414 ms

2025-02-24 15:57:57,161 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:57,162 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.01 seconds

2025-02-24 15:57:57,210 | skyrenderer.utils.time_measurement |  INFO: Setup time: 43 ms

2025-02-24 15:57:57,640 | skyrenderer.utils.time_measurement |  INFO: Context update time: 429 ms

2025-02-24 15:57:57,829 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:57,830 | skyrenderer.utils.time_measurement |  INFO: Render time: 189 ms

2025-02-24 15:57:57,879 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-24 15:57:58,319 | skyrenderer.utils.time_measurement |  INFO: Context update time: 440 ms

2025-02-24 15:57:58,526 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:58,526 | skyrenderer.utils.time_measurement |  INFO: Render time: 206 ms

aperture_jitter and focus_distance

apperture_jitter is the parameter responsible for aperture related to focal length. The higher value, the smaller
DOF of a camera. If 0, infinite DOF is present. On the other hand, focus_distance is parameter holding a value of
plane distance in meters. It is used, when aperture jitter is nonzero.

    renders = {}
    scene_composer.setup_pinhole_lens(width, height, cx_relative=0.5, cy_relative=0.5)
    renders["aperture=0(default)"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=5)
    renders["aperture=1,focus_dist=5"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=15)
    renders["aperture=1,focus_dist=15"] = scene_composer.get_render()
    scene_composer.setup_pinhole_lens(width, height, aperture_jitter=1, focus_distance=25)
    renders["aperture=1,focus_dist=25"] = scene_composer.get_render()
    scene_composer.visualize_grid_desc(
        renders, shape=(height, 4 * width), n_cols=4, font_scale=1, text_position=(20, 20)
    )
Asset is missing or invalid
2025-02-24 15:57:59,322 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-24 15:57:59,746 | skyrenderer.utils.time_measurement |  INFO: Context update time: 424 ms

2025-02-24 15:57:59,926 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:57:59,927 | skyrenderer.utils.time_measurement |  INFO: Render time: 180 ms

2025-02-24 15:57:59,977 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-24 15:58:00,385 | skyrenderer.utils.time_measurement |  INFO: Context update time: 407 ms

2025-02-24 15:58:01,385 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:58:01,386 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.00 seconds

2025-02-24 15:58:01,435 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-24 15:58:01,848 | skyrenderer.utils.time_measurement |  INFO: Context update time: 412 ms

2025-02-24 15:58:02,027 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:58:02,028 | skyrenderer.utils.time_measurement |  INFO: Render time: 179 ms

2025-02-24 15:58:02,077 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-24 15:58:02,485 | skyrenderer.utils.time_measurement |  INFO: Context update time: 407 ms

2025-02-24 15:58:03,268 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-24 15:58:03,269 | skyrenderer.utils.time_measurement |  INFO: Render time: 783 ms

Summary

In this section you have learnt:

  • All parameters from pinhole lens mathematical theory can be set in PinholeLens's create_parameter_provider
    method.
  • Pinhole lens model follows generic projection model described in INTRO_Lens.
  • Model-based parameters of pinhole lens are compatible with popular CV libraries like OpenCV, and renders can be
    undistorted using 3rd party library.
2024-12-09_By Skyengine