• lens

Fisheye lens parameters

By: SKY ENGINE AI
scroll down ↓to find out morefisheye-lens-parameters_3_resourcesTutorial

Fisheye Lens Parameters

This tutorial you will get to know about parameters in FisheyeLens. You will learn about the existing fisheye lens
parameters in SkyRenderer, their purpose and mathematical background. You will see how the change of the
parameters is affecting the render output.

Agenda:

  • Fisheye lens model details
  • FisheyeLens - render and undistortion
  • FisheyeLens - render and undistortion

Scene setup

Let's use custom scene composer to set up the scene.

    from skyrenderer.cases.utils import SensorSceneComposer
    scene_composer = SensorSceneComposer(antialiasing_level=1)
    scene_composer.setup_scene()
    renderer_context = scene_composer.renderer_context
2025-02-25 19:30:55,369 | skyrenderer.scene.renderer_context |  INFO: Root paths:
- root path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer
- assets path: /dli/mount/assets
- config path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/config
- gpu sources path: /home/skyengine/anaconda/lib/python3.6/site-packages/skyrenderer/optix_sources/sources
- cache path: /dli/mount/cache
- ptx cache path: compiled_ptx/ptx
- ocio path: ocio_configs/aces_1.2/config.ocio

Fisheye lens model details

In generalized Lens projection model (INTRO_Lens), following re-projection steps are done:

  1. (u,v) are reprojected onto projection plane (z=1).
  2. Points are being undistorted, to match original ray directions (using distortion back mapping and
    precalculated distortion map).
  3. Based on re-projection radial distance and projection equation ray angle (theta) is calculated.
  4. Projection sphere coordinates are reconstructed (unit vector with angle theta between incident plane pi and
    sensor optical axis).
  5. Ray made from O and P' is generated.

Fisheye camera distortion (2nd step)

In real cameras, lenses introduce distortion, especially radial and tangential distortion, which can affect the
image. These distortions are most apparent at the edges of the image and arise due to the physical properties of
the lens used in the camera. In fisheye camera only radial distortions are typically modelled and follow equation:

    from skyrenderer.example_assistant.markdown_helpers import show_jupyter_picture
    from skyrenderer.utils import get_presentation_folder_from_assets, get_case_root_paths
    get_presentation_folder_from_assets(get_case_root_paths()["assets_root"], "TUT_se_lenses")
    show_jupyter_picture("TUT_se_lenses/fisheye-camera-distortion.png")
fisheye-lens-parameters_1_resourcesTutorial

Fisheye re-projection (3rd step)

In fisheye lens equidistant projection is used. In that model incident angle and sensor matrix radius are
connected with each other according to equation:

    show_jupyter_picture("TUT_se_lenses/fisheye-camera-re-projection.png")
fisheye-lens-parameters_2_resourcesTutorial

Camera extrinsic parameters

As for every other camera in SkyRenderer, camera extrinsic parameters (translation, rotation in global
coordinate system) are handled by SceneLayout and it's transformations.

FisheyeLens - render and undistortion

In our implementation of the FisheyeLens, we can adjust parameters presented in the section above. For more
information about the parameters, check out FisheyeLens class documentation.

In this example, we will adjust camera focal length and principal point, as well as all distortions parameters.

FisheyeLens setup and visualization

    # Image resolution
    width = 1000
    height = 1000
    # Camera intrinsic parameters
    camera_fx = 400
    camera_fy = 400
    camera_cx = width / 2
    camera_cy = height / 2
    # Camera distortion parameters
    dist_k1 = -8 * 1e-3
    dist_k2 = 1 * 1e-5
    dist_k3 = -3.14 * 1e-8
    dist_k4 = 5.76 * 1e-12
    dist_sampling_factor = 8
    dist_range_extension = 0

We can pass the above parameters to the FisheyeLens through parameter provider, and pass parametrized lens to
VisibleLightRenderStep as in SENSOR_FisheyeLens tutorial.

    from skyrenderer.render_chain import RenderChain, VisibleLightRenderStep, Denoiser
    from skyrenderer.render_chain.camera_steps.Lens.fisheye_lens import FisheyeLens
    lens = FisheyeLens(
        renderer_context,
        FisheyeLens.create_parameter_provider(
            renderer_context,
            fx=camera_fx,
            fy=camera_fy,
            cx_relative=0.5,
            cy_relative=0.5,
            dist_k1=dist_k1,
            dist_k2=dist_k2,
            dist_k3=dist_k3,
            dist_k4=dist_k4,
            dist_sampling_factor=dist_sampling_factor,
            dist_range_extension=dist_range_extension,
        ),
    )
    rs = VisibleLightRenderStep(
        renderer_context,
        lens=lens,
        origin_name="camera_CAM_NUL",
        target_name="top_node",
    )
    renderer_context.define_render_chain(
        RenderChain(render_steps=[rs, Denoiser(renderer_context)], width=width, height=height)
    )
    distorted_image = scene_composer.get_render()
    scene_composer.visualize(distorted_image)
fisheye-lens-parameters_3_resourcesTutorial
2025-02-25 19:30:58,423 | skyrenderer.utils.time_measurement |  INFO: Setup time: 2.97 seconds

2025-02-25 19:30:58,930 | skyrenderer.utils.time_measurement |  INFO: Context update time: 506 ms

2025-02-25 19:30:59,627 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:30:59,628 | skyrenderer.utils.time_measurement |  INFO: Render time: 696 ms

Image undistortion

Our Fisheye model implementation is compatible with the state-of-the-art solutions (e.g. OpenCV), therefore we can
perform undistortion on the rendered image with 3rd party library:

    import cv2 as cv
    import numpy as np
    camera_matrix = np.array([[camera_fx, 0, camera_cx], [0, camera_fy, camera_cy], [0, 0, 1]], np.float)
    dist_coeffs = np.array([dist_k1, dist_k2, dist_k3, dist_k4], np.float)
    map_1, map_2 = cv.fisheye.initUndistortRectifyMap(
        camera_matrix, dist_coeffs, None, camera_matrix, (width, height), cv.CV_32FC1
    )
    img_undistorted = cv.remap(distorted_image, map_1, map_2, cv.INTER_LINEAR)
    scene_composer.visualize(img_undistorted)
fisheye-lens-parameters_4_resourcesTutorial

FisheyeLens - parameter change

fx, fy - focal length

    width = 450
    height = 450
    renders = {}
    scene_composer.setup_fisheye_lens(width, height, fx=200)
    renders["fx=200"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fx=450)
    renders["fx=450(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fx=700)
    renders["fx=700"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fx=450, fy=200)
    renders["fy=200"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fy=450)
    renders["fy=450(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fy=700)
    renders["fy=700"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fx=200, fy=200)
    renders["fx=fy=200"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fx=450, fy=450)
    renders["fx=fy=450(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, fx=700, fy=700)
    renders["fx=fy=700"] = scene_composer.get_render()
    scene_composer.visualize_grid_desc(renders, shape=(3 * height, 3 * width), n_cols=3, font_scale=2)
fisheye-lens-parameters_5_resourcesTutorial
2025-02-25 19:31:00,621 | skyrenderer.utils.time_measurement |  INFO: Setup time: 137 ms

2025-02-25 19:31:01,075 | skyrenderer.utils.time_measurement |  INFO: Context update time: 452 ms

2025-02-25 19:31:01,276 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:01,276 | skyrenderer.utils.time_measurement |  INFO: Render time: 201 ms

2025-02-25 19:31:01,327 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-25 19:31:01,772 | skyrenderer.utils.time_measurement |  INFO: Context update time: 444 ms

2025-02-25 19:31:01,969 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:01,969 | skyrenderer.utils.time_measurement |  INFO: Render time: 197 ms

2025-02-25 19:31:02,019 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-25 19:31:02,445 | skyrenderer.utils.time_measurement |  INFO: Context update time: 426 ms

2025-02-25 19:31:02,634 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:02,634 | skyrenderer.utils.time_measurement |  INFO: Render time: 188 ms

2025-02-25 19:31:02,685 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-25 19:31:03,111 | skyrenderer.utils.time_measurement |  INFO: Context update time: 425 ms

2025-02-25 19:31:04,168 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:04,169 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.06 seconds

2025-02-25 19:31:04,220 | skyrenderer.utils.time_measurement |  INFO: Setup time: 47 ms

2025-02-25 19:31:04,655 | skyrenderer.utils.time_measurement |  INFO: Context update time: 434 ms

2025-02-25 19:31:05,744 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:05,745 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.09 seconds

2025-02-25 19:31:05,797 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-25 19:31:06,227 | skyrenderer.utils.time_measurement |  INFO: Context update time: 429 ms

2025-02-25 19:31:06,413 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:06,414 | skyrenderer.utils.time_measurement |  INFO: Render time: 186 ms

2025-02-25 19:31:06,466 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-25 19:31:06,892 | skyrenderer.utils.time_measurement |  INFO: Context update time: 425 ms

2025-02-25 19:31:07,082 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:07,084 | skyrenderer.utils.time_measurement |  INFO: Render time: 191 ms

2025-02-25 19:31:07,136 | skyrenderer.utils.time_measurement |  INFO: Setup time: 47 ms

2025-02-25 19:31:07,558 | skyrenderer.utils.time_measurement |  INFO: Context update time: 421 ms

2025-02-25 19:31:08,584 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:08,585 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.03 seconds

2025-02-25 19:31:08,635 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-25 19:31:09,078 | skyrenderer.utils.time_measurement |  INFO: Context update time: 442 ms

2025-02-25 19:31:10,052 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:10,052 | skyrenderer.utils.time_measurement |  INFO: Render time: 973 ms

cx_relative, cy_relative - center of projection

    renders = {}
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.2)
    renders["cx=0.2"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.5)
    renders["cx=0.5(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.8)
    renders["cx=0.8"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.5, cy_relative=0.2)
    renders["cy=0.2"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cy_relative=0.5)
    renders["cy=0.5(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cy_relative=0.8)
    renders["cy=0.8"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.2, cy_relative=0.2)
    renders["cx=cy=0.2"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.5, cy_relative=0.5)
    renders["cx=cy=0.5(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.8, cy_relative=0.8)
    renders["cx=cy=0.8"] = scene_composer.get_render()
    scene_composer.visualize_grid_desc(renders, shape=(3 * height, 3 * width), n_cols=3, font_scale=2)
fisheye-lens-parameters_6_resourcesTutorial
2025-02-25 19:31:10,814 | skyrenderer.utils.time_measurement |  INFO: Setup time: 47 ms

2025-02-25 19:31:11,242 | skyrenderer.utils.time_measurement |  INFO: Context update time: 427 ms

2025-02-25 19:31:11,428 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:11,430 | skyrenderer.utils.time_measurement |  INFO: Render time: 187 ms

2025-02-25 19:31:11,483 | skyrenderer.utils.time_measurement |  INFO: Setup time: 47 ms

2025-02-25 19:31:11,910 | skyrenderer.utils.time_measurement |  INFO: Context update time: 426 ms

2025-02-25 19:31:12,102 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:12,103 | skyrenderer.utils.time_measurement |  INFO: Render time: 192 ms

2025-02-25 19:31:12,154 | skyrenderer.utils.time_measurement |  INFO: Setup time: 47 ms

2025-02-25 19:31:12,583 | skyrenderer.utils.time_measurement |  INFO: Context update time: 428 ms

2025-02-25 19:31:12,774 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:12,775 | skyrenderer.utils.time_measurement |  INFO: Render time: 191 ms

2025-02-25 19:31:12,826 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-25 19:31:13,259 | skyrenderer.utils.time_measurement |  INFO: Context update time: 433 ms

2025-02-25 19:31:14,296 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:14,297 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.04 seconds

2025-02-25 19:31:14,349 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-25 19:31:14,766 | skyrenderer.utils.time_measurement |  INFO: Context update time: 416 ms

2025-02-25 19:31:15,854 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:15,855 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.09 seconds

2025-02-25 19:31:15,905 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-25 19:31:16,324 | skyrenderer.utils.time_measurement |  INFO: Context update time: 418 ms

2025-02-25 19:31:17,390 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:17,391 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.07 seconds

2025-02-25 19:31:17,444 | skyrenderer.utils.time_measurement |  INFO: Setup time: 48 ms

2025-02-25 19:31:17,879 | skyrenderer.utils.time_measurement |  INFO: Context update time: 434 ms

2025-02-25 19:31:18,073 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:18,074 | skyrenderer.utils.time_measurement |  INFO: Render time: 194 ms

2025-02-25 19:31:18,125 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-25 19:31:18,581 | skyrenderer.utils.time_measurement |  INFO: Context update time: 456 ms

2025-02-25 19:31:18,771 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:18,772 | skyrenderer.utils.time_measurement |  INFO: Render time: 189 ms

2025-02-25 19:31:18,822 | skyrenderer.utils.time_measurement |  INFO: Setup time: 46 ms

2025-02-25 19:31:19,272 | skyrenderer.utils.time_measurement |  INFO: Context update time: 449 ms

2025-02-25 19:31:19,463 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:19,464 | skyrenderer.utils.time_measurement |  INFO: Render time: 191 ms

aperture_jitter and focus_distance

apperture_jitter is the parameter responsible for aperture related to focal length. The higher value, the smaller
DOF of a camera. If 0, infinite DOF is present. On the other hand, focus_distance is parameter holding a value of
plane distance in meters. It is used, when aperture jitter is nonzero.

    renders = {}
    scene_composer.setup_fisheye_lens(width, height, cx_relative=0.5, cy_relative=0.5)
    renders["aperture=0(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, aperture_jitter=1, focus_distance=1)
    renders["aperture=1,focus_dist=1(default)"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, aperture_jitter=1, focus_distance=5)
    renders["aperture=1,focus_dist=5"] = scene_composer.get_render()
    scene_composer.setup_fisheye_lens(width, height, aperture_jitter=1, focus_distance=7)
    renders["aperture=1,focus_dist=7"] = scene_composer.get_render()
    scene_composer.visualize_grid_desc(
        renders, shape=(height, 4 * width), n_cols=4, font_scale=1, text_position=(20, 20)
    )
fisheye-lens-parameters_7_resourcesTutorial
2025-02-25 19:31:20,246 | skyrenderer.utils.time_measurement |  INFO: Setup time: 45 ms

2025-02-25 19:31:20,696 | skyrenderer.utils.time_measurement |  INFO: Context update time: 449 ms

2025-02-25 19:31:21,732 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:21,733 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.04 seconds

2025-02-25 19:31:21,786 | skyrenderer.utils.time_measurement |  INFO: Setup time: 48 ms

2025-02-25 19:31:22,234 | skyrenderer.utils.time_measurement |  INFO: Context update time: 447 ms

2025-02-25 19:31:23,292 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:23,293 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.06 seconds

2025-02-25 19:31:23,343 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-25 19:31:23,771 | skyrenderer.utils.time_measurement |  INFO: Context update time: 428 ms

2025-02-25 19:31:24,813 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:24,814 | skyrenderer.utils.time_measurement |  INFO: Render time: 1.04 seconds

2025-02-25 19:31:24,864 | skyrenderer.utils.time_measurement |  INFO: Setup time: 44 ms

2025-02-25 19:31:25,284 | skyrenderer.utils.time_measurement |  INFO: Context update time: 420 ms

2025-02-25 19:31:25,473 | skyrenderer.utils.time_measurement |  INFO: Key points calculation time: 0 ms

2025-02-25 19:31:25,474 | skyrenderer.utils.time_measurement |  INFO: Render time: 189 ms

Summary

In this section you have learnt:

  • All parameters from fisheye lens mathematical theory can be set in FisheyeLens's create_parameter_provider
    method.
  • Fisheye lens model follows generic projection model described in INTRO_Lens.
  • Model-based parameters of fisheye lens are compatible with popular CV libraries like OpenCV, and renders can be
    undistorted using 3rd party library.
  • FisheyeLens in SkyRenderer is the parameter of CameraRenderStep (and, by extension, VisibleLightRenderStep).
2024-12-09_By Skyengine