ZEISS CinCraft LensCore

v1.0.5 - ac899ca8

Support & Feedback

We are committed to providing you with the best support and assistance for our product. If you encounter any issues or have feedback to share, please do not hesitate to contact us at cincraft@zeiss.com.

To help us address your concerns efficiently, we kindly ask you to prepare a minimal reproducing composition in Nuke. This will enable us to better understand and replicate the issue, ensuring a more effective resolution. Your input is invaluable to us, and we appreciate your cooperation in enhancing the quality of our product.

Introduction

Welcome to LensCore, a cutting-edge tool designed to bring the renowned precision and quality of ZEISS optics into the realm of virtual effects for the cinema industry. Leveraging our rich heritage in optics and lenses, LensCore enables postproduction professionals to accurately simulate the distinctive looks of physical lenses within their digital compositions. Embedded seamlessly into The Foundry’s Nuke software, this tool empowers users to enhance their projects with the authentic characteristics of ZEISS and 3rd party lenses.

A Brief History of ZEISS

ZEISS has been at the forefront of optical innovation since its founding in 1846. With over 130 years of expertise in crafting high-quality lenses, ZEISS has played a pivotal role in advancing optical technologies across various industries, including cinema, photography, and scientific research. Our commitment to precision and excellence has made ZEISS a trusted name worldwide, and LensCore continues this legacy by bringing our expertise in optics to the digital domain.

Features of LensCore

Pinhole rendering of the scene without any lens effects
Scene rendered with the ZEISS DigitalLens Pro

LensCore offers a range of features that simulate the nuanced effects of physical lenses, including:

Prerequisites

Operating Systems

Only 64bit builds

Nuke Compatibility

We offer builds for all actively supported versions of Nuke. Currently these are:

Nuke 14 Nuke 15 Nuke 16 Nuke 17
15.0 16.0 17.0
14.1 15.1 16.1
15.2

OpenCL

Currently the plugin can only run with an OpenCL runtime in version 2.0 (or higher). There are several implementations available for the different operating systems and different hardware configurations.

On linux the available OpenCL devices can be listed by clinfo which can be installed via the package manager.

❯ clinfo --list
Platform #0: Intel(R) CPU Runtime for OpenCL(TM) Applications
 `-- Device #0: 12th Gen Intel(R) Core(TM) i7-12700H
Platform #1: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) Iris(R) Xe Graphics
Platform #2: NVIDIA CUDA
 `-- Device #0: NVIDIA GeForce RTX 3050 Ti Laptop GPU

GPU devices

With a GPU-system, the OpenCL runtime typically is packed with the graphics driver. It is highly recommended to run this plugin on a graphics card or an other accelerated OpenCL device. At least 8GB of GPU video memory (VRAM) are recommended for the plugin.

CPU devices

There are OpenCL CPU runtimes available aswell, but be aware that they might be up to x100 slower than a recent GPU for processing (depends on the detailed system spec). Even the variance in OpenCL CPU implementations we observed during testing was huge. On the exact same hardware we measured a difference of up to x10 for different CPU runtime implementations for our lens rendering.

https://www.intel.com/content/www/us/en/developer/articles/tool/opencl-drivers.html#cpu-section

Intel OneAPI for Red Hat Enterprise Linux 9, Rocky Linux 9, Alma Linux 9

The plugin has been tested sucessfully with the Intel OneAPI OpenCL runtime implementation. To install, follow the instructions from Intel:

https://www.intel.com/content/www/us/en/docs/oneapi/installation-guide-linux/2023-0/yum-dnf-zypper.html#GUID-B5018FF2-B9F3-4ADC-9EB6-F99F6BFC7948

First of all import the repository and make it available for dnf or yum.

Installation instructions at the time of writing the documentation, visit the link for the latest instructions by intel.

Afterwards install the necessary components by:

dnf install -y --disablerepo=appstream intel-oneapi-runtime-opencl

to force installation from the intel repository.

pocl

pocl is a CPU implementation packaged by most linux distributions that is not compatible with our plugin.

Install Licenses

The plugin relies on RLM for licensing. You can use node-locked licenses by simply putting them into the plugin’s root dir, or floating licenses by connecting to Nuke’s license server.

In order to change where the license is read from, set the environment variable ZEISS_CINCRAFT_LENSCORE_LICENSES to, e.g. 4101@nuke.license.server. It is possible to give multiple locations as well, e.g. 4101@nuke.license.server:/home/user/licenses/ (use ; on Windows systems).

To install a license on your Nuke license server, place the ZEISS ISV file zeiss.set next to RLM’s foundry.set and upload your .lic files to its licenses path.

Attach ZEISS ISV to an Existing RLM Server

The generic RLM process for adding a new ISV to an already running server is:

  1. Add the new ISV server component (zeiss.set) where the RLM server can find it.
  2. Add ZEISS .lic file(s) to the license path used by the running RLM service.
  3. Trigger a license reread (rlmreread) so RLM picks up new files and starts new ISV servers.
  4. Verify that the ZEISS ISV appears in server status output.

RLM references:

Typical Paths

Windows (Foundry Licensing Utility setup):

Linux (typical RHEL-style setup, adjust to your service definition):

If your deployment differs, follow your rlm startup -c argument and service working directory.

No-Restart Command-Line Procedure

If you prefer no service restart, use rlmreread:

Windows:

cd "C:\Program Files\The Foundry\LicensingTools8.0\bin\RLM"
.\rlmutil.exe rlmreread
.\rlmutil.exe rlmstat -a

Linux:

cd /path/to/rlm/bin
./rlmutil rlmreread
./rlmutil rlmstat -a

Expected result: rlmstat -a shows the ZEISS ISV in the ISV server list with status Running/Yes.

Optional Full Restart Procedure

If reread is not sufficient in your environment:

  1. Restart the RLM service.
  2. Run rlmstat -a again and confirm ZEISS server availability.

Client Configuration Reminder

Clients should point to the RLM server as port@host, for example 4101@nuke.license.server.

For Lenscore, this maps to:

ZEISS_CINCRAFT_LENSCORE_LICENSES=4101@nuke.license.server

Quick Start

The following figure visualizes the minimal usage of the plugin in a Nuke composition.

Example usage of the LensCore plugin in a Nuke composition. The Camera and DeepImage nodes are mandatory, while Kernel and Inpaint Map are optional inputs.

By executing the following steps, a minimal rendered image can be produced using the plugin:

  1. Select a lens in the Lens tab of the node instance
  2. Ensure the Camera’s properties are correctly set. (See Interface → Camera)
  3. Ensure World to meter matches the given input texture.

Selecting the Supreme Prime 50 1.4T with a focus of 1.6 m, a fstop of 1.4f produces the following image.

Unprocessed, pinhole rendered RGBA+depth.Z image, converted with the Nuke-native DeepFromImage node.
Simulated Supreme Prime 50 1.4T lens with all of it’s characteristics.

Interface

Node Inputs

Camera (required)

The following elements are used directly from the given Nuke Camera.

All six camera properties used by the plugin
  1. World to Meters world_to_meters: The plugin has a own world_to_meter knob in it’s user interface. The value of the camera (if present) is used as a default value for the internal one. If different factors are specified, the software will always prioritize it’s internal value.
  2. Focal Length focal: The plugin has an own lens_focal_length knob in it’s user interface. It is only enabled if the selected lens is a Zoom-Lens, that allows to vary the focal length. By default the camera’s value will be linked to the internal one. The software will always read the internal value.
  3. Horizontal Aperture haperture: This value will be used as the simulated horizontal sensor size in [mm]. If the aspect ratio of haperture to vaperture doesn’t match the current format aspect ratio, a warning will be displayed in the plugin.
  4. Vertical Aperture vaperture: This value will be used as the simulated vertical sensor size in [mm]. If the aspect ratio of haperture to vaperture doesn’t match the current format aspect ratio, a warning will be displayed in the plugin.
  5. Focal Distance focal_point: The focal_point in the camera is defined in the scene unit and thus will be converted with the currently active World to Meters factor. It is used to define the focus area of the scene.
  6. Fstop fstop: Defines the current iris f/t-stop setting. The plugin has an additional internal option to select this knob’s unit. If the bounds of the currently selected lens are exceeded, a warning will appear.

DeepImage

The plugin only supports Nuke’s Deep-Workflow. Nevertheless it is not required to render the data with a real deep pass. Oftentimes studios used a layered approach with multiple cuts of a scene in RGBA+Depth. Those can be indivdually converted using the Nuke internal DeepFromImage node and combined using DeepMerge. As the plugin correctly resembles physics through the virtual scene, it is adviced to combine all the data to a single stream before the plugin. Often a single instance of the plugin may be the best option.

Nuke expects the depth.Z pass to be in the unit of 1/z. If the renderer outputs forward values (e.g. meter), an Expression node is required to correct the depth.Z to the correct representation. Using DeepFromImage afterwards will automatically convert the 1/z depth.Z values to a forward representation for deep.front and deep.back.

Expression node to mathematically correct the depth by calculating the reciprocal before feeding it to the DeepFromImage node.

Ensure to use the Expression node either after shuffling the data to depth.Z or do it on the correct color channel that currently holds the scenes depth.

Visualization of the nuke composition workflow to combine multiple depthmaps + RGBA images to a merged DeepImage that can be processed by the plugin

Kernel (optional)

Real world lenses do often induce certain characteristics by manufactoring imperfections that project in the bokeh of the lens. Well known examples are Onion-Rings or other patterns looking like a mixture of swiss cheese and moonscape.

It is extremely important to use this feature only for the shapes and the grain/crater charateristics inside the bokeh. Other characteristics like outline, iris shape, spherical aberration and cateye will be induced in a physically correct manner by the plugin anyhow!

A collection of lens kernel assets from https://www.compositingpro.com/free-lens-kernels

To ensure the kernel texture is not limiting the physical bounds, a best practice is to define no circular pattern, but simply have the full image defined like so:

No kernel active
Example kernel that only fills the internal grain and noise, as other characteristics are induced physically correct by the lens.
Kernel active

The kernel texture will be internally converted to grayscale, so no color is preserved throughout the bokeh. If this is necessary, feel free to reach out to us.

Inpaint Map (optional)

The inpaint map needs to have the exact same format/resolution as the image, otherwise it will fail. The input is only in use when Use Inpainting is enabled.

This optional input allows to partly scale the effect of the inpainting on a per-frame basis. Typical usecases are in combination with Nuke’s RotoPaint knob to either manually enable areas in a black texture, or use it the other way round by using a white texture by default and disabling the parts that shall not be inpainted. The best visualization for the inpainting is the Edge detection visualization, as no lens effects are visible in this mode. Only the red channel of the texture is used by the plugin. The red channel value will be multiplied by the Inpainting intensity field. 0 will fully disable inpainting and also the edge detection in this area. With edge detection turned off also Enhancement/Step 4 is disabled. Values > 0 will scale the inpainting effect accordingly.

Inpaint map texture with 0.0 for black and 1.0 for white
Resulting edge detection visualization
Manual overlay of the inpaint map to the edge visualization

Environment Variables

The plugin provides several environment variables to configure its behavior, particularly useful for guided customer support and internal testing procedures.

Here’s a list of key environment variables:

Environment variable Description
ZEISS_CINCRAFT_LENSCORE_DIR Allows to override the default lens directory. The currently active lens directory can be inspected by hovering on the open_lens_directory button
ZEISS_CINCRAFT_LENSCORE_TELEMETRY_DIR Allows to specify the default telemetry destination directory used by the plugin.

The plugin uses https://www.boost.org/ under the hood - specifically the compute module. It offers environment variables to customize the default OpenCL device used by the system. Table from the source boost/compute/system.hpp.

Environment variable Description
BOOST_COMPUTE_DEFAULT_DEVICE name of the compute device (e.g. “GTX TITAN”)
BOOST_COMPUTE_DEFAULT_DEVICE_TYPE type of the compute device (e.g. “GPU” or “CPU”)
BOOST_COMPUTE_DEFAULT_PLATFORM name of the platform (e.g. “NVIDIA CUDA”)
BOOST_COMPUTE_DEFAULT_VENDOR name of the device vendor (e.g. “NVIDIA”)
BOOST_COMPUTE_DEFAULT_ENFORCE If this is set to “1”, then throw a no_device_found() exception if any of the above environment variables is set, but a matching device was not found.

UI Description

Tab: Main

Overview of the Main tab

Group: Input_properties

Contains 2 elements: World to Meter, Extra Channels

Everything in this group tells the plugin, how to see the world. This currently are the world scale, as well as what additional channels to “see” and work on.

Knobs: World to Meter

world_to_meter, Slider, default: 0.000000

Factor to multiply the world unit with to get meters. Examples: 0.1 for decimeter, 0.01 for centimeter, 0.001 for millimeter

The value to enter is the multiplier needed to transform the image data into meters.

Example: The input data is in cm scale. To transform a centimeter scale to meters, we need to divide the actual number by 100, resulting in a “world to meter” factor of 0.01.

If the value is wrong, render time will increase drastically and the whole image will be in defocus.

Knobs: Extra Channels

input_channels, channelset

Allows to add up to 3 additional channels to the processing. The channels will be treated identical to the RGB colors except for the vignette. RGBA + deep.front will be always used for the effect calculation.

The extra channels option can be used to create matching masks for elements in the defocus chain.

Given the following example of the second chair in the example scene.

The color layer
The alpha mask

The alpha layer shall be used to create a mask to later recolor and finetune some elements of the scene. In this case the second chair. The mask needs to represent occlusion, distortion and lens breathing. All these transformations can be done to the mask alongside the actual lens effect application.

Shuffling the alpha mask to a custom channel
The custom channels are automatically padded with zeros for other depths in the deep comp

With this additional info of the second chair’s alpha channel, we can enable thee extra_channels feature in the plugin:

Extra channel selection in the plugin, up to 3 additional channels can be used

The resulting mask can be used by e.g. ColorCorrect node to manipulate the chair after the lens effects have been applied.

The resulting mask with the correct transformations, occlusion and defocus
Color graded output image by using the resulting mask

Group: Render_quality

Contains 5 elements: Quality Level (GUI), x, Quality Level (Farm), x, Disable rendering optimizations

The lens effect simulation is a complex algorithm. In order to find the best trade-off between quality and speed, the quality level can be tweaked here.

Knobs: Quality Level (GUI)

quality_level, Dropdown, default: Medium

Options: [PreVis, Low, Medium, High, Ultra]

The quality level required. Higher values mean higher computation times in the gui.

This option only applies for Nuke instances with a graphical interface. It will be ignored for headless instances, e.g. on a rendering farm.

In order to get a quick preview or a spotless image, select one of the quality levels PreVis (pre-visualization), Low, Medium, High, Ultra. High and Ultra come with a multiplicator.

Knobs: x

quality_scale, Slider, min: 0.2, max: 5, default: 1.000000

Keeps the settings of the selected quality level, but scales identified elements by the applied factor.

This option only applies for Nuke instances with a graphical interface. It will be ignored for headless instances, e.g. on a rendering farm.

Only applying to quality levels “High” and “Ultra”, the quality scale increases the output quality over the given bounds.

Knobs: Quality Level (Farm)

quality_level_nogui, Dropdown, default: High

Options: [PreVis, Low, Medium, High, Ultra]

The quality level required. Higher values mean higher computation times on the farm.

This option only applies for Nuke instances without a graphical interface. It will be ignored for local instances with a visible UI.

In order to get a quick preview or a spotless image, select one of the quality levels PreVis (pre-visualization), Low, Medium, High, Ultra. High and Ultra come with a multiplicator.

Knobs: x

quality_scale_nogui, Slider, min: 0.2, max: 5, default: 1.000000

Keeps the settings of the selected quality level, but scales identified elements by the applied factor.

This option only applies for Nuke instances without a graphical interface. It will be ignored for local instances with a visible UI.

Knobs: Disable rendering optimizations

disable_adaptive_ray_count, Checkbox, default: 0

Disables ALL performance optimizations. Quality levels still have an effect. WARNING: will be slow! Only use if artifacts cannot be removed otherwise.

Disables all optimizations. This will get rid of all artifacts that are due to optimizations to the lens effect core algorithm.

Important: Only check this box if visual artifacts are present, that cannot be fixed with other means e.g. inpainting. If the malfunction is not a rendering relict of the internal optimizations, uncheck this box again for better performance.

Group: Bounding_box

Contains 2 elements: Show overscan coordinates, Increase Bounding Box by pixels

Knobs: Show overscan coordinates

overscan_visualization, Checkbox, default: 0

The scanned area will be displayed in absolute scene coordinates. Inactive in no-gui mode.

Knobs: Increase Bounding Box by pixels

bounding_box_expansion, Number, default: 0

Adds selected amount of pixels as padding. Image dimensions will grow by 2*value in both width and height.

Big scene with a format of 3000x2000 px, but a reduced bbox to match only the hightlight.
By default the input bbox is preserved, in this case the bbox cannot hold the content of the defocused image and thus will only render the resulting gray color inside the bbox. The viewer displays the edge pixels outside thus the impression of an all gray picture.
By actually increasing the bbox by 120 px, the output plane is big enough to contain the defocused image and the bokeh is displayed correctly.

Group: Feedback

Contains 2 elements: Local telemetry log folder:, Report Issue

Knobs: Local telemetry log folder:

telemetry_folder, FilePicker

If nothing is selected the logging directory can be selected with the environment variable ‘ZEISS_CINCRAFT_LENSCORE_TELEMETRY_DIR’. By default the lens directory will be used as a fallback. Current fallback destination: ZEISS_CINCRAFT_LENSCORE_DIR

Our software collects telemetry data to aid support and further development. Telemetry is entirely opt-in; all collected data is stored locally on the user’s device and requires manual transmission by the customer, ensuring no data exchange occurs by default. This data is strictly unpersonalized, meaning it does not include any personal information or identifiers. Importantly, no sensitive information, such as image content, is collected during the process. We only gather metadata, such as usage patterns, system and feature configurations, to help us understand how our software is being used and identify areas for improvement. Additionally, the data is formatted in a human-readable manner, allowing customers or their IT departments to review and oversee the information before it is sent to ZEISS.

Knobs: Report Issue

provide_feedback_btn, Button

Will open a preconfigured mail in the OS’s default mail program. It will automatically fill the draft with important information. Nothing is sent automatically, user has to always send out the mail manually. All information is copied to the clipboard automatically aswell.

Use this button to get in contact with the ZEISS service. Please follow the instructions in the mail draft to provide as much information about either your feedback or issues. Your insights will help us enhance the plugins functionality and better meet your needs.

Knobs: Clear plugin cache

clear_cache, Button

This plugin utilizes an internal cache to avoid reloading the input image with every call. However, there are instances when Nuke updates the input image without refreshing the previous hash. As a result, the plugin may continue to render using the outdated input image. If you notice that changes in the scene are not reflected in the plugin’s output, click this button to clear all internal caching within the plugin.

Knobs: Open Documentation

open_documentation, Button

Opens local copy of the documentation with the default html tool, probably the browser. The version of the documentation is guaranteed to match the version of the currently loaded plugin.

The button is only visible if a file called LensCore.<VERSION_HASH>.html or documentation.html with the current version hash exists in the nuke plugin path.

Tab: Lens

Overview of the Lens tab

Here, all the configuration about the simulated lens look happens. In the simplest case, just choose a lens, and the effect is applied.

Currently, there are three kinds of lenses provided:

Depending on the selected lens, you will see more or less content on the lens tab.

Knobs: Lenses

lens_dropdown, Dropdown, default: Other (enter path)

Options: [Please select a lens..., Other (enter path)]

Knobs: Lens path

lens_file, FilePicker

Knobs: Presets

preset_dropdown, Dropdown, default: New preset...

Options: [New preset...]

Select a preset here. Presets are stored in ZEISS_CINCRAFT_LENSCORE_DIR as zlp files.

Presets are a simple way to store and share user-created artistic lenses. They are a file representation of how the Artificial Lens has been set up.

Presets are currently stored in the lens dir, but as “.zlp” files. Selecting a preset in the drop down box will automatically apply the preset. Applying a preset will replace all look-relevant values in the Lens tab. Saving a preset will overwrite the file, replacing all stored values.

Knobs: Reset

preset_reset, Button

Revert all changes to the currently selected preset defaults

Knobs:

open_lens_directory_in_explorer, Button

Open the current lens folder specified in environment variable ZEISS_CINCRAFT_LENSCORE_DIR: ZEISS_CINCRAFT_LENSCORE_DIR

The Nuke tooltip will reveal the exact evaluated path of the lens directory folder. If the folder doesn’t exist, it will be created automatically.

Knobs: /

preset_collapse, Button
preset_expand, Button

Hide extra options

More options for presets

Knobs: Save

preset_save, Button

Preset is saved to its file. If it doesn’t yet exist, behaves like save as.

Knobs: Save as

preset_save_as_init, Button

Opens the Save As configuration

Knobs: Scan for presets

preset_refresh_dropdown, Button

Scan for presets in ZEISS_CINCRAFT_LENSCORE_DIR and refresh dropdown

Knobs: Preset name:

preset_name, Text

Will be displayed in dropdown menu.

Currently, names must be unique.

Knobs: Preset description:

preset_description, Text

More detailed description. Not yet displayed except here.

This value is only stored and read from and to his textbox. It currently is not used otherwise.

Knobs: Preset artist:

preset_artist, Text

Preset artist/vendor. Not yet displayed except here.

This value is only stored and read from and to his textbox. It currently is not used otherwise.

Knobs: Filename:

preset_filename, FilePicker

Where to store the preset. Currently only will be found in lens dir: ZEISS_CINCRAFT_LENSCORE_DIR

Currently, all presets MUST be placed in the lens dir for the Plugin to find them.

Knobs: Abort

preset_save_as_cancel, Button

Close the save dialog, do not save.

Knobs: Save

preset_save_as_execute, Button

Executes save as with the given settings. Warns before overwriting a file.

Knobs: Focal length of the lens

lens_focal_length, Slider, min: 1, max: 100, default: 30

The focal length the currently selected lens shall use.

Knobs: Lens design sensor width

lens_design_sensor_width, Slider, min: 1, max: 100, default: 36

The design sensor width of the currently replicated physical lens.

Knobs: Lens design sensor height

lens_design_sensor_height, Slider, min: 1, max: 100, default: 24

The design sensor height of the currently replicated physical lens.

Knobs: Original lens characteristics

original_lens_characteristics, Checkbox, default: 1

Disable to tweak the different lens effects

Knobs: Match scene with lens focal length

match_scene_with_lens_focal_length, Checkbox, default: 1

Disable to enable zoom behavior for lenses

This value is intended to be used when no exactly matching lens is available.

Given a 45 mm rendered scene and a selected 55 mm lens, if the parameter is set, the scene is to rescaled to match the FOV of the lens and no zooming into the scene happens. Unsetting the parameter would cause the 55 mm lens to naturally zoom into the scene.

Group: Bokeh Preview

Contains 3 elements: Preview the bokeh, Use camera data for preview, Preview divider location

Knobs: Preview the bokeh

enable_bokeh_preview, Checkbox, default: 0

Creates a perforation board to preview the currently configured lens effects. Uses generic focus point and wall distances.

This checkbox enables the bokeh preview.

In order to see the behavior of the effect, two black walls with backlights are created: the left wall is placed in front of the focus, the right wall is placed behind the focus. The two walls will be separated by a white line. All together, this will look like a perforated plate over two depths:

Screenshot from bokeh preview in Nuke

The effect is applied to this just like any other scene. All lens effect settings are in place. It is not possible to render this preview in headless mode.

Knobs: Use camera data for preview

bokeh_preview_use_view_data, Checkbox, default: 0

Simulates the behavior with current Camera settings. Can have an effect on maximal possible bokeh sizes.

While only enabling the bokeh preview will create perforated plates specifically to get a bokeh as large as possible, it’s using a fixed focus and f-stop. Using camera data will allow to preview the actual bokeh as it will be seen in the scene.

Knobs: Preview divider location

bokeh_preview_divider_location, Slider, min: 0, max: 1, default: 0.600000

Moves the border between front and back defocus

This slider allows to move the white line (and with it the border between front and back defocus) across the image.

Group: Aperture

Contains 6 elements: Aperture Stop, Number of aperture blades, Aperture blade curvature, Aperture rotation offset, Simulated minimum fstop, Fstop rotation scaling

Knobs: Aperture Stop

stop_type, Dropdown, default: Fstop

Options: [Fstop, Tstop]

Fstop is typically used for CGI renderings. No transmittance losses are taken into account for. Tstop is the default unit for engravings on real lenses.

Allows the use of the Camera node’s f-stop as a t-stop (as it is typically used for real world footage). This option is not available for ZEISS DigitalLens lenses, i.e. ZEISS DigitalLens Lite and ZEISS DigitalLens Pro.

Knobs: Number of aperture blades

num_aperture_blades, Number, default: 0

Sets the number of aperture blades freely.

Use 0 to disable and use no aperture blades. Invalid values will round to the next valid one, e.g. inputs of 1 or 2 will be treated as 3 blades.

Artifcial Lens with num_aperture_blades: 4
Artifcial Lens with num_aperture_blades: 5
Artifcial Lens with num_aperture_blades: 9
Knobs: Aperture blade curvature

aperture_blade_curvature, Slider, default: 0.000000

Sets curvature on the aperture blades. 0 for no curvature, 1 for full circle curvature.

This is treated as an interpolation factor of how close the blades are to a circle. The physical pendant to this feature is the curvature in the aperture fins. With a curvature of 1, the blade effect is fully deactivated independent of the setting in Number of aperture blades. The example images below are rendered with 3 blades.

aperture_blade_curvature of 0.0
aperture_blade_curvature of 0.5
aperture_blade_curvature of 1.0
Knobs: Aperture rotation offset

aperture_rotation_offset, Slider, min: 0, max: 360, default: 0.000000

The rotation offset of the aperture in deg (positive numbers rotate clockwise).

Knobs: Simulated minimum fstop

override_min_fstop, Slider, default: 0.000000

Simulates the minimum fStop. Will impact the interaction between aperture blades and cateye simulation. Should match the physical lens pendant that is simulated. Minimum fstop: 1.0 - set to 0 to use lens default.

This feature is only available for the ZEISS DigitalLens Pro. This allows to define the fStop that corresponds to full the designed cateye shape by using Cateye Bokeh Simulation. The same minimum fStop is used as a starting point for the blade rotation simulation. With increasing fStop, the cateye is less and less visible, as the resulting bokeh is limited by the aperture blades.

The following renders have been made all with the exact same lens settings with the ZEISS DigitalLens Pro with 5 aperture blades and cateye simulation active. The Camera fStop is set to 2.0 for all tests - only the Simulated minimum fStop option is modified in between renderings.

Simulated minimum fStop of 1.0
Simulated minimum fStop of 1.4
Simulated minimum fStop of 2.0
Knobs: Fstop rotation scaling

aperture_fstop_rotation_scaling, Slider, default: 1.000000

The aperture shape rotation is directly related to the selected f-stop. Changing the f-stop results in a dynamic adjustment of the aperture blades’ rotation. This knob controls the scaling of that f-stop-dependent rotation.

A value of 0 disables the fStop dependency on the rotation offset. Negative values will reverse the direction from clockwise to counter-clockwise.

The rotation effect dependent on the f-stop value (fstop ranging from [1; 5]) using aperture_fstop_rotation_scaling with a value of 1.

Group: Focus & Defocus

Contains 9 elements: Anamorphic aspect ratio, Front defocus intensity, Back defocus intensity, Lens transmission, Focus sharpness, Color-code the focus in the image, Field curvature ^2, Field curvature ^4, Plot curvature

Knobs: Anamorphic aspect ratio

pseudoanamorphic_aspect, Slider, min: 0.25, max: 4, default: 1.000000

Fakes anamorphic behavior for any lens with the given aspect. Values > 1.0: keep y, scale x by factor. Values < 1.0: keep x, scale y by 1/factor. Only works if both ‘Lens design sensor size’ and ‘Match sensor to image aspect ratio’ are disabled! Max scale factor is 4.000000

Anamorphic lenses have an aspect ratio and thus lead to ellipsoid bokeh throughout the image. The anamorphic behavior does not depend on the position in the image. This feature can be used to stretch the behavior of any given lens to aproximate the look of an anamorphic even if the lens itself isnt.

Artificial Lens with an anamorphic aspect ratio of 0.5
Artificial Lens with an anamorphic aspect ratio of 2.0
Knobs: Front defocus intensity / Back defocus intensity

front_defocus_intensity, Slider, min: 0, max: 5, default: 1.000000
back_defocus_intensity, Slider, min: 0, max: 10, default: 1.000000

Front defocus intensity: Scale the effect of the defocus before the focal plane

Back defocus intensity: Scale the effect of the defocus behind the focal plane

These modifiers change the effect of the defocus to either emphatise or weaken blur elements in front or behind the focal plane. The effect works by scaling the physical world in the scene.

For front_defocus_intensity elements in the back of the focal plane stay untouched. The closer to the camera, the bigger the bokeh. Front defocus bokeh can scale into infinity so ensure to make little increments when using with this feature.

front_defocus_scaling of 0.5
front_defocus_scaling of 1 (default)
front_defocus_scaling of 1.5

For back_defocus_intensity elements in front of the focal plane stay untouched. A lens always has a maximum bokeh size in the back defocus. This means higher values do not always lead to increasing bokeh. The physical limit of the lens cannot be overwritten with this feature.

back_defocus_scaling of 0.5
back_defocus_scaling of 1 (default)
back_defocus_scaling of 2
Knobs: Focus sharpness

enforce_focus_threshold, Slider, min: 0, max: 15, default: 0.000000

Enforce the focus layer to be sharp for a 1:1 pixel match (0 to disable)

Real, physical lenses do not have a 1:1 pixel match, even if the inspected area is in focus and in the center of the image. This is caused by non-perfect materials, build tolerances, wavelength specific behavior through the glass and other effects. To forcibly overrule those remaining (physical) aberrations this value is introduced. The higher the value, the bigger the forced sharpness around the focal plane.

Knobs: Color-code the focus in the image

focus_visualization, Checkbox, default: 0

The focus area will colored blue, elements in front green and elements in the back red. Inactive in no-gui mode.

As the effect is for previewing and editorial changes only, the quality-level will be automatically lowered significantly when using this overlay.

Focus visualization in action
Knobs: Field curvature ^2

field_curvature_2, Slider, default: 0.000000

Weight for the ^2 term for field curvature simulation. The weight is the depth offset in [mm].

Adds a quadratic term offset to the focus behavior of the lens. Will vary with image position. No offset in center (0), full offset in diagonal (1). All offsets will be accumulated.

offset=img_pos2*field_curvature_2m\text{offset} = \text{img\_pos}^2 * \text{field\_curvature\_2} \si{\m}

Group: Cateye Bokeh Simulation

Contains 49 elements: Enable bezier cateye bokeh simulation, Transition start, Transition end, Radius fall-off, Aspect ratio, Roundness x, Roundness y, Simulate cateye as circles, MOD focus pos, INF focus pos, INF pupil scale, INF pupil scale start focus [m], INF pupil scale end focus [m], Field interpolation pos fp0, Field interpolation pos fp1, Field interpolation pos fp2, Field interpolation pos fp3, mod Circle 1 radius field pos 0, mod Circle 1 offset field pos 0, mod Circle 2 radius field pos 0, mod Circle 2 offset field pos 0, inf Circle 1 radius field pos 0, inf Circle 1 offset field pos 0, inf Circle 2 radius field pos 0, inf Circle 2 offset field pos 0, mod Circle 1 radius field pos 1, mod Circle 1 offset field pos 1, mod Circle 2 radius field pos 1, mod Circle 2 offset field pos 1, inf Circle 1 radius field pos 1, inf Circle 1 offset field pos 1, inf Circle 2 radius field pos 1, inf Circle 2 offset field pos 1, mod Circle 1 radius field pos 2, mod Circle 1 offset field pos 2, mod Circle 2 radius field pos 2, mod Circle 2 offset field pos 2, inf Circle 1 radius field pos 2, inf Circle 1 offset field pos 2, inf Circle 2 radius field pos 2, inf Circle 2 offset field pos 2, mod Circle 1 radius field pos 3, mod Circle 1 offset field pos 3, mod Circle 2 radius field pos 3, mod Circle 2 offset field pos 3, inf Circle 1 radius field pos 3, inf Circle 1 offset field pos 3, inf Circle 2 radius field pos 3, inf Circle 2 offset field pos 3

The cateye simulation creates a physically correct (but not necessarily physically sane) cateye effect.

No cateye simulation applied
Default cateye simulation applied
Artistic,physically insane cateye configuration applied with physically correct effect simulation

Several parameters are used to tweak the bokeh, aiding users in configuring their desired lens look.

Knobs: Enable bezier cateye bokeh simulation

cateye_simulation, Checkbox, default: 0

Knobs: Transition start / Transition end

cateye_simulation_morph_start, Slider, min: 0, max: 1, default: 0.000000
cateye_simulation_morph_end, Slider, min: 0, max: 5, default: 1.000000

Transition start: Defines the start of the cateye simulation in relative image coordinates. 0 is the center, 1 is the corner.

Transition end: Defines the end of the cateye simulation in relative image coordinates. 0 is the center, 1 is the corner.

“Transition” addresses the shape change from a perfectly round bokeh shapes to the characteristic cateye-shape:

Example transition from center to right edge

The bokeh shape morphs in a linear fashion from the center of the image to the edges, increasing the cateye effect to its maximum on the outermost parts of the image. The controls for morphing start/end allow to change the begin and end of the area in which this transition takes place:

Transition start: 0.5, Transition end: 1 Transition starts only at 50% of the image. Every bokeh inside the circle has an identical shape, the linear transformation starts only outside of this circle.
Transition start: 0, Transition end: 1 Default transition over the whole image area, from 0 to 1. The linear transformation covers the whole area between the center and outside pixel.
Transition start: 0, Transition end: 0.5 Transition already ends after 50%. Every bokeh outside the circle already has the full cateye shape.

The values are a percentage of the image circle’s radius. Transition start can only be inside the image, i.e. a value between 0 % and 100 %. Transition end can also lie outside of the image, allowing to create the visual equivalent of a cropped shot.

Knobs: Radius fall-off

cateye_simulation_radius_falloff, Slider, min: 0, max: 1, default: 1.000000

Radius fall-off of the cateye. Values <1 shrink the radius linearly from morph_start to morph_end.

The slider is a percentage to which size the bokeh radius will fall off.

No radius falloff (falloff to 100%)
Radius falloff to 50%
Knobs: Aspect ratio

cateye_simulation_aspect_ratio, Slider, min: 0.25, max: 4, default: 1.250000

Aspect ratio of the cateye. Values >1 shrink the width, values <1 shrink the height of the cateye.

The aspect ratio controls the “cateye-ness” of the bokeh. This will always shrink the bokeh, never extend its size.

Aspect ratio of 0.75
Neutral aspect ratio of 1
High aspect ratio of 2
Knobs: Roundness x / Roundness y

cateye_simulation_scaling_x, Slider, min: 0, max: 1, default: 0.500000
cateye_simulation_scaling_y, Slider, min: 0, max: 1, default: 0.150000

Roundness x: Specify the roundness of the cateye in x direction (0.5 gives a perfect circle, 1 to round more, 0 to have a sharp corner)

Roundness y: Specify the roundness of the cateye in y direction (0.5 gives a perfect circle, 1 to round more, 0 to have a sharp corner)

The roundness controls allow you to adjust the horizontal and vertical scaling of the cateye simulation, creating different elliptical shapes for the bokeh highlights.

These controls work together to create the desired cateye effect:

  • Roundness X: Controls horizontal scaling
  • Roundness Y: Controls vertical scaling

Group: Bokeh Outline

Contains 2 elements: Gain, Thickness

Knobs: Gain

bokeh_simulation_outline_gain, Slider, min: 0, max: 20, default: 0.500000

Controls the brightness of the bokeh outline (0 to disable)

Gain refers to the intensity or brightness of the bokeh outline. By increasing the gain, the edges of the bokeh become more pronounced and vivid, making them stand out more against the background. Conversely, reducing the gain results in softer, less distinct edges, which can be useful for creating a more subtle and natural look.

Knobs: Thickness

bokeh_simulation_outline_thickness, Slider, min: 0, max: 50, default: 3.000000

Controls the thickness of the bokeh outline in px (0 will give no outline).

Thickness pertains to the width of the bokeh outline. Adjusting the thickness allows users to control how wide or narrow the edges of the bokeh appear. A thicker outline can give the bokeh a more defined and bold appearance, while a thinner outline can create a delicate and refined effect.

Group: Spherical aberration

Contains 4 elements: Mode, Intensity, Fall-off, Spherical aberration ray amplitude [mm].

Visualization of spherical aberration with constant focus to the midground chair in 3.0 m but moving highlights in the scene. The x axis shows the hightlight depth, the y axis the spherical aberration mode. Note the behavior change before and after focus for each row.
Knobs: Mode

spherical_aberration_mode, Dropdown, default: Off

Options: [Off, Positive, Negative]

The implemented spherical aberration definition follows the wikipedia entry definition of ‘Spherical aberration’. No spherical aberration (Off) will cause no density variations in the rendered bokehs. With either a positive or negative spherical aberration, the behavior will change depending on the location of the bokeh to the fokus. With negative spherical aberration, elements in front of the focus will be under-corrected, elements behind the focus will be over-correced. For positive spherical aberration the behavior flips.

Knobs: Intensity

spherical_aberration_weight_intensity, Slider, min: 0, max: 1, default: 1.000000

The intensity defines how much effect the spherical aberration has. It describes the delta between the brightest and darkest bokeh element. The density in the bokeh always spans from [1-intensity; 1].

Knobs: Fall-off

spherical_aberration_pow_exponent, Slider, min: 0, max: 10, default: 1.000000

This is an exponential factor that applies as (distance_from_bokeh_center)^fall_off.

Group: Vignetting

Contains 3 elements: Scale, Fall-off, Offset

The “vignette” effect in optics refers to a gradual darkening of the image towards the corners compared to the center. This phenomenon is primarily caused by the lens design and the way light is transmitted through the lens elements. As light rays enter the lens at oblique angles, particularly at the edges, they may be less effectively focused onto the image sensor or film, leading to reduced brightness in those areas. Vignetting can also be influenced by factors such as lens aperture, focal length, and the use of filters or lens hoods.

Both of the following parameters should be used in combination to reach the desired outcome.

Knobs: Scale

vignette_intensity_scale, Slider, min: 0, max: 10, default: 1.000000

Scale the intensity of the vignette linear.

Default behavior of the vignette with a scale of 1.0
Pushed vignette with linear scaling of 1.5
Knobs: Fall-off

vignette_intensity_falloff, Slider, min: 0, max: 10, default: 1.000000

Scale the intensity of the vignette by intensity^value.

Default behavior of the vignette with a falloff of 1.0
Pushed vignette with exponential scaling falloff 3.0
Knobs: Offset

cos4_vignette_weight, Slider, min: 0, max: 10, default: 0.000000

Add additional vignetting on top of the cateye-caused vignette via a generic cos^4 function. Will be modified by scale and intensity falloff.

Group: Distortion & Breathing

Contains 6 elements: Distortion intensity, Distortion coefficients, Distortion clamp, Lens breathing, Lens breathing, Plot lens breathing

Knobs: Distortion intensity

distortion_intensity, Slider, default: 1.000000

Scale the native lens distortion. Negative values will flip the behavior.

Distortion Intensity is a parameter that allows to control the degree of optical distortion introduced by the lens. Optical distortion can manifest as barrel or pincushion effects, where straight lines appear curved or bent. By adjusting the distortion intensity, the unique characteristics of various lens types can be simulated.

Knobs: Lens breathing

lens_breathing, Checkbox, default: 1

Enables/Disables lens breathing effect with varying focus.

This checkbox is only available for predefined lenses. It is used to enable/disable the breathing effect, i.e. field of view changes due to a slight variance in the physical focal length of a lens over various focus points.

Group: Chromatic Aberration

Contains 13 elements: Enable Chromatic Aberration, Draw color combinations, Front chroma color, Back chroma color, Chromatic aberration intensity, Lateral chromatic aberration intensity, Longitudinal chromatic aberration intensity, Transition minimum for inf CA, Transition maximum for inf CA, Chromatic aberration at inf back wl, Chromatic aberration at inf front wl, Back WL lateral CA scaling, Back WL longitudinal CA scaling

Chromatic aberration is an optical phenomenon where a lens fails to focus all colors to the same convergence point, resulting in color fringing around high-contrast edges in an image. This occurs because different wavelengths of light are refracted by varying degrees as they pass through the lens, causing them to focus at different points along the optical axis.

Knobs: Enable Chromatic Aberration

use_multiple_wavelengths, Checkbox, default: 1

Knobs: Back chroma color / Front chroma color

chroma_bg_color, color, default: 0, 0, 1
chroma_fg_color, color, default: 1, 0, 0

Only available for the ZEISS DigitalLens Pro.

The following visualization usese the Bokeh preview feature in combination with the Longitudinal chromatic aberration intensity.

Resulting chromatic aberration with Front chroma color of r: 1.0, g: 0.0, b: 0.0 and Back chroma color of r: 0.0, g: 0.0, b: 1.0
Resulting chromatic aberration with Front chroma color of r: 0.3, g: 1.0, b: 0.0 and Back chroma color of r: 0.8, g: 0.2, b: 0.85
Knobs: Chromatic aberration intensity

chromatic_aberration_intensity, Slider, min: 0, max: 10, default: 1.000000

Scales the lenses natural chromatic aberration effect.

Only available for physical lenses.

Chromatic aberrations of the Supreme Prime 50 lens turned off.
Chromatic aberrations of the Supreme Prime 50 scaled by a factor of 3
Knobs: Lateral chromatic aberration intensity

lateral_chromatic_aberration_intensity, Slider, default: 1.000000

Scales the resulting image and distortion depending on wavelength. Color fringes will appear towards the image edges.

Only available for the ZEISS DigitalLens Pro.

No chromatic aberration effect
Lateral chromatic aberration scale factor of 10
Knobs: Longitudinal chromatic aberration intensity

longitudinal_chromatic_aberration_intensity, Slider, default: 1.000000

Scales the bokeh size depending on wavelength. Will result in color-fringes rotational symmetric to bokeh.

Only available for the ZEISS DigitalLens Pro.

No chromatic aberration effect
Longitudinal chromatic aberration scale factor of 10
Knobs: Chromatic aberration at inf back wl

chromatic_aberration_inf_fstop_delta_back_wl, Slider, default: 1.000000

Modifies the focal length of the back chroma wavelength towards infinity to simulate chromatic aberration at focus infinity.

Knobs: Chromatic aberration at inf front wl

chromatic_aberration_inf_fstop_delta_front_wl, Slider, default: 0.000000

Modifies the focal length of the front chroma wavelength towards infinity to simulate chromatic aberration at focus infinity.

Group: Filter Kernel

Contains 1 elements: Kernel scale

Knobs: Kernel scale

bokeh_kernel_skew, Slider, default: 0.000000

Movement of Kernel within the bokeh. The scaled content of the kernel will be shifted by the position in the image. Kernel windows will move!

For more detailed information about the kernel usage, refer to the Kernel section.

Shifts the kernel depending on the output sensor pixel by a certain amount. Negative values flip the shift direction.

Tab: Enhancement

Overview of the Enhancement tab

The plugin typically operates with pointcloud data rendered from a single point of view using a pinhole camera. When executing the sensor simulation, which is necessary for simulating lens effects, with an actual physical dimension of the sensor (e.g. 36×24 mm), the angles slightly change, and thus areas of the image become visible that were not visible to the pinhole renderer.

As only front faces were rendered, this information is missing in the pointcloud and therefore shadowing artifacts occur. These are referred to as ‘depth shadows’. The inpainting algorithm resolves these by estimating the color and depth of the missing areas.

The following figures visualize the origin of depth shadows by missing information in the scene without the lens simulation, but with shifting the camera position slightly.

No depth shadows visible, as FOV and position match the rendering setup.
Camera slightly shifted to the right, slight depth shadows occur on the background chair.
Camera shifted to the left, depth shadows occur on the background chair.

Depth shadows can be visualized by either having a bright background behind a foreground subject or depending on the actual layers or contents of the scene it might be sufficient to just look at the alpha channel.

Visualizing the alpha channel with extreme gamma/contrast setting and no inpainting active.
Alpha channel after applying the inpainting algorithm

Not all inpainting problems can be visualized by looking at the alpha channel. The figure below shows the foreground chair arm rest. There are no issues in the alpha mask, but clearly the background carpet is shining through the chair.

Depth shadows occur because the foreground chair has not enough data at the armrest and instead of hitting either the side of the armrest or the chair’s seat, the background layer is hit and thus visible.
The shine-through behavior of the background layer is fixed.
Knobs: Use Inpainting

inpainting_enabled, Checkbox, default: 0

Enables or disables the whole inpainting algorithm. Further UI elements will be hidden if disabled.

Group: Step 1: Choose edge detection algorithm and threshold based on the observed ‘depth shadows’

Contains 3 elements: visualize, Mode, Threshold

The algorithm works based on edges, and will only inpaint in regions around detected edges. Use the visualize feature to ensure the depth shadows observed are detected as edges.

Depth shadows visible at the armrest with no inpainting applied
Edge detection is to insensitive. Not all depth shadows are detected as an edge
Correct edge detection setting: Detected edges match the observed depth shadows
Knobs: visualize

visualize_edges, Checkbox, default: 0

Visualizes the edges in bright pink and overlays them onto the input point cloud. Lens simulation is not yet executed at this step.

Knobs: Mode

edge_detection_mode, Dropdown, default: Relative

Options: [Relative, Absolute]

Detection of edges according to the selected mode. Either relative to the individual pixel’s depth [0;1] representing 0-100% or absolute in scene unit.

In relative mode, a pixel is classified as an edge pixel if:

|own_depth−neighbour_depth|>own_depth×factor\lvert\text{own\_depth} - \text{neighbour\_depth}\rvert > \text{own\_depth} \times \text{factor}

Example R1: Relative
  • Pixel Depth: 2m2 \si{\m}
  • Threshold: 0.50.5
  • Neighbor Pixel Depth: 3m3 \si{\m}
  • Calculation: |2m−3m|>2m×0.5→false\lvert 2 \si{\m} - 3 \si{\m} \rvert > 2 \si{\m} \times 0.5 \rightarrow \text{false}
  • Result: This will not detect an edge, as 1m≯1m1\si{\m} \not> 1\si{\m}.
Example R2: Relative
  • Same as Example R1 except for Threshold 0.5 0.490.49
  • Calculation: 1m>2m×0.49→true1 \si{\m} > 2 \si{\m} \times 0.49 \rightarrow \text{true}
  • Result: An edge pixel is detected, as 1m>0.98m1\si{\m} > 0.98\si{\m}.

In absolute mode, a pixel is classified as an edge pixel if:

|own_depth−neighbour_depth|>abs_value\lvert\text{own\_depth} - \text{neighbour\_depth}\rvert > \text{abs\_value}

Example A1: Absolute
  • Pixel Depth: 2m2 \si{\m}
  • Threshold: 0.5m0.5 \si{\m}
  • Neighbor Pixel Depth: 2.5m2.5 \si{\m}
  • Calculation: |2m−2.5m|>0.5m→false\lvert2 \si{\m} - 2.5 \si{\m} \rvert > 0.5 \si{\m} \rightarrow \text{false}
  • Result: This will not detect an edge, as 0.5m≯0.5m0.5 \si{\m} \not> 0.5 \si{\m}.
Example A2: Absolute
  • Same as Example A1 except for Threshold 0.5 0.49m0.49\si{\m}
  • Calculation: 0.5m>0.49m→true0.5 \si{\m} > 0.49 \si{\m} \rightarrow \text{true}
  • Result: An edge pixel is detected, as 0.5m>0.49m0.5 \si{\m} > 0.49 \si{\m}.
Knobs: Threshold

edge_detection_threshold, Slider, default: 0.075000

Detection of edges according to the selected mode. Either relative to the individual pixel’s depth [0;1] representing 0-100% or absolute in scene unit.

Group: Step 2: Set inpaining area / affected pixels for inpainting impact

Contains 4 elements: visualize, Alpha threshold, Close threshold to original pixel, Enlarge inpaint area

This step allows for a more granular selection of how and specifically where the inpainting algorithm is applied.

Enlarge inpaint area is a common parameter to adjust to the specifics of the given scene.

Restricting the inpainted area is often not necessary and thus it’s not adviced to change the default settings of Alpha threshold and Close threshold to original pixel except there is good reason.

Knobs: visualize

visualize_heuristic_inpainting, Checkbox, default: 0

Colors the inpainted pixels inserted by the heuristic algorithm (Step 2 and Step 3) in bright pink.

Inpainted scene, visualization not active
Visualization of inpainted pixels
Knobs: Alpha threshold

inpaint_alpha_threshold, Slider, min: 0, max: 1, default: 0.985000

No inpainting is applied if the foreground alpha is below this value.

Use and change this parameter only if you have non seperable alpha layers in the image and observe inpainting artifacts behind those. It is adviced to overlay the alpha elements later onto the image.

The inpaint algorithm is designed to insert pixels in obscurated areas behind foreground objects. If the foreground object is transparent this behvaior will be visible and thus might cause trouble.

The example uses very low levels of alpha to have strong transparency and thus a more easy visualization for documentation purpose.

Visualization of pointcloud, no lens effect applied. A foreground object was inserted with a transparency of alpha=0.1.
Visualization of the inpainted pixels. Alpha threshold was set to 0.11. No inpainted pixels are inserted, as they would be visible and possibly alter the input scene.
Visualization of the inpainted pixels. Alpha threshold was set to 0.09. Inpainted pixels are heavily visible behind the transparent foregound object.
Knobs: Close threshold to original pixel

skip_if_too_close_perc, Slider, default: 0.010000

Don’t add any inpainted pixels, if the original pointcloud has a pixel in that relative range (same calculation applies as in Edge detection mode ‘Relative’).

For an explanation on how the Relative calculation works, see Edge detection mode.

The following example shows when this parameter comes into effect. With a scene that already has original data in places where inpainting would happen, inpainting pixels are ignored if they would be inserted in the given range around the original depth pixel. Selecting a value of 0 will disable this optimization and inpainted samples will be inserted no matter what.

Camera view of the inpainting edge detection
Scene view with DeepToPoints. Inpainted samples are highlighted, but carry the same information as the original scene does anyhow. This is why they can be fully ignored.
Knobs: Enlarge inpaint area

increase_inpaint_ray, Number, min: 0, max: 10, default: 2

Inpainting is applied perpendicular to the detected edges. This parameter specifies the width of the inpaint brush around the inpainted pixel (width = 2*n+1)

The coverage of individual inpaint rays can be troublesome, as the edge detection algorithm loses information and precision by rasterization in the pixel grid. Especially curved edges and high frequent changes need a bigger coverage in the width of the ray to have all of the necessary area inpainted accordingly.

Visualization of how inpainting is applied: ray perpendicular to a detected edge
Padded ray with enlarge inpaint area of 1

Group: Step 3: How many pixels get generated within previously defined area

Contains 1 elements: Inpainting intensity

Knobs: Inpainting intensity

inpaint_intensity, Slider, min: 0, max: 10, default: 1.000000

Scales the length of the inpainting brush stroke perpendicular to the detected edge.

Visualization of how inpainting is applied: ray perpendicular to a detected edge
Inpainting ray scaled by inpaint_instensity

Group: Step 4: Repeat edge pixels in z-direction

Contains 4 elements: visualize, Mode, Edge repetitions, Step

Rendering objects front-faced only makes them two dimensional for the renderer, even if they realistically have a depth.

This algorithm allows to extend the detected edge pixels backwards in z-direction to simulate this depth of the foreground object. Opposed to the Heuristic inpaint approach in Step 2 and 3, this will extend the foreground object instead of the background.

Visualized edge repetition to extend a flat foreground object and simulate depth along the z dimension.
Knobs: visualize

visualize_virtual_depth_on_edges, Checkbox, default: 0

Colors the inpainted pixels inserted by the edge repetition algorithm (Step 4) in bright pink.

Knobs: Mode

edge_repetition_mode, Dropdown, default: None

Options: [None, Relative, Absolute]

Defines how the depth will be extrapolated by the pixel inserter (same definition of modes as edge_detection_mode).

Knobs: Edge repetitions

edge_repetition_count, Number, default: 1

The number virtual samples inserted on a detected edge with the given step spacing.

Relative

new_depthcount=repetition_sample_depth×(1+count×edge_repetition_step) \text{new\_depth}_{\text{count}} = \text{repetition\_sample\_depth} \times (1 + \text{count} \times \text{edge\_repetition\_step})

Absolute

new_depthcount=repetition_sample_depth+count×edge_repetition_step \text{new\_depth}_\text{count} = \text{repetition\_sample\_depth} + \text{count} \times \text{edge\_repetition\_step}

Knobs: Step

edge_repetition_step, Slider, default: 0.030000

Step between each repetition (unit specified by edge_repetition_mode).

Group: Clamp_to_edge

Contains 1 elements: Enable

This feature is designed to fix small black “shadows” at the edges of the image.

Missing scene information will be filled with the most recent data along the closer axis.

If this behavior is insufficient, Show overscan coordinates can be used to visualize the amount of missing scene information, allowing the content to be rendered accordingly.

Clamp to edge algorithm not active. Missing scene pixels caused by distortion of the lens
Fixed by applying clamp to edge
Knobs: Enable

clamp_to_edge, Checkbox, default: 0

Enables or disables the feature.

Group: Light_source_booster

Contains 3 elements: Enable light source modification, Detection threshold, Scaling

The contrast of any given bokeh depends on its size and the brightness of the source. The energy is distributed over the entire bokeh area, so the larger the bokeh, the less contrast it will have.

This feature allows for manual fine-tuning of bokeh contrast for highlights.

No light source booster
Light source booster applied
Knobs: Enable light source modification

light_source_modification, Checkbox, default: 0

Enables or disables the feature. Further UI elements will be hidden if disabled.

Knobs: Detection threshold

light_source_detection_threshold, Slider, min: 0, max: 100, default: 1.000000

Pixel values with grayscale color bigger than this value will be selected for scaling.

This parameter is the threshold value in grayscale that defines when a pixel is detected as a light and will be boosted. The light_source_detection_threshold shifts the scaling curve accordingly.

Knobs: Scaling

light_source_scaling, Slider, min: 0, max: 10, default: 0.100000

The non-linear gain of the color scaling for all selected pixels.

This parameter describes the strength of the nonlinear scaling applied to elements above the light_source_detection_threshold.

The curve below the detection threshold remains an identity to avoid manipulating the input data. At the threshold, both the identity and the nonlinear curves have the same slope to ensure continuous behavior.

The curve appears as follows:

Scaling curve in action. t is the detection threshold, a is the scaling paramter. x-Axis: input_grayscale_color, y-Axis: output_color

Group: Troubleshooting

Contains 2 elements: Rescale alpha, Close alpha holes

Knobs: Rescale alpha

rescale_alpha, Slider, min: 0.9, max: 1, default: 1.000000

The alpha channel of the whole image is rescaled from the range [0; rescale_alpha] to [0; 1]. Alpha values greater than rescale_alpha will be capped at 1.0.

This algorithm is designed to fix very small deviations in the alpha channel, specifically those smaller than 5%. It remaps the alpha values of the entire image from the range [0; rescale_alpha] to [0; 1], converting values above the specified threshold to an alpha of 1 in the output image while linearly scaling all other areas.

Small alpha holes visible (with extreme gamma settings to enhance contrast) after applying the inpaint algorithm. The darkest alpha values observed in the mask are 0.95189.
No more holes or degraded alpha visible after applying the rescale_alpha algorithm with a value of 0.95
Knobs: Close alpha holes

close_alpha_holes, Number, min: 0, max: 2.14748, default: 0

The algorithm closes holes in the alpha channel by specifying a pixel radius for filling, recommended for use after inpainting.

This enhancement algorithm is designed to close holes in the alpha channel and is recommended for use after inpainting. It specifies the number of pixels to be filled, with a value of 0 disabling the feature and values greater than 0 indicating the radius of pixels to close the holes.

This algorithm offers an alternative to Rescale alpha and is particularly effective for small holes or very fine edge artifacts. It operates on the entire image, so users should be cautious of potential artifacts at the edges.

Slight depth shadows along the feathers of the character. The alpha error for these is quite severe with values of < 0.8
Fixed depth shadows by using the close_alpha_holes algorithm with a value of 2 px

Troubleshooting

Artifacts when using Nuke’s DeepRecolor

Rendering real deep data is costly and thus oftentimes just the depth and alpha passes are rendered and Nuke’s DeepRecolor node is used to have a approximation of the deep render.

This usecase is specifically troublesome, as there is only a single color layer (the alpha blended result) that is overlayed to all depths in the scene.

Example of a deep recolor workflow

Even if the Nuke-UI seems to have different RGB colors for each individual entry in a deep pixel, the RGB contents actually are all the same, just multiplied by the varying alpha values.

Overview of the deep pixels with DeepSample

The main problem with DeepRecolor is, that pixels are now in the back of the scene (wing) are colored with the foreground color (neck). The higher the contrast in the area of the image, the bigger the problems. The plugin always respects and prioritizes the input data before inpainted pixels. By feeding wrong pixel values, the inpainting is effectively turned off, despite inpainting most probably do a better job in these situations than the DeepRecolor.

By just using the data as is, the defocus seems off as the brighter brown has defocus applied around the neck.
Same rendering with the DataPrep node applied directly after the DeepRecolor

Solution: DataPrep

DataPrep is an additional node available with the plugin. It effectively merges the faulty DeepRecolor data to a single layer depth representation with the frontmost depth. This node is a helper node added to support existing workflows. If starting from scratch, it is not advised to render and use the DeepRecolor workflow in the first place as we currently do not know of any DeepRecolor setup that really enhances the quality of the scene - at least when using the LensCore-plugin.

A second use case of the DataPrep node is a usage with a very low group factor of <= 0.05. This will merge deep pixels that are very close to a single one and will thus reduce memory footprint - especially regarding VRAM on the GPU.

Nuke’s DeepToImage uses alpha-blending for the deep.front entries aswell to create the depth.Z layer. This is why a workflow like DeepToImage → DeepFromImage can not be used to flatten a DeepRecolor image. This is why we added the DataPrep node.

The plugin DataPrep shall always be used directly after the DeepRecolor node.

The plugin has a single parameter Relative group factor that allows tune the bandwidth in depth that pixels are grouped in. For an explanation on how the Relative calculation works, see Edge detection mode.

DataPrep turned off by setting a group factor of 0.00
A group factor of 0.05 will merge the front group and the back group. This usecase will improve overall performance without noticible changes in the produced output image. This usage will not fix the DeepRecolor problem.
Relative group factor of 1.00 big enough so that all pixels in the troublesome area are merged into a single front-depth pixel. This essentially throws away the back pixels and will rely on inpainting to fill in the missing parts.

The following visualization uses Nuke’s DeepToPoints to showcase the resulting 3D-Pointcloud with the varying usecases of the DataPrep node.

Relative group factor of 0.00
Relative group factor of 0.05
Relative group factor of 1.00

Artifacts when using a single layer depth.Z

Transparency combined with depth data in a physical point-cloud scene can be very challenging. Many renderers, by default, treat depth data identically to color layers when dealing with fine detail or edges exhibiting transparency. This can lead to significant problems because the depth information for discrete objects must not be alpha-blended. Alpha blending introduces hallucinated pixels between two discrete states.

Part of the scene rendered with the DeepRecolor workflow. There are two visible layers to the image (neck and wing) seperated by missing data.
Same cutout of the scene, rendered as a single layer with depth.Z. The depth was produced with antialiasing and thus invalid depth values are inserted between wing and neck.
Lens effect rendered on the alpha-blended depth input image.
Same section with fixed depth.Z samples

Solution: Fix depth

Nuke doesn’t open after installing the plugin

Example error message when launching the plugin with an incompatible version of nuke

Solution: Use matching plugin version

Inpainting map error

Format mismatch error

Solution: Use the same resolution

Select the correct format in the producing texture or use a Reformat node to scale the Inpaint map before connecting it to the plugin.

Nuke window is not responsive

Depending on the specific scene and lens configuration, the plugin can be quite heavy on the system. Unfortunately Nuke is implemented in a way, that the main thread stalls until the first lines are returned from the plugin. In e.g. Rocky Linux 9 with the GNOME system manager, this will lead to an “Nuke is not responsive” popup.

Linux GNOME “not responsive” warning dialog, for a correct lens configuration just wait for Nuke to get responsive again, otherwise “Force Quit” and fix rendering settings.

Solution: Fix settings, reduce or wait

Heavy impact on the system can have multiple reasons.

If everything seems to be configured correctly, it is best to not click anything and wait until the process recovers, otherwise Nuke will detect the changes directly after rendering and restart the new setting frame directly afterwards which makes the window unresponsive for even longer.

Black output image with intel-compute-runtime

Hardware configuration with an integrated GPU card might use the https://github.com/intel/compute-runtime driver. For a linux system with the Iris Xe Graphics onboard GPU, we observed a malfunction of the driver+plugin combination that leads to a full 0, 0, 0, 0 output image.

Solution: Use other driver

Either use another device like a dedicated GPU or a CPU implementation for OpenCL or install another driver like opencl-mesa with rusticl. In our tests this resolved the issue for the same graphics card.