Teaser
Video frame from [Fröhlich et al. 2014]

Abstract

Real-time high quality video tone-mapping is needed for many applications, such as digital viewfinders in cameras, display algorithms which adapt to ambient light, in-camera processing, rendering engines for video games and video post-processing. We propose a viable solution for these applications by designing a video tone mapping operator that controls the visibility of the noise, adapts to display and viewing environment, minimizes contrast distortions, preserves or enhances image details, and can be run in real-time on an incoming sequence without any preprocessing. To our knowledge, no existing solution offers all these features. Our novel contributions are: a fast procedure for computing local display-adaptive tone-curves which minimize contrast distortions, a fast method for detail enhancement free from ringing artifacts, and an integrated video tone-mapping solution combining all the above features.

Video

Overview

We propose a new tone mapping operator (TMO) for video, which controls the visibility of the noise, adapts to the display and viewing environment, minimizes contrast distortions, preserves or enhances image details, and can be run in real-time on an incoming sequence without any preprocessing:

Overview
Diagram of our video tone-mapping solution. The input is split into a base and a detail layer using an edge preserving filter specifically designed for the purpose of detail extraction in the context of tone-mapping. The base layer is then compressed using a novel method for calculating locally adaptive tone curves that minimize contrast distortions and controls the visibility of image noise. The layers are then combined and converted with an inverse display model into pixel values before display.

As indicated in the figure above, our algorithm can be divided into three main parts, where each part contains novel key features that are central for the goals we set out to achieve:


I. Minimum contrast distortion tone curves

We propose a fast method for computing local display-adaptive tone curves. The tone curves minimize contrast distortions introduced by the mapping while controlling the visibility of noise, and their shapes are filtered over time to avoid temporal flickering.

Global tone-curve
Global tone-curve
Local tone-curve
Local tone-curve
Tone-mapping using global (left) or local (right) tone-curves. The image was generated assuming high ambient light viewing (low contrast) to better show contrast reduction. Note that other components of the operator, such as local contrast, were disabled for this example. Video frame from [Fröhlich et al. 2014]

II. Filter design for base – detail layer decomposition for tone mapping

The filter used for separating image details is constructed by analyzing current edge-preserving filters and their potential problems in the context of tone mapping. From the observations, and a unified formulation of bilateral filtering and anisotropic diffusion, we design a fast edge-stopping non-linear diffusion approximation for detail enhancement without ringing artifacts.

Bilateral filter
Bilateral filter
Anisotropic diffusion
Anisotropic diffusion
Our approach
Our approach
The figures use the same tone curve, but different edge-preserving filters for detail extraction. The details are scaled by a factor 5 to highlight filtering differences and artifacts, and enlarged versions show the areas indicated by the red squares in the images. Video frame from [Fröhlich et al. 2014]
Our approach
Our approach
Guided filter, 5x5 kernel size
Guided filter, 5x5 kernel size
Guided filter, 10x10 kernel size
Guided filter, 10x10 kernel size
Comparison with the guided filter [He et al. 2013]. The guided filter shows no ringing artifacts, but either produces details on a too small scale (brick wall in middle image), or tend to show some haloing artifacts (fingers in right image). Video frame from [Fröhlich et al. 2014]

III. Noise-aware control over image details

In addition to the use of noise-aware tone curves adaptive to image noise, the visibility of noise is also controlled in the processed detail layer by taking into account the noise characteristics of the original base layer and the tone-mapped base layer. This is to allow the user to preserve or artistically enhance local contrast and details by scaling the detail layer, while still making sure that the noise is kept below the visibility threshold.

Noise-aware comparison
Comparison of tone-mapping disregarding noise (top left) with our noise-aware approach (top right), where excessive noise is hidden in darker tones. The bottom row shows the result of complementing the tone mapping with a denoising step (V-BM4D). Video frame from [Fröhlich et al. 2014]

Downloads

Download Size Description
Paper 28.9Mb Main paper
Supplementary document 5.5Mb Supporting document with additional figures and results.
Presentation 38.3Mb Presentation from Siggraph Asia 2015, Kobe.
Video 94.3Mb The video overview presented above.
Source code - Code available for research purposes on request (for more information, please contact gabriel.eilertsen@liu.se).

Citation

Reference

Gabriel Eilertsen, Rafał K. Mantiuk, and Jonas Unger. Real-time noise-aware tone mapping. In: ACM Transactions on Graphics (Proc. of SIGGRAPH Asia 2015), 34(6), 2015.

BibTeX

@article{Eilertsen:2015,
  author    = {Eilertsen, Gabriel and Mantiuk, Rafa\l K. and Unger, Jonas}, 
  title     = {Real-time noise-aware tone mapping}, 
  journal   = {ACM Transactions on Graphics (TOG)}, 
  volume    = {34}, 
  number    = {6}, 
  year      = {2015}, 
  numpages  = {15}, 
  url       = {http://doi.acm.org/10.1145/2816795.2818092}, 
  doi       = {10.1145/2816795.2818092}, 
  publisher = {ACM}, 
  address   = {New York, NY, USA}, 
  keywords  = {tone mapping, video tone mapping, hdr imaging}, 
}