High Dynamic Range Video

High dynamic range video (HDRv) is an emerging technology (What is HDR?), by many considered to be one of the key components in future of imaging. HDRv will enable a wide range of new applications in image processing, computer vision, computer graphics and cinematography. A few prototype HDR-video camera systems have recently been developed, showing that the computational power and bandwidth is now high enough to handle high resolution HDR-video processing and storage. Current CMOS and CCD imaging sensors are still unable to accurately capture the full dynamic range in general scenes. Here we describe how we capture HDR video.

HDRv capture

The HDRv sequences are captured using an HDRv imaging system developed in a collaboration between the camera manufacturer SpheronVR and the Visual Computing Laboratory at Linköping University in Sweden.

The 4 Mpixel HDRv camera.

The camera is a multi-sensor imaging system that captures HDR images with a resolution of 2336x1752 pixels at up to 30 fps with a dynamic range of over 24 f-stops. The high bandwidth HDR data stream can be viewed in real time through GPU processing, and is written without compression to file on an external storage unit. Both the camera and storage unit runs on battery for up to three hours of sustained capture. A detailed overview of the imaging hardware setups and the image recosntruction algorithms can be found in the following two papers: Unified HDR Reconstruction from RAW CFA Data, and A Unified Framework for Multi-Sensor HDR Video Reconstruction.

Displaying HDRv

One of the main challenges in HDR imaging and video is to map the dynamic range of the HDR image and real world to the usually much smaller dynamic range of the display device. While an HDR image captured in a high contrast real scene often exhibit a dynamic range in the order of 5 to 10 log10 units a conventional display system is limited to a dynamic range in the order of 2 to 4 log10 units. The mapping of pixel values from an HDR image or video sequence to the display system is called tone mapping, and is carried out using a tone mapping operator (TMO). Most TMOs rely on models that simulate the human visual system. Over the last two decades, extensive research has been carried out and led to the development of a large number of TMOs. However, only a handful of them consider the temporal domain, that is for HDR-video content, in a consistent and robust way. This lack of HDR-video TMOs is likely due to the (very) limited availability of HDR-video footage. Until recently, artificial and static scenes have been the only material available. Recent development in HDR video capture, however, opens up new possibilities for advancing techniques in the area.
In order to display our HDRv sequences on ordinary monitors, projectors and TV-sets, we have made a thorough evaluation of exising techniques. Our study (a detailed overview can be found here) shows that the existing TMOs perform very differently and that new display algorithms need to be developed. The video below displays an example comparison between six different TMOs.

What is High Dynamic Range?

The dynamic range (Wikipedia) of a camera is the ratio between the largest and smallest possible light intensities that the camera can capture. If the intensity is too small, the pixel value will be black. If the intensity is to large, the pixel will be saturated and be encoded as being white.
An ordinary camera exhibits a dynamic range in the order of 1000 : 1 or less, while an ordinary indoor scene usually exhibits a dynamic range of more than 100.000 : 1 or more. An outdoor scene including the sun usually exhibits a dynamic range of up to 10.000.000 : 1 or more. This means that the dynamic range of the ordinary camera cannot capture the full range of light intensities within a single exposure setting. The exposure settings such as exposure time and aperture size controls the amount of light that reaches the sensor. This means that the dynamic range available on the camera can be adjusted to capture a certain range of light intensities.

The images in sequence above are captured with a semi-professional digital SLR camera, where the exposure time has been changed between each frame. From left to right, the exposure time is increased. It is evident that the dynamic range of the camera cannot capture that of the scene, and that the bright background requires very short exposure times while the much darker foreground requires long exposure times.
An high dynamic range (HDR) (Wikipedia) image covers the full dynamic range of the scene and thus captures all its light intensities. In the example above, this means that an HDR image would image both the background and the foreground simultaneosly.
HDR imaging makes it possible to accurately capture and represent the exact lighting conditions (scene radiance) found in the real world scene. In contrast to conventional images, this enables significantly more advanced: display, computer vision and image post processing algorithms, as well as applications such as computer graphics rendering of photo-realistic images of synthetic objects placed into the captured real world scene. Early adopters of HDR imaging are the movie special effects, computer games, and product and architectural visualization industry.