The Luma HDRv codec uses a perceptually motivated method to store high dynamic range (HDR) video with a limited number of bits. The method, as originally described in the HDR Extension for MPEG-4, ensures that the quantized video stream will be visually indistinguishable from the input HDR. The stream is then compressed using Google's VP9 video codec, which provides a 12 bit encoding depth. This is more than enough to ensure that quantization errors are invisible.
Having an HDR video stream, distributed at a low bandwidth, with the same flexibility as standard encoding schemes, enables a range of possible new applications. HDR displays are becoming more and more available, and to be used they need a solution for broadcasting of HDR video in an efficient way. Also for standard displays, HDR video material increases the possibility to perform post-processing on location, where for example a TV could be tuned for optimal experience, e.g. in terms of the tone reproduction utilized.
Luma HDRv provides capabilities to encode HDR video with a number of different settings (transfer function, color space, bit-depth etc.). The default settings encodes luminances between 0.005 and 10000 cd/m2 at 11 bits by means of the perceptual quantizer (PQ, SMPTE ST 2084) function, and colors at 8 bits/channel in the Lu'v' color space.
It is also possible to encode in a format that is compatible with existing HDR video standards. For example, by encoding luminance with PQ at 10 bits, and colors in the YCbCr color space according to recommendation ITU-R BT.2020, the result complies with the widely used HDR10 standard. Luma HDRv also stores the appropriate metadata, so that this makes the video compatible with applications that support HDR10, and that can decode VP9 codec stored in a Matroska container. It is for example possible to upload the result to Youtube and Vimeo, where it is properly recognized as HDR video.
Comparing HDR10 and the default settings of Luma HDRv, the latter is able to provide a better trade-off between bit-rate and quality, due to the perceptually motivated Lu'v' color encoding. The following plot shows the difference in performance for a range of encoding quality settings, as measured by means of HDR-VDP-2:
The Luma HDRv project is released as open source, under the BSD license. It provides both libraries for including the HDR video encoding and decoding functionality into any project, as well as applications to perform encoding and decoding with a number of different settings, and for playback of encoded HDR video material:
The Luma HDRv library provides a C++ API for encoding and decoding of HDR video. The video compression is performed with Google's VP9 video codec. The encoded stream is stored using the Matroska (.mkv) container format, for flexibility and compatibility.
Included in the Luma HDRv project are command line applications to encode and decode HDR video sequences with a large number of different settings. The input and output HDR frames could be in OpenEXR, or be passed from the PFStools HDR image application package.
The Luma HDRv project also provides an HDR video player, which can load and playback the encoded HDR streams in real-time. The player interface has features for performing manipulations such as changing the exposure and applying tone-mapping algorithms.
An open source, perception-motivated HDR video format
The videos have been encoded at quantization scalings 2 (qs2) and 6 (qs6), providing different trade-offs between quality and size. These can be downloaded and played with the provided lumaplayer application. The original data is available as separate OpenEXR frames at the HDR repository provided by Jan Fröhlich et al., and may be used freely for research purposes:
Froehlich J., Grandinetti S., Eberhardt B., Walter S., Schilling A., Brendel H. (2014). Creating Cinematic Wide Gamut HDR-Video for the Evaluation of Tone Mapping Operators and HDR-Displays. In Proc. SPIE 9023, Digital Photography X, pp. 90230X–90230X–10.
The use of the codec and the accompanying HDR video player are described in the manual pages of lumaenc, lumadec and lumaplay. Furthermore, simple examples of how to use the luma_encoder and luma_decoder libraries are included in the source code. More information is also available on the GitHub project page.
The codec and its design are described in our ICIP 2016 paper, and as a talk at Siggraph 2016:
Eilertsen G., Mantiuk R.K., and Unger J. (2016). A high dynamic range video codec optimized by large-scale testing. 2016 IEEE International Conference on Image Processing (ICIP).
Eilertsen G., Mantiuk R.K., and Unger J. (2016). Luma HDRv: an open source high dynamic range video codec optimized by large-scale testing. ACM SIGGRAPH 2016 Talks.
A more recent summary of Luma HDRv and the evaluation from the above papers, as well as a comparison to HDR10, can be found in Chapter 4 of the following publication:
Eilertsen G. (2018). The high dynamic range imaging pipeline: Tone-mapping, distribution, and single-exposure reconstruction. Linköping Studies in Science and Technology. Dissertations, ISSN 0345-7524 ; 1939. Linköping University Electronic Press, 2018.
If you use the Luma HDRv software for your research work, please consider citing as follows:
@inproceedings{EMU2016,
author={Gabriel Eilertsen and Rafal K. Mantiuk and Jonas Unger},
booktitle={Image Processing (ICIP), 2016 IEEE International Conference on},
title={A high dynamic range video codec optimized by large-scale testing},
year={2016},
}
Luma HDRv was developed in the Visual Computing Laboratory. It is a lab and research environment hosted by the Computer Graphics and Image Processing group in the division of Media and Information Technology at Linköping University, Norrköping, Sweden.
The Luma HDRv encoding and decoding applications provide support for the PFStools HDR image application package. In this way HDR video encoding and decoding can be included in a PFStools command line pipeline.
The HDR repository at Linköping University contains a wide range of HDR videos, captured with a custom built HDR video camera capable of covering a dynamic range of up to 24 fstops. These can be used to test Luma HDRv.
One possible application area for the Luma HDRv codec is tone-mapping on the user side. That is, an HDR video stream can be compressed with Luma HDRv, transmitted, and finally be tone-mapped for the receiver display. Our project on Real-time noise-aware tone mapping provides a set of novel contributions to enable efficient and high-quality tone-mapping of HDR video.
Luma is an open source implementation of lossy high dynamic range (HDR) video compression. HDR video can represent the full range of colors and brightness levels that can be found in the real-world, and is therefore a much more accurate representation of real-world scenes. HDR video is especially important for upcoming HDR displays, which can take advantage of the additional range of colors and brightness stored in HDR video files.
To learn more on HDR, you can check a free review article.
Luma is the name used for achromatic (black and white channel) pixel values used in video compression. For improved efficiency of video compression, red green and blue color channels are converted into one luma channel (black and white) and two color channels (red-green and blue-yellow). One of the main challenges of HDR video compression is the transformation from very a large range of physical luminance (contrast up to 100,000,000:1) into 12 bits or less (contrast 4096:1) that can be encoded in video, hence the name Luma for the codec.
The color and luminance encoding used in the Luma HDRv coded was derived from human visual performance data so that the color encoding is aligned with the sensitivity of the eye. In this way, the loss of information introduced by converting 32-bit floating point values of the original HDR frames into 12-bit luma and 8-bit chroma channels is not visible to the human eye.
The encoding and decoding command line tools support OpenEXR frames. However, almost any imaginable HDR formats is supported if the encoder is used in combination with pfstools. Check the manual page of lumaenc and lumadec for more details.
Yes and no. In order for perceptual encoding to function properly, the input frames must be scaled in physical units of cd/m^2. Most HDR frames are given in relative units, where a value of 1 is either the brightest or close to the brightest pixel value, much lower than the corresponding physical value. Such frames will be encoded with very poor quality as the codec will assume that those pixels are very dark (1 cd/m^2 is the luminance of relatively dark shadow), and the human eye will not be able to notice much difference for those dark pixels anyway.
Therefore, all input frames should be at least approximately scaled in order to make them closer to physical luminance values. This can be done either manually by passing the --pre-scaling parameter to lumaenc (a value between 1000 and 10000 should work well in most cases), or automatically using, for example, the display adaptive tone-mapping operator from pfstools:
pfsin relative_frame_%04d.exr | pfstmo_mantiuk08 -e 1 -d l=4000:b=0.01:a=0 | pfsdisplayfunction -d l=4000:b=0.01:a=0 -l | lumaenc --output video.mkv
The above command will tone-map the video for an HDR display with the peak luminance of 4000 cd/m^2 and black level of 0.01 cd/m^2. pfsdisplayfunction is used to convert values from luma to luminance.
The codec and its design is described in our paper presented at ICIP 2016, and as a talk at Siggraph 2016:
Eilertsen G., Mantiuk R.K., and Unger J. (2016). A high dynamic range video codec optimized by large-scale testing. 2016 IEEE International Conference on Image Processing (ICIP).
Eilertsen G., Mantiuk R.K., and Unger J. (2016). Luma HDRv: an open source high dynamic range video codec optimized by large-scale testing. ACM SIGGRAPH 2016 Talks.
A more recent summary of Luma HDRv and the evaluation from the above papers, as well as a comparison to HDR10, can be found in Chapter 4 of the following publication:
Eilertsen G. (2018). The high dynamic range imaging pipeline: Tone-mapping, distribution, and single-exposure reconstruction. Linköping Studies in Science and Technology. Dissertations, ISSN 0345-7524 ; 1939. Linköping University Electronic Press, 2018.
The luma encoding is based on the concept from the paper:
Mantiuk, R., Krawczyk, G., Myszkowski, K., and Seidel, H.-P. (2004). Perception-motivated high dynamic range video encoding. ACM Transactions on Graphics (Proc. of SIGGRAPH), 23(3), 733. doi:10.1145/1015706.1015794 (project page)
The default encoding function comes from the more recent paper:
Miller, S., Nezamabadi, M., and Daly, S. (2013). Perceptual Signal Coding for More Efficient Usage of Bit Codes. SMPTE Motion Imaging Journal, 122(4), 52–59. doi:10.5594/j18290
which was demonstrated to perform very well in terms of hiding quantization artifacts:
Boitard, R., Mantiuk, R. K., and Pouli, T. (2015). Evaluation of color encodings for high dynamic range pixels. In Human Vision and Electronic Imaging (p. 93941K). doi:10.1117/12.2077715
The color encoding is based on the idea from the paper:
Ward-Larson, G. (1998). LogLuv Encoding for Full-Gamut, High-Dynamic Range Images. Journal of Graphics Tools, 3(1), 15–31. doi:10.1080/10867651.1998.10487485
Certainly yes. The BSD software license was selected to simplify the integration of the encoding/decoding library with both Open Source and commercial software. However, as this is the first release of the project and the library interface may still be altered, we recommend that you contact us before you put effort into integration with the library.
The codec uses Google's VP9 codec as the base encoder, and Matroska for packaging the encoded video and metadata into .mkv files. If any of those three libraries is missing on the system, the build script will compile a static version of the library, which is bundled with the code.
We performed an extensive evaluation of the codec using HDR-VDP-2.2. The evaluation is reported in detail in our paper presented at ICIP 2016, and as a talk at Siggraph 2016:
Eilertsen G., Mantiuk R.K., and Unger J. (2016). A high dynamic range video codec optimized by large-scale testing. 2016 IEEE International Conference on Image Processing (ICIP).
Eilertsen G., Mantiuk R.K., and Unger J. (2016). Luma HDRv: an open source high dynamic range video codec optimized by large-scale testing. ACM SIGGRAPH 2016 Talks.
The evaluation enabled us to optimize different aspects of the coded and choose the best encoding options for HDR video. As compared to the legacy code of the MPEG-4 HDR video codec (proprietary), Luma offers a substantial improvement in compression efficiency.
Gabriel Eilertsen - Ph.D. candidate at the Visual Computing Laboratory, Linköping University (Sweden) and main author of the codec.
Jonas Unger - Assistant professor at Linköping University, has been active in the fields of HDR-video capture and tone mapping for over a decade.
Rafal K. Mantiuk - Senior Lecturer at the University Cambridge (UK), the (co-)author of the first HDR video codec, the backward-compatible HDR video compression, pfstools and the HDR-VDP.
Questions, comments and bug reports are most welcome at the Luma HDRV google group
For direct contact, contact information for Gabriel Eilertsen can be found at the Visual Computing Laboratory (VCL)
