This project provides efficient vectorized Python code to perform depth <-> color encoding based on

Sonoda, Tetsuri, and Anders Grunnet-Jepsen. "Depth image compression by colorization for Intel RealSense depth cameras." Intel, Rev 1.0 (2021).

with the following significant changes:

1. We stop the hue range at 300° (default) in order to avoid having close and far ranges assigned redish color.
2. We detect de-valued and de-saturized colors in the decoding and assign it a pre-defined failure depth.

The encoding is designed to transform 16bit single channel images to RGB color images that can be processed by standard (lossy) image codec with minimized compression artefacts. This leads to a compression factor of up to 80x while maintaining acceptable depth accuracy for many RGBD systems.

The underlying method is described in

Sonoda, Tetsuri, and Anders Grunnet-Jepsen. "Depth image compression by colorization for Intel RealSense depth cameras." Intel, Rev 1.0 (2021).

However, the formulae presented appear to be inaccurate and specific important details seem to be left out. Here we describe the base encoding plus our adaptations.

The base method relies on encoding depth as fully saturated/valued colors and assigns increasing depth values increasing hues. For bitdepth $b$ this allows for

unique colors. In the original method $\textrm{Hue}_{\textrm{max}}=360°$, whereas in this implementation $\textrm{Hue}_{\textrm{max}}=300°$ to avoid ambiguitis between close and far depth values that would otherwise get assigned a reddish color.

Limiting $\textrm{Hue}_{\textrm{max}}$ limits the number of available bits to $\log_2 n$ (~10.3bit).

The encoding allows for linear and disparity depth normalization. In linear mode, equal depth ratios are preserved in the encoding range, whereas in disparity mode more emphasis is put on closer depth values than on larger ones, leading to more accurare depth resolution closeup.

This implementation is vectorized using numpy and can handle any image shapes (*,H,W) <-> (*,H,W,3). For improved performance, we precompute encoder and decoder lookup tables to reduce encoding/decoding to a simple lookup. The lookup tables require ~32MB of memory. Use use_lut=False switch to rely on the pure vectorized implementation. See benchmarks below for effects.

With Python >= 3.11 execute

git clone https://github.com/cheind/hue-depth-encoding.git
pip install -e .

# or development install adds more packages to run tests/analysis
pip install -e '.[dev]'
# or with gpu codec support in dev mode
pip install -e '.[dev]' --no-binary=av 

Basic usage

# Import
import huecodec as hc

# Random float depths
d = rng.random((5,240,320), dtype=np.float32)*0.9 + 0.1

# Encode
rgb = hc.depth2rgb(d, zrange=(0.1,1.0), inv_depth=False)
# (5,240,320,3), uint8

# Decode
depth = hc.rgb2depth(rgb, zrange=(0.1,1.0), inv_depth=False)
# (5,240,320), float32

The script python analysis.py compares encoding and decoding characteristics for different standard video codecs using hue depth encoding. The reported figures have the following meaning:

• rmse [m] root mean square error per depth pixel between groundtruth and transcoded depthmaps
• <x [%] percent of absolute errors less than x
• failed [%] percent of reconstructions failed during decoding. This appear when using lossy video codecs and rgb values are de-saturated or de-valued too much by the codec.
• tenc [milli-sec/frame] encoding time per frame
• tdec [milli-sec/frame] decoding time per frame
• nbytes [kb/frame] kilo-bytes per encoded frame on disk.

All tests are carried out on a 12th Gen Intel® Core™ i9-12900K × 24 with NVIDIA GeForce RTX™ 3090 Ti.

# run analysis only for specific variants
python analysis.py variant=[hue-only,h264-lossless-gpu]

Each test encodes/decodes a sequence (100,512,512) of np.float32 depthmaps in range [0..2] containing a sinusoidal pattern plus random hard depth edges. The pattern moves horizontally over time.

The reported values are

Each tests encodes a sequence of (30,600,800) of np.float32 depthmaps taken with a RealSense 415 in range [0..2] containing a sitting person moving.

The reported values are

python analysis.py data=path/to/npy

Here are benchmark results for encoding/decoding float32 depthmaps of various sizes with differnt characteristics. Note, this is pure depth -> color -> depth transcoding without any video codecs involved.

------------- benchmark: 8 tests ------------
Name (time in ms)              Mean          
---------------------------------------------
enc_perf[LUT-(640x480)]        2.1851 (1.0)    
dec_perf[LUT-(640x480)]        2.2124 (1.01)   
enc_perf[LUT-(1920x1080)]     17.9139 (8.20) 
dec_perf[LUT-(1920x1080)]     16.4741 (7.54)   
enc_perf[noLUT-(640x480)]     22.3938 (10.25)  
dec_perf[noLUT-(640x480)]      6.9320 (3.17)   
dec_perf[noLUT-(1920x1080)]   74.6038 (34.14)  
enc_perf[noLUT-(1920x1080)]  158.0871 (72.35)  
---------------------------------------------

# run tests and benchmarks  
pytest

When tuning the prameters for a video codec you need to take into account lossy compression that happens on different levels of encoding:

• spatial/temporal quantization based how images change over time. Usually controlled by the codec preset/profile (lossless, low-latency,...) and quality parameters such as crf, qp.
• color space quantization due to converting RGB to target pixel format. Different codecs support different pixel color formats of which most perform a lossy compression from rgb to target space.

Print encoder supported options and pixel formats

# print encoder options
ffmpeg -h encoder=h264_nvenc

See packing and compression of color formats https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/pixfmt.h

Here are some take aways to consider

• adjust zrange as tightly as possible to your use-case. This increases filesize but reduces reconstruction loss.
• prefer loss-less codecs if affordable
• when using lossy codecs ensure that your application does not rely on depth-edges.

The following plot shows a lossy h264 encoding on real data. Most of the errors are sub-mm, only few > 1cm.

Turns out these errors are located on depth edges where the lossy codec starts interpolating values.

The original paper referenced is potentially inaccurate in its equations. This has been noted in varios posts #10415,#11187,#10302.

This implementation is based on the original paper and code from https://github.com/jdtremaine/hue-codec/.