The CineD Camera Lab is Back – Dynamic Range Tests

Lab Test

by Gunther MachuOctober 22^nd, 2018

As many of you are well aware, the CineD lab (formerly cinema5D lab) was built up a few years ago, and huge efforts were put into detailed camera tests concerning dynamic range, sharpness, rolling shutter etc …

No new measurements were done in a while, but now we decided to start again, in a step by step manner, building on this foundation.

Therefore, as a first activity we are bringing back the dynamic range measurements. This article is intended to describe the workflow and also philosophy of our test and should serve as a reference for those readers who want to engage / contribute to our method and findings.

Dynamic Range Measurements – the Conundrum

To the best of the author’s knowledge there is no industry standard available for movie cameras, and even the most simplistic definitions are vague as you will see quickly (there is an ISO 15739:2003 standard for digital still picture cameras though from which we will borrow some of our terms and definitions).

That has left a lot of room for interpretation and also allows each camera manufacturer to publish numbers which are not always directly comparable between the cameras.

Because dynamic range of a camera is an important feature our target here is to establish an independent base reference benchmark which allows direct comparison between the various cameras on the market.

What is the Dynamic Range of a Camera?

The dynamic range of a camera can be simplistically defined as the ratio of the maximum and minimum luminance that a camera can capture.

Sounds good, but how to quantify the term “capture”?

The ability of a camera to capture highlight detail is quite obvious, but the problem starts in the shadow details – here, noise starts to kick in quickly and we are left with room for interpretation which kind of noise is still usable or not – at this point, let’s talk quickly about sensor noise and signal to noise ratio (SNR).

Noise is the random reported variation of luminance of pixels compared to their actual luminance value.

Therefore, below a certain base luminance the image may be so corrupted by noise that we cannot really say that detail in the shadows is captured accurately or that image detail is usable.

The ISO 15739:2003 standard is helpful here as it defines a signal to noise ratio of 1 as being the threshold value for dynamic range measurements. However experience shows that for movie cameras a signal to noise ratio of 2 (or a root mean square value of noise RMS = 0.5 = 1/SNR) is a more conservative approach leading to “usable” shadow detail.

Lucky for us the ARRI ALEXA factory dynamic range statement of 14 stops seems to correspond very well to the 0.5 = 1/ SNR equation as past CineD results showed, hence we will continue to use this threshold value to determine the dynamic range of different cameras.

The Testing Procedure

We are using a DSC labs Xyla 21 backlit transmissive chart in a completely dark room, which provides 21 stops of dynamic range.

This chart is filmed offcenter from the horizontal axis (to avoid lens flare, camera’s internal whitebalance set to 5600K) using the following methodology: 

At the highlight (left) end we hard clip the first two patches, then stop down until the second patch is on the cusp of clipping. 

Now we have 2 methods to identify the dynamic range:

a) a visual inspection of a waveform plot and the recorded patch image

b) IMATEST: we export frames from the video file and run it through the IMATEST software which calculates a numerical value for the dynamic range using advanced image analysis algorithms (see http://www.imatest.com/docs/stepchart/).

While transmissive tests are the simplest/most accurate/comparable tests of dynamic range, they also give us very little information regarding how the camera responds to colors and detail across that range. Hence, it is just one piece of information out of many other factors when making a judgement about a camera.

Visual Inspection and the IMATEST Software Results

As the first patch is clipped, but the second is still within range, we start to count visually from this second patch downwards, hence from patch two to three is your first stop, from 3 to 4 is your second stop, and so on. So far so good.

At the shadow end, a visual inspection is more difficult as we have to count until the last discernable stop sticking out from the noise floor.

See figure 2 below from the Sony FS7 (SLOG3 10bit internal, values scaled to 8bit), which I tested again among other cameras to ensure consistency with past measurements.

The noise is (visually) indicated by the red line, and if we count all clearly discernible stops from the noise floor we get about 12 stops, maybe a bit more.

Now there is of course some ambiguity in the visual method (where exactly is the noise floor?), and here is where the IMATEST software comes in very handy.

Our Workflow using IMATEST:

Using the FFMPEG library we extract the highest quality I-frames directly from the video file, using this command (thereby avoiding any influence of an editing system): 

ffmpeg -i videofile.mov -vf „select=eq(pict_type\,I)“ -vsync vfr framegrab%04d.tiff 

Those high quality tiff framegrabs are imported into IMATEST, and the region of interest is selected:

IMATEST will then analyse each patch via advanced algorithms to calculate signal to noise ratios and then spits out the following chart (among a lot of other stuff) –  the dynamic range for various SNR values:

For the Sony FS7 IMATEST spits out a dynamic range value of 12.1 stops for 1/SNR = 0.5, and 13 stops for a SNR value of 1. Also, the software identified 17.7 discernible patches.

Hence, the CineD dynamic range value for the Sony FS7 is 12.1 in UHD (3840×2160).

Image Downscaling

One word about image scaling: if you scale the Sony FS7 image from UHD to full HD (1920×1080), you get an increased dynamic range in IMATEST of about 0.3 stops in the Sony FS7 example (resulting in 12.4 stops), as the downscaling averages 4 UHD pixels into 1 FHD pixel thus lowering the noise. Another argument for 4k acquisition and 2k delivery.

What happens by downscaling and how is the dynamic range increased?

Well, the signal value of four neighboring pixels has a high degree of correlation (luminance will be roughly the same), whereas noise for those pixels in general should show no correlation as it is random. 

Now, by downscaling (or averaging) four pixels of an 3840×2160 image to 1 pixel of an 1920×1080 image the signal value should stay very much the same, but the noise value will reduce because noise is random (uncorrelated), hence averaging (or in mathematical terms the root mean square value of noise which is measured in the recorded image) scales with the inverse square root of the number of samples that we average.

Hence 4 pixels into 1 gives a factor of two (sqrt(4)=2) higher signal to noise (SNR) ratio. Higher signal to noise ratio for a given luminance = higher dynamic range.  

Scaling for our testing is also done using the ffmpeg library (libswscale) to avoid any influence of the editing software.

The nice part about IMATEST is the fact, that all analyses are based on pure mathematics and not subjective opinions. Also, as long as 1 patch is clearly clipped you can underexpose / overexpose the stepchart and IMATEST will still give you the same result within a few percent difference. I did a lot of tests recently with the software and retested the Sony FS7, the Sony A7sII, the Blackmagic Pocket Cinema Camera, the GH5, GH5s and the FUJI X-T2 / X-T3 to make sure I can reproduce all past results – a very interesting experience, as it gives a lot of insight on the capabilities of the different sensors, and also manufacturer’s claims.

So here is a table of those recent results for your convenience (the ARRI ALEXA values are from past testing, but reproduced here as a reference too):

Manufacturers will probably use the less strict ISO15739:2003 SNR = 1 definition (signal value equals noise) as it gives a higher dynamic range reading. As can be seen from the results above, for this definition in the highest native resolution the Panasonic GH5s would be a 12 stop camera, the Blackmagic Pocket Cinema Camera a 12.5 stop camera, etc… 

One more thing: in the beginning of the article, the definition of the dynamic range was given as the ratio of maximum to minimum luminance the camera can capture – IMATEST will also spit out a “patch range” value which is the maximum number of patches that IMATEST was able to discern (not shown in figure 5). Here, for example the GH5s value was about 14.7 detected patches, and the Sony FS7 value is 17.7 (see figure 4). 

Hence, whenever a dynamic range reading of a camera is stated be aware that it only makes sense if also a clearly stated noise threshold value for this dynamic range result is given.

Last but not least, never forget that also the objective results of the IMATEST software will just be a relative value. Some cameras with advanced codecs allow you to still retrieve image detail from the noise floor with advanced post processing, while others don’t. So don’t be discouraged by a better / or lower dynamic range value of this or that camera, at the end it is still the guy behind the camera who makes the film ;-)

As a summary, we hope that you will find our results helpful, and we are looking forward to include dynamic range measurements again for future camera reviews! What do you think? Let us know in the comments below.