Sony FX9 Lab Test Part 1 – Rolling Shutter, Dynamic Range and Latitude

Lab Test

by Gunther MachuSeptember 9^th, 2020

When the Sony FX9 arrived at the CineD headquarters earlier this year, we were eager to run it through our lab test procedures. But when we reviewed the first results from the lab test, we reached out to selected industry experts. Read on why…

EDIT: Surprisingly and earlier than expected, Sony has just released a new firmware update for the Sony FX9. We will evaluate how the new update affects the results of our tests and if needed we will repeat them.

You might have wondered why we have not published a lab test of the new Sony FX9 yet, although my colleague Nino already bought one in January 2020. Well, we did the lab test as early as January – but the results using the internal codec made us wonder what is going on here.

Was the firmware/image not final yet? Would there be an update soon? So we reached out to Sony and some selected industry experts and shared all our findings and measurements with them. The initial replies and subsequent conversations were not leading to further insights. Sony stated that they have their established ways to measure sensor performance and if CineD does their own lab tests then fine, it is what it is.

In addition, nothing hinted at a new firmware or any kind of image improvements arriving anytime soon – and we learned in April already that a the 2.0 FW release would happen only later this year in October, after originally being announced for “summer 2020” when the camera was released last year.

(Sony recently announced another 3.0 firmware update without giving a release date, with a few more additional features: S700PTP Remote Protocol, Center Scan mode for Super 16mm lenses, and B4 Lens support when using the adaptor).

So after a careful review again (and again …) we decided to publish what we found earlier this year.

Rolling Shutter of the Sony FX9 – full-frame UHD 16:9

Using our strobe light at 300Hz, we get the typical sequence of black and white bars (due to the readout nature of CMOS sensors). Counting the bars gives a result of 22.2ms for the Sony FX9, see the figure below.

This value is a tad worse than the readings we got for the SIGMA fp (20.8ms) for example but is in the middle field for a full frame sensor reading a 6K image. The Panasonic S1H for example has 24.2ms in full-frame 4K DCI 17:9 mode (7% less picture height than 16:9 UHD mode).

However, comparing that to the new Canon C500 Mark II which shows a value of 15.8ms (in DCI 17:9 mode) the FX9 clearly falls behind – and now the new Canon R5 showed even 15.5ms in 17:9 8K full frame readout – the best result we have ever measured for a full-frame sensor in 17:9!

Now we know why the FX9 cannot do 60 fps in UHD full-sensor readout – there is simply not enough time to read the entire sensor. By the way, this explains why there will be 5K crop modes for 60fps in the upcoming new firmware (recording in 4K).

Dynamic Range of the Sony FX9 in XAVC UHD ISO800

If you are not familiar with how we test dynamic range please have a look here.

The Sony FX9 has a dual ISO sensor with 800 and 4000 as the two native ISO’s. Hence we first test ISO800 – here is the waveform plot of the XYLA21 stepchart – it shows 12 stops. A 13th stop and even a 14th stop are still visible, but inside the noise floor:

Running the files through IMATEST this result is confirmed:

This dynamic range result of 11.5 stops at our threshold value of a signal to noise ratio of 2 (SNR = 2) in UHD (3840×2160, internal noise reduction “off”) for a full-frame sensor which downsamples from 6K is OK but nothing special.

As you can see from past measurements, it is about one stop better than the full-frame mirrorless Sony a7S II or a7 III cameras show in UHD mode, but it is less than the full-frame Panasonic S1 and Panasonic S1H, the latter having 12.7 stops in 6K mode (see our article here). Also, the Canon C500 Mark II comes in at 13.1 stops (internal XF-AVC, full-frame) according to IMATEST (see the article here). Note that we have yet to receive a production model version of the Sony a7S III to perform our lab tests with it (but you can watch and read our real-world review here).

What is important to note: yes, while our noise threshold (at SNR = 2, Medium value (pink line)) kicks in already at 11.5 stops, there are clearly additional stops visible, have a look at the left-hand side of the middle graph (RMS noise) in the IMATEST result – see the figure below.

Given that the used codec can encode those additional (noisy) stops properly, you can work with those in postproduction using noise reduction, etc… If you fast forward to our latitude test (further down), you will see that the internal codec is not capable to do so properly. However, future firmware RAW options should definitely allow for that.

Cranking up internal noise reduction to “high” by the way leads to 11.9 stops for the Sony FX9 at ISO800.

Now let’s have a look at the second native ISO, 4000.

Dynamic Range Sony FX9 at ISO4000

At ISO 4000 we are getting exactly the same result as with the lower native ISO 800 – this is the first camera we have tested that fulfills the promise that the higher native ISO settings have no impact on the image. The Panasonic S1H for example shows a slight drop of dynamic range (0.4 stops) when switching to the higher native ISO circuit, as does the BMPCC6K.

The waveform plot of the stepchart at ISO 4000 confirms this – it looks strikingly similar to the waveform plot at ISO 800:

Also, the IMATEST analysis shows this result at ISO 4000, 11.5 stops are calculated for a signal to noise ratio of 2 (Medium value in upper right-hand table):

Latitude (underexposure test) of the Sony FX9

Of course, we also performed our standard underexposure test for the FX9: using S-Gamut3 / Slog3, noise reduction “off”, we exposed Nino’s face at max. 60% luma value on the waveform monitor, the white sheet of paper comes in at around 65%.

This gives a base exposure of 360° shutter at f4.0 at ISO 800 (UHD 25fps) – our 0 stops under baseline. From there we successively underexpose the studio scene by decreasing the shutter angle to 180°, 90°, 45°, 22.5° and finally 11.25° (5 stops underexposure).

Then, in post production (using DaVinci Resolve 16.1 studio) we push the underexposed images back to the 0 exposure value.

This test reveals a lot of insight about how well a camera (and the used codec) retains colors and detail. Most cameras break apart at 3 to 4 stops underexposure. The most recent exception so far: the Canon C300 Mark III which retains a usable image at 5 stops underexposure.

The aforementioned Canon C500 Mark II for example retains a usable image at 5 stops underexposure as well (but having faint lines in the image and using additional noise reduction in post).

So, let’s see how far we can push the Sony FX9 …

The above image shows the base exposure. Now let’s jump ahead and have a look at 3 stops underexposure, pushed back to zero:

Unfortunately, already at three stops underexposure, pushed back to zero we can see very blotchy patches of noise, especially in darker areas on the lower right-hand side, or on the dark side of Nino’s face.

Here is a noise reduction test using DaVinci Resolve 16.1, with 3 frames temporal noise at 5 threshold and 5 spatial:

Alright, with noise reduction the 3 stops underexposed image is still usable. Now, moving on to the 4 stops underexposed image:

As can be seen, the noise is corrupting the image to a point where it is not fully recoverable – I tried even stronger noise reduction again in DaVinci Resolve, see the resulting image below:

The nature of the noise shown in the 4 stops underexposed image is not finely dispersed, but represents rather blotchy patches of chroma noise – those cannot be easily removed in post without further sacrifices to the image.

Finally, just for reference here is the image at 5 stops underexposure, pushed back to zero:

It is difficult to tell why the image falls apart rather early, also compared to other cameras that we tested so far.

A potential reason for the poor performance could be the internal codec and how capable it is to capture noise. Although highest quality settings were used (and I reconfirmed them in post using Sony’s catalyst software, see the screenshot below) the all-I codec has only 250 Mbit/s – for an All-I codec in UHD this is a rather low bitrate and may be contributing to those large, nasty patches of chroma noise that we are seeing.

Summary

The results we got for rolling shutter and dynamic range are more in the middle to lower end of the spectrum for a full-frame sensor – to an extent that we reached out to Sony and selected industry experts. Those discussions took a while but didn’t reveal further insights – it is what it is as it seems.

In addition, the latitude test displayed above showed that you cannot go lower than 3 stops of underexposure and push back the image, otherwise ugly noise is corrupting the image to an extent that it is not recoverable – using the internal XAVC codec.

Of course we are curious to see if future firmware upgrades potentially improve the image quality. The next one is coming quite soon (in October), as mentioned above, so we will see – and this one should also allow 16-bit RAW output via the optional XDCA-FX9 unit (though we will have to see if at that point, there will also be a recorder available that is powerful enough to record 16-bit RAW). Also don’t forget that Sony already announced that it will release a little sibling to the FX9, the Sony FX6, until the end of 2020. No technical details were given yet though.

Are you using the Sony FX9 already? What is your experience using the camera? Please let us know in the comments below.