Perhaps THE definitive web site for objective lens and sensor measurement – and free! – DxOMark.com is the lab and consumer face of DxO. The parent company is a highly regarded software developer (Optics Pro) and industry consultant (DxO Analyzer), and this is what gives DxOMark access to so much data.
In a free-ranging and candid interview, we recently spoke with DxOMark.com’s manager, Hervé Macudzinski. Herewith, highlights from that interview (NB: we didn’t talk about the DxO One because that is a separate initiative by the parent company – but rest assured, dear planet5D readers, we’re working on that for you, too).
It began in December of 2014 with an email we sent to DxOMark.com’s general address. We wanted to make sure we properly understood two of their metrics: their Low-Light ISO score, and their perceptual megapixel (P-Mpix) rating.
Because if we did, some sacred cows were about to be slaughtered.
Like how much sharpness you really need when shooting video.
Now it was mid-July, and there on the other end of a Skype video call sat Hervé Macudzinski, DxOMark’s manager.
Much had changed since we’d first contacted Hervé.
Panasonic and Sony – especially Sony – had continued to chip away at industry leader Canon’s grasp of the video DSLR space. Canon had, with their 5Ds [B&H | Amazon] and 5Ds R [B&H | Amazon] twins, answered a different question than most people had been asking. Meanwhile, Sony had launched an anti-tank missile at the heart of Canon’s hegemony with its a7 Mk II [B&H | Amazon] with 5-axis in-body image stabilization and purportedly superior autofocus, making it feasible – in theory — to finally, practically, use Canon glass in autofocus mode on Sony bodies.
Was it time to switch to Sony? Was it a good idea to stay with Canon glass? How much low-light performance did we really need or want (that a7s [B&H | Amazon] still beckoned many of us with its king-of-the-hill low light performance)? Did we even need to go a7 series? What about the a6000[B&H | Amazon]? Was Canon glass all that, or had the Sony/Zeiss partnership created a viable alternative lens set?
So that became one of the very first questions we asked.
planet5D: Why haven’t you tested any Blackmagic cameras? In fact, why – other than the RED Epic Dragon, haven’t you tested ANY pure video cameras?
Hervé: It’s a good question. It helps to understand that our measurements are based on RAW sensor numbers. In fact, we tried to assess Blackmagic RAW files, but it was very difficult. They are not real RAW files like what you find on a stills camera. The only video camera we could measure was the RED EPIC Dragon, but it was a prototype.
We will try to do more on video cameras, but we don’t know when. It’s exotic for our roadmap, which is more oriented toward photography – and it’s very difficult to put something new into this roadmap. But with this said, Blackmagic is a very interesting proposition.
planet5D: How do you select which cameras and lenses to test? Why, for example, haven’t you tested the Voigtländer Nokton series of lenses, a favorite among micro-four-thirds users?
Hervé: It is a function of what we have in the lab; our existing priorities (we are working hard on mobile); what is interesting to the industry; and what is available. I would say about 30% of the equipment we test comes directly from manufacturers; another 20% comes from rental houses; and the balance – about half – comes from our own purchases.
planet5D: What about the idea of testing the same lens across different camera body brands? We now have the ability to mount Canon glass on Sony or MFT bodies because of things like the Metabones Speedbooster and adapter [B&H | Amazon]. Wouldn’t it be valuable to see how a Canon lens when attached to a Metabones adapter – say their 100mm f/2.8L Macro [B&H | Amazon]– compares to the new Sony 90mm f/2.8 Macro G [B&H | Amazon]?
Hervé: We don’t do exotic combinations like a Canon lens on Sony because the adapter doesn’t do autofocus well and sometimes you have to worry about other things like sharpness or flare. There is uncertainty there. [Author's note: Michael Reichmann of Luminous Landscape does not share this opinion].
But there is internal discussion about this very idea, specifically of Canon and Nikon lenses on Sony. We haven’t moved on this yet because it means a lot of additional work, but yes, this is one topic in the backlog. Sony has pretty good success with their latest cameras. Everybody’s asking, “I have Nikon or Canon lenses, but I don’t know how they’ll perform.”
Right now we’re doing a lot of work on the Canon 5Ds and 5Ds R. It is our first priority in this segment. Very interesting to see how the older lenses work on this new sensor. Very small pixels.
We also get a lot of requests about the Sigma Art series. We performed a test for the Sigma Art 50mm in Canon mount. We didn’t have time to test on Nikon. A lot of people are getting annoyed about that. We will have to solve this over the summer.
planet5D: How long does it take you to test gear?
Hervé: It takes a lot of time. One lens can take anywhere from one day to two weeks. We make at least 50 shots per camera for each lens and at least one focal length/aperture combination. We use at least three cameras.
planet5D: So let’s talk a bit more about that testing.
planet5D: The way we read your low-light ISO score is based on what we know of ISO and what we understand from reading your notes on the site: a doubling of your low-light ISO is the equivalent of one additional stop of light sensitivity. Is that right?
planet5D: So if the Canon 7D Mk II 7D Mark II [B&H | Amazon] received a Low-Light ISO Score from DxOMark of 1082 ISO (which it did), then on a purely numerical basis, this is 26% and 30% better, respectively, than the Panasonic GH4 [B&H | Amazon] and Olympus OM-D EM-1 [B&H | Amazon]. But I would have interpreted your results to mean that the 7D2 is perhaps 1/4 – 1/3 of a stop more sensitive than these two, which I would not regard as significant. Is that right?
Hervé: Yes and no. Yes, the Canon 7D Mk II is up to 1/3 stop more sensitive than the Panasonic and the Olympus. But especially in video, I would say that 1/3 stop in low light is significant. One full stop clearly is significant. I cannot share the details with you at the moment, but I can tell you that we have conducted a video test comparison of two cameras of these types, and the differences were dramatic.
planet5D: May I put in my vote to share that with us sooner rather than later?
planet5D: Let’s talk perceptual megapixels.
planet5D: This is the big one. Your perceptual megapixel rating means we don’t have to stare at modulation transfer function (MTF) charts and continue to be flummoxed by them. But how does the P-Mpix rating you use for lenses apply to video? You write, for example:
P-Mpix is the unit of a sharpness measurement. The number of P-Mpix of a camera/lens combination is equal to the pixel count of a sensor that would give the same sharpness if tested with a perfect theoretical optics, as the camera/lens combination under test.
For example, if a camera with a sensor of 24Mpix when used with a given lens has a P-Mpix of 18MPix, it means that somewhere in the optical system 6Mpix are lost, in the sense that as an observer you will not perceive the additional sharpness that these 6Mpix should have added to the photos if everything was perfect.
In other words it indicates the ability of the lens and other optical components of a camera to utilize, from a visual perspective, the number of pixels of the camera sensor. P-MPix expresses the result using a figure that can easily be compared to the camera sensor’s MPix figure to show the quality of the lens.
Does this mean in your example above that the 25% of sharpness lost from this lens/camera combination would translate linearly? In other words, if one had a 16mp camera body from the same sensor family, would it be reasonable to impute a rough perceptual megapixel score of 12?
Or, to put it more directly in terms of 4K, could we impute that the same lens, from the same sensor set to video, would have a perceptual megapixel score of approximately 9 P-Mpix? And if not, how SHOULD we think about this?
Put differently: will we see as dramatic a difference in lenses between the best and worst at 1080p or even 4K video footage as we do from the same sensor shooting 36mp stills? I’m guessing not.
Hervé: Very good question! We translate MTF into one number as P-Mpix. There is a visualization condition between these two numbers.
planet5D: A use case?
Hervé: Yes. You can assess lens quality for the specific visual condition. If you are looking at a smartphone screen at one meter, for example, vs. looking at a 4K TV, these are two different visualization conditions.
We use a very difficult visualization condition to discriminate sharpness.
So, you’re asking, will a very sharp lens show much difference in full HD and 4K? The lenses will certainly still maintain their rank ordering. But especially for full HD, while the ranking will be the same, the apparent difference a superior lens offers will be thinner. I cannot speak for DxOMark because I do not have the data, but from a personal point of view, I’m not sure you would notice much of a difference between a very good prime and a very good zoom at full HD. 4K will be more interesting – now you’re at 12 megapixels, and you’ll notice differences more easily.
planet5D: Are contrast and micro-contrast considered in your measurements of P-Mpix?
Hervé: Yes, although we don’t measure depth of field or quality of bokeh. And dynamic range is very difficult, with different companies using different definitions.
planet5D: Yes! I got very excited when I read that the Blackmagic URSA Mini 4.6K can generate 15 stops of dynamic range – that’s ARRI territory! But it would be great if I knew it was in fact the same definition, the same test. Another reason for you to test video!
Hervé: (more laughter)
planet5D: Switching gears: you don’t charge for any of this data. What is your business model?
Hervé: Well, we are part of DxO, and our role is to help consumers, journalists and professionals understand differences among gear based on rigorous process and definitions. DxOMark is 100% independent — the only ads on our site are for our own DxO products; there are no ads from other sources. Even so, more and more, we try to have a Chinese wall even within the company. We are the lab, and we provide data to photographers.
planet5D: Hervé, thanks so much!
Hervé: Hugh, thank you!
And there you have it.
What About Those Sacred Cows?
So what about those sacred cows I mentioned at the beginning of the article?
Well, it turns out I did have a pretty good understanding of DxOMark’s ratings – and those ratings helped cement my resolve to complete my transition from Canon to Sony.
That fantastic L glass I accumulated over the years? Made more sense to get out of them while I still could extract value (I’ve been busy on eBay) and move to much lighter and less expensive primes like the Sony 50mm f/1.8 OSS [B&H | Amazon] and the Sony/Zeiss 28mm f/2.0 [B&H | Amazon] which are actually sharper, quieter and faster. And – at least for now — I will keep the dainty little Canon EF-S 10-18mm f/4.5 – 5.6 IS STM [B&H | Amazon]with Commlite adapter (the Zeiss Touit 12mm f/2.8 [B&H | Amazon]just isn’t that much better). I don’t mind going manual and hyperfocal!
Finally, what about my sense that the Sony a6000 [B&H | Amazon] is unbeatable not only as a hybrid video cam but also as a still camera at the price? Well, to each his own: but that little guy outpoints the Panasonic GH4 and Canon EOS 5d Mk III [B&H | Amazon] in ways that I have SEEN and are validated by DxOMark scores — while giving up less than one stop of low-light performance to the big Canon – at 1/5 the price.
Will I ever want more low light capability, 4K or a stonking Canon super-telephoto?
Though I do have to get my mitts on the just-announced Sony FS5 for a test drive, don't you think?
As always, your mileage may vary.
(cover photo credit: snap from DXOMark)
Additional Stories You Will Want To Read:
Hugh is the founder of Three Blind Men and An Elephant – a small production company with big animal logo – and the author of Apple’s iPhone: The Next Video Revolution. Follow him on Twitter (@hughbrownstone) or write to him at firstname.lastname@example.org.