Benchmark scores may not be a true reflection of how well a device runs in real life but they are nonetheless useful at comparing devices in a controlled environment. Unfortunately, benchmark scores have been called into question given recent findings that device manufacturers try to manipulate these with custom code and tricks like turning on all processor cores when benchmarking software is detected. Samsung, in particular, has been caught doing so both with the Galaxy S4 and the Galaxy Note 3. But it’s far from being alone in doing so. At least one benchmark software provider is now taking measures to try to stop the practice.
Futuremark has delisted from its results a number of HTC and Samsung devices over manipulation of its benchmarking tools. Among them are the HTC One, Samsung Galaxy Note 10.1 (2014 Edition) and Samsung Galaxy Note 3. It explains its decision like this:
People rely on Futuremark benchmarks to produce accurate and unbiased results. We safeguard that trust by having clear rules for hardware manufacturers and software developers that specify how a platform must interact with our benchmark software. In simple terms:
A platform must run the benchmark without modification as if it were any other application.
Futuremark will now delist any device found not to comply with its rules. Such devices will appear at the bottom of the 3DMark Device Channel and the Best Mobile Devices on the Futuremark website. They are clearly identified as being delisted and show no score. It has also implemented an appeal process by which a manufacturer can work with Futuremark to restore a device’s score. People can also report devices with suspect scores but will need to provide evidence that supports the claim.
Futuremark’s new policy appears to be a work in progress as other devices that have been found to game benchmark scores have yet to be delisted. Among them are the Samsung Galaxy S4 and the LG G2.
Kudos to Futuremark for taking this step in restoring the credibility of benchmarking.
Do you believe that this will push manufacturers to stop gaming benchmarking or will they simply try harder to hide their efforts? Let us know below.
Sources : Futuremark // The Register