Futuremark recently released the iOS version of their popular graphics benchmarking app 3DMark to go along with the Android version which has been available since April. Within an hour of the iPhone 5S launch its 3DMark Ice Storm benchmark scores were posted to Futuremark’s website. Like Ron Burgundy, this is kind of a big deal. For the first time people have the ability to compare Apples to Apples Androids using a single score. So did the iPhone 5S grab the #1 spot?
The answer is that the iPhone 5S is the fastest smartphone – sort of. To explain the ‘sort of’ we need to take a little bit of a deeper look at exactly how the 3DMark benchmark generates its scores.
Both the iOS and Android versions of 3DMark utilize Futuremark’s Ice Storm benchmark tests. When you launch the test you are treated to a roughly two minute CGI demo video of an imaginative but arbitrary
space ambush. My favorite part is the kamikaze buzzsaw. These are followed by two more animations focused on the graphics processor (GPU) and then a third animation based on physics formulas which targets the CPU. Once the tests are done running, the app gives you a single benchmark score as well as the scores for the individual components.
The Ice Storm benchmark test has three presets, Default, Extreme, and Unlimited. Default renders these videos in 720p and is for older / entry level phones. The Extreme preset ups the resolution to 1080p making the test more suitable for the latest high powered phones. These two tests are designed to give smartphones an overall score which can then be used to directly compare models. The iPhone 5S is one of only six devices in the world that was able to max out the scores for both the Default and Extreme presets. “Maxed Out”, in the simplest terms, means that the phone can render the video faster than the display can show them. The display screens in these phones are limited to showing 60 frames per second (fps) so even if the hardware can render the video at a higher fps, the device limits the output to 60 fps.
That is where the third preset comes in. In Unlimited, the same tests are run as before except the tests skip the post-processing step of actually displaying the video. While running the Unlimited preset, you aren’t treated to the actual video. Instead, a small series of thumbnail images are displayed on screen. Because the Unlimited test skips over various internal components of the phone, it is really only useful for comparing specific chip sets or components within the phone, but not the entire device as a whole. There are a lot more differentiators between the tests then just resolution. If you want to geek out on the nitty-gritty details, Futuremark has a technical guide posted on their site.
Now that we know what each of these preset scores mean we can explain the ‘sort of’. Like I said, only six devices in the world can max out both the Default and Extreme presets. Let us take a deeper look at these top six devices. Turns out one of them, the NVIDIA Shield, is not even a phone but is actually the mutuant offspring of an XBOX controller and a wi-fi only Android Tablet. There are three Xperia Z smartphones, with virtually identical innards, none of which are sold in North America (yet). And finally in the number one spot there is a Galaxy S4. But before the Android fanboys throw a parade, note that it is a rare variant of the S4 designed for LTE-Advanced and is also not available in North America and probably will not be anytime soon since LTE-Advanced has only been rolled out in South Korea and parts of Russia.
This leaves the iPhone 5S as the fastest smartphone that anyone in North America can run out and buy today – sort of. I wish I could get rid of the ‘sort of’ but I cannot just yet. If you click over to the scores for the Unlimited preset you can see the iPhone 5S right there in seventh place. Spots one through five are occupied by the red herrings and white elephants listed above. But in sixth place is the very real and very available LG G2 ($199 on contract at AT&T and Verizon, $99 plus $21/month for 2 years at T-Mobile – and 32GB to boot). Even though the LG G2 was not able to max out the Unlimited test, when the internal processors were untethered from the 60fps limit, they were able to spit out a higher overall Unlimited score than the iPhone 5S.
For the record, I am a total Excel geek so please forgive the charts. By looking at the component Graphics and Physics scores for the top seven Unlimited scores, we see that the Physics score for the iPhone 5S takes a big drop down from all the Android phones ahead of it. The Physics score is a test of the CPU so the dip is really no surprise. All the Android phones ahead of it use quad-core behemoths running at 2.3 Ghz. Apple’s newest chipset employs a dual-core CPU running at 1.3 Ghz. Hold on though, the Apple fanboys shouldn’t cancel their parade just yet. Futuremark is very clear in stating that the Unlimited preset is NOT designed to directly compare phones but should be used to compare specific chipsets, CPUs, GPUs, and APUs within the device. So without further ado and definitley no “sort of”, I can state that the iPhone 5S is the fastest smartphone currently available in North America based on its overall benchmark score.
While I wish the answer wasn’t so complicated, due to the complexity of the smartphone combined with the aspiration for thorough benchmarking, the answer will alwasy be unavoidably complicated. Throw in the monkey wrench of benchmarking two different systems and explenations only get worse. Benchmarking a computer used to be pretty simple. A benchmark program works by running a routine of various computing tasks while measuring how quickly the computer performs them.
In the early years of computing benchmark tests were done by throwing insane mathematical tasks at the CPU. The amount of time it took to solve the equation would then be used as the benchmark. However, today’s devices are far more complex so the speed which your phone can calculate π out to 32,000 digits is not really a good benchmark. But it is still loads of geeky fun. Check out SuperPi for Android or Calculate Pi for iOS.
The challenge that Futuremark overcame with the creation of 3DMark for iOS and Android was to ensure a fair benchmark score while accounting for the hundreds of little hard-coded differences between operating systems. A tricky task to say the least. Luckily the coders at Futuremark eat tricky for breakfast. Using a variety of open protocols that are platform independent combined with over a decade of experience creating industry leading benchmark apps, they knew what pitfalls to avoid to ensure that no one hardware design or operating system would have an unfair advantage over another.
The next question is if having a freely available reliable cross-platform benchmark will affect the market. The iPhone 5S shows that when Apple releases a new product that it is right on the edge of leading technologies. However, that edge moves pretty fast and Apple does not budge on price until the next model is released a year later.
Last week after 3DMark for iOS hit the App Store I did some data mining and came up with the following chart. It shows the scores for 3DMark Ice Storm Extreme for the iPhone 5 and the most popular Android phones released since the iPhone 5 hit the market in September of last year. Less than two months after the iPhone 5 launch, the LG Optimus G and LG Nexus 4 were launched and they benchmarked 48% and 79% faster than the iPhone 5 respectively. In fact every Android phone on the chart released since November of last year is significantly faster than the iPhone 5. Basically everyone who bought an iPhone after November 2012 overpaid for their phone. Now before you fire up your indignation editor for a forum flame please accept that for the purposes of this article I am referring to speed and speed alone. Not design or aesthetics or usability or personal taste. Those are qualitative factors which are beyond the scope of benchmarking.
As of now, the iPhone again reigns supreme. But within a few months the bleeding edge of technology will most likely pass it by. 3DMark can has given fresh loads of ammo to all the forum wars that are sure to come as the fanboys prepare to plant their flags and rattle their sabres, digitally speaking. But will people’s purchasing choice be swayed by a simple number. People have some pretty strong feelings about which smartphone system they choose and I am surely not suggesting that droves of people will abandon their beloved system of choice just because of that number. No, not droves, but maybe a few. And for the record my personal phone is an iPhone 5 which I purchased in March of last year.