My Quadrant is bigger than your Quadrant

9


2014 - My Quadrant Score (OCed 1.2Ghz, Virtual EXT4 partition as /data/data)


With the recent explosion of high-end Android devices hitting the market on all of the various carriers, it seems that the benchmark app Quadrant has become the ‘standard’ to test out and compare these new ‘superphones’, and to see how fast we can tweak them using mods, hacks, or other related improvements. Having just upgraded to a Samsung Vibrant myself, I too have found myself falling victim to this “benchmark fever”, relentlessly trying to squeeze every last benchmark point out of my already-speedy device, through the use of optimized kernels, overclocking, “lag fixes”, and other misc tweaks.

With my background in repairing and optimizing PCs for gaming and video editing, I knew the importance of a reliable and reproducible benchmark. Without one, you can optimize, tweak, and tune all you want, but in the end, you will have no idea if you had any impact on overall performance. Likewise, without some kind of baseline to compare against, or a “control group” for your scientific method enthusiasts, the ‘scores’ being produced are completely useless, as you have nothing to compare them against. In this case, Quadrant provides several baseline scores for several popular Android phones to compare your score against.

My question of Quadrant’s reliability first started when I applied a few of the “lag fixes” found in the XDA-Developers Forums. Prior to applying the mod, I was averaging ~900 in Quadrant Standard. After applying & testing a few of the “lag fixes”, I was averaging between 2100-2700. All of the “lag fixes” involved moving the app data and dalvik-cache away from Samsung’s slow internal Storage, which should only affect Input/Output scores in a overall system benchmark, but this test (see my score below) gave me over 2x the score, raising my score from ~900 to ~2100. For an general system benchmark, improving only one aspect of the system (I/O speed) should not raise the score exponentially, as it did. Within a few days, my finding were confirmed by a very reputable source, Cyanogen himself.3178 - Cyanogen's Quadrant Score

A recent tweet from Cyanogen included the Quadrant results to your right,and the following comment: ““Don’t rely on this stuff as your guide to how fast your phone is. There are a lot of other factors”. In a subsequent tweet, Cyanogen explains that he was able to ‘cheat’  the benchmark by mounting, “a [temporary file system] over quadrant’s data directory”, which may artificially increase the benchmark results, making them unreliable and unrealistic. Many of the “lag fixes” for the Samsung Galaxy-S (Vibrant and Captivate included) are based on a similar concept, which mounts a “virtual” EXT partition on top of Samsung’s proprietary RFS file system,which  buffers the data and gives the increase in data access speeds.

Take a look at Quadrant scores below, one from Cyanogen and one from my own testing using Quadrant Advanced. Taking at look at my score, you can see that Quadrant Advanced breaks down the score to the individual areas of testing, and that green, which is I/O speed, takes up a large portion of my score. If this were an “accurate” system benchmark, my score would have only improved slightly, as only one area of my phone was tweaked (Think “a chain is only as strong as its weakest link”, or the slowest component). Seeing that my I/O scores takes up almost two-thirds of my overall system score reinforces the notion that Quadrant isn’t a very reliable system benchmark. Hopefully, Aurora Software can improve on how Quadrant calculates its scores to make it more balanced and accurate.

You can download Quadrant from the Android Market by clicking or scanning the QRCode below:

About the Author

Jason Bowdach
Jason is a technology and media enthusiast. He loves the Android OS, although his interests go far beyond the cell phone platform. He enjoys keeping himself up-to-date on the film and cinematography industry, as well as the ever-changing world of personal computers and gaming. He studied Communications - Media Studies and Organizational Psychology at San Diego State University, and is pursuing a career within the broadcast media industry. In his free time, besides playing with the never-ending supply of custom Android builds or theming, he enjoys caring for his 60-gallon aquarium, training his white-golden retriever puppy, catching up on quality films and TV shows, and playing the occasional video game.

  • Linuxluver

    I get 1450-ish without any “lag fixes”, tweaks or tricks. Nexus One, CM6 RC3.

  • Ron

    Nice article, but when will the article come out that provides any alternative benchmarking apps?

  • http://twitter.com/Jbowdach Jason Bowdach

    At the moment, there are only 3 benchmarks for Android: Quadrant, Linpack, and Neocore. Those are the big 3, and Linpack is CPU only, and Neocore is GPU focused. To answer your question, we will write an article about an alternative benchmark app when it becomes available. At the moment, those are the only 3, and non being a great system-wide benchmark (there are a few misc ones on the market but those don’t count)

  • yhbae

    Quadrant has become very popular these days. There’s even a site that lists Quadrant results for just about all Android phones currently on the market. As expected, Samsung Galaxy S series is dominating on this list, even without any help from Froyo. I wonder if this “XDA fix” really returns any real-life performance benefits. Otherwise, Quadrant will need to retune their score formula.

    Here’s the site I mentioned above:

    http://smartphonebenchmarks.com/

    I personally own a Nexus One using Froyo. I get Quadrant score in line with what is shown on this site, but I have to wonder if Galaxy S is x3 faster at everyday operations Quadrant is suggesting.

  • nab

    theres also fps2d for some more gpu benchmarking

  • Guest

    The Quadrant score is deceptive for the SGS. As you can see from the colored images, most of the score is coming from I/O operations, the only time you would see a real time difference is when copying files, etc. However, the Memory/3D/2D beat out every other phone(but not by much) which you would see a difference in when dealing with graphics, or running memory intensive apps, or multiple apps. Once again, probably not a huge noticeable difference between some of the other phones like the Nexus One for example. However, running apps also relies on CPU power which as you can see the Nexus One destroys other phones. So all in all I would say Nexus One is definitely the fastest since it’s CPU & Memory scores are high, which is primarily what matters in most apps. SGS would probably beat Nexus One noticeably when running graphic intensive apps like 3D games.

  • RyanZA

    Hey — I think you have forgotten something very important in your rant. You take the huge increase from one source (CPU with JIT) and say it is fine. However, you say a different huge increase is cheating (Improved I/O Speed).

    It isn’t like JIT will improve your phone speed more than improved I/O will. For most people, the slow part of their phone is going to be opening new apps. That is completely I/O dependent. The quadrant scores are designed to reflect this. (Maybe a logarithmic scale would be better? Either way, the point still stands.)

    Improved I/O will affect every day tasks more than JIT will – JIT will only affect cpu intensive, java based (anything in NDK won’t be affected), loop intensive applications.

    And there are far more I/O intensive than loop/java intensive applications. As far as your chain goes, for phone use, that weakest link (especially especially on the SGS) is the I/O. 1ghz cpu is much more than required for displaying some emails.

    If your background really is in gaming benchmarks, then you should understand the importance of real world usage — buying a super CPU is going to have a huge effect in calculating PI, but it won’t help you in a gaming benchmark nearly as much as upgrading the real issue – the GPU.

  • Jason Bowdach

    RyanZA posted a comment that I thought I should address.

    Yes, an increase in I/O will generally speed up a phone more than a faster CPU, as IO is generally the slowest component, and real-life usage performance will improve if you can eliminate the bottleneck. Yes, I also make mention that the N1 gets a benefit from JIT which increases the CPU score, however it doesn’t exponentially increase the system score, as we can see with the I/O scores. Comparing that to the IO increase, which jumps the overall system score from ~800 to ~2100 (if using EXT2), you can see that the jump is significant. However, just because your benchmark says a score over 2x what you got before doesnt mean you phone is over 2x as fast. I agree most users benefit from increases in the IO speed more than of increases in graphics or cpu performance, but that doesn’t mean that the scores should be calculated as it is, because simply increasing IO speed doesnt even increase real-work system performance by ~250%, as the current Quadrant tests are saying. That is my only point in this article. I am not ranting, and not “bad mouthing” quadrant (as I still like the benchmark, especially the advanced version), just making a point about how its final system scores weren’t exactly accurate to real-life usage, which is true for a LOT of benchmarks. Although I wouldn’t go as far as to call Quadrant a “synthetic benchmark”, the scores it produces certainly doesn’t mirror real-life usage.

  • jack

    5081 using a htc one s stock

© 2010  AndroidSPIN. All rights reserved.
A Subsidiary of MobileTweek Inc.