The company stayed in BAPCo through 2011 when, in a much-publicized blowup, it quit and walked away, accusing the test of being cooked for Intel’s CPUs. Although they didn’t say why, Nvidia and VIA left BAPCo at the same time.
BAPCo has primarily been made up of PC OEMs, along with Intel and other companies. At one point, even Apple joined BAPCo, as well as media organizations.
Sysmark uses off-the-shelf applications such as Photoshop, Premiere, Word, and Excel. It tasks the apps with a workload and then measures only the response time to the task.
AMD’s problems haven’t always been the apps, but the workloads. When it quit in 2011, the company told me at the time that it just didn’t think Sysmark exploited the “future” of computing and didn’t test the GPU.
Unsurprisingly, five years later, AMD’s complaints are the same. In the company’s video, Hampton says: “There is an excessive amount of high CPU tasking being done (in SYSMark). That is, the benchmark is really only evaluating the CPU side of the system.”
Benchmarking vs. benchmarketing
Part of the problem is the politics behind benchmarking—the not-so-fine line when it might turn into "benchmarketing," when numbers and tests are cherry-picked to make one product look better than the other. In this case, AMD is likely telling the truth that BAPCo 2014 1.5 focuses mostly on pure CPU performance. But isn’t that what it’s supposed to do? Measure the CPU performance?
From AMD’s perspective, no. The company has long insisted the future is about GPU computing. And, well, no surprise, AMD has also long enjoyed a performance advantage over Intel’s CPUs in graphics performance.
In fact, one of the tests AMD uses to show it’s behind Intel, but not that far behind, is PCMark 8 Work Accelerated. The test has two options: One uses OpenCL, which taps the GPU, while the other relies on just the CPU.
This begs the question: What was the score on that same laptop if the GPU wasn’t factored into it? Is there a little benchmarketing going on there from AMD?
You’d also have to ask yourself, how many common work or office apps today heavily rely on OpenCL? Few to none, I’d guess.
What we run and why
As someone who has burned too many hours coaxing Sysmark to run on systems, I was glad to leave it behind. I didn’t have any proof it was cooked, but it took forever to install and forever to run. In those days, it would often bomb out, meaning you wasted yet another day.
The methodology seemed very solid, though. For example, rather than “type” a document at 1,000 wpm (which many Office suite tests did and still do), Sysmark found a way to “type” at realistic human speeds while measuring only the response time.
Sign up for CIO Asia eNewsletters.