"The TOP500 list has now exceeded its usefulness and actually is causing behavior and decisions that are not helpful to the high-performance computing environment and our users," Kramer said last November.
Other lists, such as the Green500, measure supercomputer performance as a measure of power efficiency, trying to highlight real-world systems that don't break the bank. An assessment of the industry by The New York Times last year found that the world's data centers consume about 30 billion watts, or about the capacity of 10 nuclear reactors.
Others have tried to develop a new benchmark for measuring performance altogether. A new high performance conjugate gradient (HPCG) benchmark published jointly by Michael A. Heroux of Sandia National Laboratories in New Mexico and Jack Dongarra of the U. of Tennessee will try to push past the Top500's Linpack benchmark with a better metric that stresses the system.
According to a white paper seen by PCWorld, the Linpack benchmark only really stresses the accelerator chips, not the general CPUs. (More details on the Tianhe-2, plus images used by this story, are available in a report by Dongarra.)
"In fact, we have reached a point where designing a system for good [Linpack] performance can actually lead to design choices that are wrong for the real application mix, or add unnecessary components or complexity to the system," the paper argues.
For the time being, however, Intel's Hadra argued that Linpack was necessary
"Our view is that some ability to talk about how systems perform and relate to each other in terms of capabilitiy has been important for technical reasons and will be important in the future, in spite of what could be perceived as obvious, shortcomings of a single number, such as Linpack," Hadra said. "It is the thing we have today, and is relevant today, and needs to be looked at going forward so that is consistent with the original vision" of its creators, he said.
Sign up for CIO Asia eNewsletters.