Chinese antivirus firm Qihoo 360 Technology today was censured by three major testing organizations for cheating on the evaluations.
The three antivirus testing labs — AVComparatives, AVTEST and Virus Bulletin, of Austria, Germany and the U.K., respectively — stripped Qihoo of all awarded certifications and rankings for this year. They will also put in place controls to make sure that Qihoo or others cannot "game" the tests in the future.
Qihoo denied the charges, saying they were "without merit."
Qihoo 360 is headquartered in Beijing, and reported revenue of $1.4 billion in 2014. Although the company's security products are little used outside of China, inside the People's Republic (PRC) the company claimed almost 750 million people used its free mobile security app, 360 Mobile Safe, last year. Qihoo is also known for its 360 Browser, which relies on Microsoft's Trident rendering engine, the same that powers Internet Explorer (IE).
AVComparatives, AVTEST and Virus Bulletin concluded that Qihoo had provided a customized version of its security software to them — they received the software from the Chinese company because the programs are not widely available outside the PRC — that replaced the company's own antivirus engine with the one created by Romanian security firm BitDefender.
"After several requests for specific information on the use of thirdparty engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users," the three labs said in a joint statement (download PDF).
That skewed the results in Qihoo's favor. "According to all test data, this would provide a considerably lower level of protection and a higher likelihood of false positives," the labs said.
Prior to acknowledging the AV engine swap, Qihoo pointed fingers at two of its rivals, Baidu and Tencent, both also based in the PRC. Although Qihoo's allegations proved accurate, the testing organizations concluded that it could find no evidence that their actions — setting code flags marked with the names of several test labs, which in turn implied "some difference in product behavior" — gave them a significant advantage. "Both firms were able to provide good reasons for including these flags in their products," the labs added.
Qihoo's security products include similar flags.
"Users rely on independent results to make an educated decision regarding their protection software," argued Maik Morgenstern, AVTEST's CEO, in the statement. "If vendors start to manipulate the testing process, they are hurting everyone involved."
Qihoo has been criticized before for manipulating numbers. While it claims that the 360 Safe Browser has a majority share of China's market, analytics firms like Ireland's StatCounter say different: In March, Qihoo's browser accounted for just half a percentage point of user share, a measurement of how active each browser's users are on the Internet.
Sign up for CIO Asia eNewsletters.