Security testing firm NSS Labs has publically defended itself against furious accusations by security firm FireEye that a cool assessment of the security vendor's breach detection technology published last week was based on a flawed methodology.
Testing security products is a complex undertaking riven with uncertainties about whether any assessment can possibly simulate real-world attacks, which doesn't, of course, stop security vendors quoting these results when they do well.
Less frequently, when a vendor does badly - or just not as well as its rivals - the fur can start to fly. And so it was when last week NSS Labs' Breach Detection Systems Comparative Analyst Report gave FireEye's Web MPS 4310 and Email MPS 5300 systems a lower rating on its Security Value Map (SVM) compared to equivalent products from SourceFire, Trend Micro, Fortinet and Fidelis.
NSS Labs' assessment could be described as relatively stinging, slapping FireEye's product (and one from South Korean firm AhnLab) with a 'caution' while the others received a' recommended'. Anyone who believes that nobody reads these reports, or that they have little effect, might want to ponder the effect on FireEye's share price, which dropped nearly 8 percent on 3 April (although tech stocks were hit anyway the next day).
This would be a troubling day for any security company but for a firm barely six months on from a high-profile and well-subscribed IPO, any bump is unpleasant. Wounded, FireEye senior vice president Manish Gupta came out swinging, criticising the test methodology on a number of counts, in particular the selection of malware against which systems had been assessed, which he believed skewed FireEye's results down.
He also said the firm had "declined to participate in this test because we believe the NSS methodology is severely flawed," and that the "FireEye product they used was not even fully functional, leveraged an old version of our software and didn't have access to our threat intelligence."
It's a high-risk strategy for FireEye because it draws more attention to the results and risks the firm getting drawn into a verbal exchange that attracts even more rubber-neckers who don't understand the complex issues at hand. Sure enough, NSS Labs has today published its rebuttal of Gupta's claims.
In a post Don't Shoot the Messenger NSS Labs' Bob Walder denied that FireEye had not been a willing participant and said the firm's products were installed and configured by its engineers during 2013. Walder also rebutted Gupta's various claims over the testing methodology in some detail.
"In the grand scheme of things, FireEye's results were not that bad. The real issue here is that FireEye now has credible competition in the BDS [breach detection system] market place and the data from this NSS test shows it," wrote Walder.
Sign up for CIO Asia eNewsletters.