2 February 2010 No Comment

Today I was scanning through various industry blogs when I stumbled upon an entry from a certain competitor.  What was interesting was that under the veil of improving Testing quality, the blog openly admitted that the organization in question had been trying to play tricks on competing organizations just to position itself favorably in the media.

They explained that they had deliberately created clean files and added fake detection in order to “show” that other vendors copied them.  This was a risky decision. Across the industry, research organizations share a level of trust and participate in sample sharing programs in order to enable the protection of customers – which for Trend Micro, is what always comes first. 

Aside from their prank, we were very pleased that the other resounding message that came from the blog post was that they finally understood and supported the message Trend Micro has been promoting for a long time – the need for a change in testing methodology to include real-world testing, such as is delivered by NSS Labs.

This need to evolve testing methodology was also a primary reason for the foundation of AMTSO, which aims to come up with more realistic and useful benchmarks.

This story really shows just how influential the media is to our industry.  That a respected vendor should manipulate detection rates just so that they can positively position themselves in the press, rather than focus on their customers.

But another, more positive learning, is also that the route of AMTSO is the right way to go. Pure detection rate by numbers, or one to one comparisons are yesterdays methods when verifying the value and performance of a security solution.

Customers need holistic reviews, giving them a real world scenario based feedback about what the different solutions can offer to them instead of pure “I detect more then you” headlines. I am glad to see that testing organizations like NSS Labs, AV-Comparatives or AV-TEST meanwhile understood and started to pickup these principles.

Post from: TrendLabs | Malware Blog - by Trend Micro

On the Trustworthiness of the AV Industry and AV Tests