Milos Korenko

4 February 2011

Eset’s creative interpretation of testing

In 2009, Symantec ran a full-page advertisement in PC Magazine explaining to readers that because Norton has the most virus definition updates (222 per day) they also have the best detection score of 97.9% compared to all others. We at AVAST liked the ad for a better reason: we had only 1 update per day and were second best after Norton in detection. Look at the advert snap shot.

However, driven by curiosity I looked up the actual test from AV-Comparatives.org and to my surprise learned that Norton has used a decent dose of creative-results interpretation by ignoring Avira and G-Data, both of which scored even better than Norton in detection!

I thought this level of creativity would be difficult to beat but I was wrong!

Our resellers pointed us to these comparison documents from Eset:

- Eset vs. avast! Pro Antivirus
- Eset vs. avast! Internet Security
- Eset vs. avast! Exchange

Each comparison starts with features overview and, since the features are based on “vendor Marketing Materials available online” and each brand is using different feature names, I might have some sympathy for some of the misinterpretations. But several of them are just screaming inaccuracy:

- Yes, avast! 4 Exchange Server is using heuristic detection…
- Yes, indeed avast! has a sample submission system (avast! CommunityIQ) ….
- Yes, avast! has a fast scan (Intelligent Scan) that increases the speed up to 80% …

Read the website. It’s there

But the real horror was to plow through the listed detection and performance indicators. Reading these, It must be “clear” to anyone that based on the cited tests of AV-Comparatives, AV-Test, and Virus Bulletin that avast! can’t measure up. But guess what – it is not the case at all. It is just a highly creative interpretation of facts.

For starters, all tests listed here to prove the superiority of Eset are dated 2008 and 2009. But the actual products compared (avast! Pro Antivirus and avast! Internet Security) were only launched in January 2010. To prove that Eset has better scanning speed, it argues (!) by listing a Windows 2008 Server Edition test from 2008!

Boot-time overhead is argued based on an AV-Comparatives test from October 2008!

And it gets better. When you check the actual tests, you will find a False Positive score of 33 listed for avast! for all tests in 2009. But the real number in the same tests for Eset is not 9 (as listed), but 25. A typo?

And my favorite: the documents argue that Eset has superior heuristics detection as documented in AV-Comparatives test. But it fails to mention the compulsory requirement of AV-Comparatives to notice all who publish the results and explain about the limits of this test. In nutshell, the test does not measure the impact of behavior blockers, which limits the data value. As if this wasn’t enough, the cited tests from 2008 and 2009 correctly list the Eset score at 55% but the score for avast! is incorrect. A typo again?

So let’s look at the more relevant 2010 tests and see how we do vis-à-vis with Eset. It also needs to be said that the 2010 tests include avast! Free Antivirus and how it compares to all other paid-for products, including Eset Nod32 AntiVirus.

Detection:
Aug 2010: On-Demand Test AV Comparatives.org
avast! Free Antivirus 5.0 99.3%
Nod32 Antivirus 4.2 98.6%

Dec 2010: VB100 Comparative Review on W7 (On-Demand / On-Access)
avast! Free Antivirus 5.0 97.7% / 98.2%
Nod32 Antivirus 4.2 97.1% / 97.5%

Heuristic Detection:
Since the heuristic test has its specifics as described above, we could look at the Proactive detection tested by Virus Bulletin (heuristic capability of the application = detection of new malware with an outdated virus-definition database).

Dec 2010: VB100 Comparative Review on W7 (Proactive Detection)
avast! Free Antivirus 5.0 80.65%
Nod32 Antivirus 4.2 81.29%

Speed:
Aug 2010: AV Comparatives.org
avast! Free Antivirus 5.0 17 MB/s
Nod32 Antivirus 4.2 10 MB/s

Dec 2010: VB100 Comparative Review on W7
avast! Free Antivirus 5.0 28 MB/s
Nod32 Antivirus 4.2 12 MB/s

To avoid any doubts, Eset is an excellent product and has an impressive record of achieving the best scores in various tests and awards. But in my opinion, because of its record and its reputation, it would be better to stop “creative interpretations” and use the facts as they are.

Security News