Can testing paid for by an AV company be trusted?
Recently two tests by what is described as the “respected independent testing company Dennis Technology Labs” have been released by Symantec. The first of these was a comparison of Norton 2009 AntiVirus and the beta of Microsoft Security Essentials (MSE)—it concluded that Norton was far superior. The second was a comparison of 10 AV products (including Avast Home Edition) to determine which was best at detecting 40 real-life infected websites—Norton Internet Security was the best. Needless to say, these reports and results are highly suspect.
Why do AV companies commission such tests? Usually because they cannot get the results they want from independently published tests. Why do “testing labs” perform such tests? Because they are paid to. Why don’t AV companies just do their own tests and publish the results? Because they know that reviewers and readers won’t believe the results.
Both of these reports have gotten a fair amount of coverage in the media and on blogs. Sometimes they are described as “commissioned by Symantec”. Sometimes they are referred to as “independent” with no mention of Symantec’s sponsorship. Interestingly, Symantec seems to prefer referring to the reports as “independent” with no mention of their sponsorship. For example, take a look at the first page of the transcript from their quarterly earnings call with financial analysts: http://seekingalpha.com/article/169699-symantec-corp-f2q10-qtr-end-10-02-09-earnings-call-transcript?source=yahoo
I was intrigued as I had never heard of “Dennis Technology Labs” (DTL) even though in press releases they were described as “independent” and “respected”. Now, I have heard of “West Coast Labs”, “AV-Comparatives”, and many other respected and independent labs. But I had never heard of Dennis Technology Labs. So I googled them a few weeks ago…..and found nothing (if you google them now you will find a lot of references to these two reports but no information about the lab itself). Finally, I found them in the members list for the Anti-Malware Testing Standards Organization (http://www.amtso.org/members.html). AMTSO is an industry organization and we and most other security companies are members.
So, I clicked on their AMTSO link expecting to be taken to a website for Dennis Technology Labs. This is what I got:
So, the link for Dennis Technology Labs is the homepage for a UK publisher of Men’s magazines. The site’s rotating images show Maxim (with a half-naked woman), Monkey (a half-naked woman and two women kissing), Men’s Fitness (a half naked man this time), Auto Express (no naked men or women), and PC Pro (also no naked men or women). But, try as I might, I could find no information whatsoever about Dennis Technology Lab on this web site. Nor could I find any information about these two test reports.
Now I know that I am probably unfairly making fun of Dennis Technology Labs; they are probably associated with PC Pro and they are members of AMTSO. So they do have some credentials. But the point is that for-hire tests are not very useful—when is the last time you read a for-hire test that was not positive towards the company paying for the test?
One of our founders after having read this entry and being more adept at internet searches than me, thinks he has found the Lab: http://simonedwards.blogspot.com/2009/05/virus-lab-upgrade.html . I think he is right and the lab does exist. I do like the Norton logo that is prominently displayed with the picture of the lab director……
Reading through the 2nd report comparing the 10 security products, four items jump out:
- The report never says it was paid for by Symantec. Such a disclosure is usually a hallmark of a respected and independent lab
- The report never discloses how the 40 infected websites to be tested against were selected. Were they picked with prior knowledge that only Symantec detected all of them?
- The report never explains how 40 potentially cherry-picked websites are representative of the tens of thousands of infected websites that exist.
- The report goes to extremes to downplay Symantec’s weaknesses—note the clarifying comments in the sections on firewalls and false positives
I ran this report past our virus lab people and they just laughed and said “Pay us, give us a little time and we will set up the respected and independent ‘Vince’s Technology Lab’ to get you whatever results you want.” I took them up on their offer—I promised them a bonus and told them I wanted a test where we got a 100% score and Norton got 0%. An hour later they had 40 javascript infections from today that we catch and Norton does not:
#01 #02 #03 #04 #05 #06 #07 #08 #09 #10
#11 #12 #13 #14 #15 #16 #17 #18 #19 #20
#21 #22 #23 #24 #25 #26 #27 #28 #29 #30
#31 #32 #33 #34 #35 #36 #37 #38 #39 #40
So, we have our new Version 5 about to be released and have decided we need to have it tested. Here are our options:
- Hire Dennis Test Lab to perform a test (this will be difficult as their website contains no information about how to hire them to do testing).
- Use “Vince’s Technology Lab” where we are guaranteed to have a great result
- Hire Playboy to test our product. This was the favorite of a lot of staff who argued that it was obvious there was a correlation between half-naked pictures and respected test labs
- Just wait and let the truly independent and trusted labs test our products and publish the results.
I vote for #4. What do you all think?
Hey check this site again, u goofed up big time http://www.amtso.org/members.html
@razer
What is the goof up?
“when is the last time you read a for-hire test that was not positive towards the company paying for the test?”
-> that is because the customer (the av vendor) holds the publishing rights, so if the outcome of a commissioned is negative, the av vendor decides to do not allow to publish it. -> due that, commissioned tests that get finally published always look more or less good. it does not mean that the testers give good test results on purpose to the av vendor which ordered the test.
@Dan
Not always Dan. I have run across labs where as a condition of hiring them, they have the right to release the results regardless of whether you, the customer, approves. To me those are the much better labs.
What about NOD32 “comparison”? http://www.eset.com/products/comparison/
The only one with Avast! is in the antivirus comparison.
Hi!
When Avast 5 non-beta is published?
Thank you.
New AV-comparatives came out for removal tests: http://www.av-comparatives.org/images/stories/test/removal/avc_removal_2009.pdf
This is the first one to have MSE put through its paces. You should take a look Vincent S.
@Jyri Väätäinen
Sometime in the second half of November. Check the comments in the previous post on the blog for more details.
@Liam
)
The results ESET uses on their comparison page are outdated (November 2008). (And they know why they don’t use newer results
@Liam
Liam, we have seen that one. And yse, OneCare/MSE is good at removing viruses. But if you look in the august report you will see that OneCare did not do well at detecting/stopping viruses. And MSE would have scored the same as OneCare in that test. So, we are better than MSE in detecting/stopping and MSE is better than us in cleaning up–according to AV-Comparatives.
VirusTotal is a big joke. They use flat file scanning. They do not run the samples, they do not download them. The fact that they posted VirusTotal logs to Prove their point, slows how clueless Avast really is.
@Noshow
O, Rly? We tried to demonstrate the point that you can construct any set you like and have the desired results, and… we succeeded; with crystal clear ‘methodology’ and easy results verification. If you think we tried to demonstrate something else, like some behaviour blocking, HIPS or whatnot, you’ve been mislead. But not by us.
AHAHAHAH…I’m spliting my sides with laughing. It’s simply ridicolous to pay someone to obtain the results you want…nowaday everyone is able to search in internet details about labs that made tests on a software, and then, it’s not a fair behaviour..especially by a “big name” as Symantec.
Anyway…your article, Vincent, is very very amusing
Your Javascript test is a joke.
You know – and your test lab people know that virustotal is just a file scan test. If you want to test an internet security suite – testing all the security, you have to use the product as it was intended – in other words, give the IPS, browser protection, real time protection and other non file scan technologies a chance to run. This is what Dennis Labs claims they did.
No, av-test and av-comparatives do not do that, nor do virus bulletin, isca or westcoast labs.
If you want to to attack Symantec or any competitor’s detection – do a real test and don’t post such an stupid bit of misdirection.
I’ve never heard of Dennis Technology Labs – however it does appear to be a new division of Dennis Communications from some research we have done at ID Theft Protect. Dennis Publishing produce several UK computer magazines (and other magazines in other industries) one of which is the popular PC Pro and the another is IT Pro to name a few. Dennis Tech Labs appears to be an integral part of Dennis publishers providing lab reports/reviews on everything in computing and technology. You’d think they would provide a reputable landing page, especially as they are AMTSO members. Personally I would recommend these guys for “independent” evaluation: Lorenzo Martignoni from : http://security.dsi.unimi.it/
Keep up the good work avast!. Looking forward to the release of v.5 later next month.
Julian
@dlevinson
I love misinterpretations. We did no Javascript test. It’s just a db query over our samples, and VirusTotal links are just for the cross verification we did not ‘make up’ the numbers. This proves just what we said – that it’s easy to construct the set to prove anything, in this number of samples. It does not matter what methodology you use, when you can twist your miniature sample set easily and that such tests are completely useless.
And ‘just a file test’ is the only reliable thing out there and the only which can be somewhat tested. Other than that, it’s usually mumbo jumbo over few samples, thus worthless. And – if the products have the engines incapable of detection of any part of the malware, does it matter to you or not? For me, it means that the product is inferior to one capable of doing that.
@dlevinson
Dlevinson, it would be nicer if you informed readers that you are a Symantec employee. This is exactly the kind of behavior we are complaining about. Second, if you would like to hide that fact, you should make postings to our blog using a proxy, anonymizer, or your home computer.
Third, and most important, you are missing the point. What the heck does detecting 40 bad websites have to do with anything? Especially when one has no idea how they were chosen. If one wants to do a test like this properly, you have to set up a bunch of products to crawl hundreds of thousands of websites (in synch) from different IP addresses. Then one has to see which product alerts on which website….and then determine if that was valid or not. This is a test that no one has been able to set up. We would love someone to do it. Unfortunately until that happens all we have to go with are the large file scan tests.
Testing how products react to 40 sites means absolutely nothing. And comparing a mature product to a beta (as in the Norton to MSE beta) also means nothing. The point of this blog entry was to point out how easy it is to setup a test to get the results you want.
And in actual practice, we do not detect such JS’s with “file scan technologies”. They are detected way in advance of the payloads being downloaded. I would bet that for the vast majority of these, Norton even with all you cite would not catch them. The use of VirusTotal is just convenient to display the results.
Now, since you are a Symatnec employee, how about informing our readers how this test was set up, how the infected websites were identified, whether it was known in advance that Norton would catch them, etc. And of course it might be nice to know why the “independent” tester has a Norton logo on his picture…..
Why not a picture with the safer web browser in Earth: Opera….?
Is Avast 5 getting out of beta tomorrow?
@Rafael
Because Vincent use Firefox.
Hello,
Here is a quote from the report that I am intrigued about :
“Additionally, in the online channel, Spyware Doctor Starter Edition has been selected as the only security product for the Google Pack. This relationship enables us to gain access to Google’s 165 million unique visitors.”
As I though that avast was also included in the Google Pack. Further more I understood that PC Tools had been acquired by Symantec, is that right ?? Doesn’t it violate an anti-Trust Law ?
Thanks
Al968
Hi Vincent,
Well, I think each AV Vendors or Company have their own business strategy.
Either they do tricky or based on fact things, but i think your article it would be attacked some people which specially mentioned in this blog.
Anyway back again to AV Comparatives, in this last Oct 09 as Liam mentioned that avast only got Advanced award and more sad if the others vendor like Microsoft Essential can hit Advanced plus which they are new player in this business. Iagree with Liam, why avast couldn’t hit same as they are new comer in this business?
But my hope for ALWIL Software could consistently maintain your quality of products and services to customer.
Good luck, and we are looking to hear about avast ver.5 later.
Regards,
Yanto Chiang
@Björn Lundahl
Opera browser is safer, he should use it since he works in a security company…
@Al968
Hello Al. I was actually thinking of offering a prize for anoyone who could what was wrong in that earnings transcript. You beat me to it. Originally Symantec with a Scanner only and PCTools with anti spyware were in google pack. Symantec acquired PCTools about 15 months ago. In the latest Google pack is PC Tools spyware and antiviurs PLUS Avast. Avast is currently offered in (and PC TOols is not offered in) non-english europe. So the statement they made to the analysts was wrong. I assume they did not purposefully lie but somehow got confused.
I do not see an antitrust issue even if only PCTools were offered around the world. Google Pack is just one route to market. And it is not even the most valuable.
@Yanto Chiang
Yanto, it is true we are not as good at cleanup. We have historically focused on finding, stopping, and killing malware. We have not been focused on removing every bit of their existence. We don’t leave behind anything that can execute or cause harm in any fashion. Is just that malware is very, very messy. We would rather focus our resources on finding and stopping malware. We think that is more important. Remember, this is a strange test. The other test is one that measures the value of the insecticide (anti virus) in killing bugs. This test measures how good the broom and dustpan are in sweeping up the dead bugs.
@Rafael
Bjorn, I like to use what most of our users use so I use FF and IE. Now, I am actually an old Mosaic guy…..then Netscape….etc.
Firefox is best, no Opera, thanks.
@Rafael
Maybe he should switch to Mac instead.
@Vincent Steckler
OK that is fine with me.
Hello Alwil guy.
I use Avast! on my windows 7 but i recently see test from ICSA Labs and only PC TOOLS,ESET,Norman,Microsoft and Webroot are certified.
https://www.icsalabs.com/press-release/icsa-labs-offers-first-anti-virus-certification-program-microsoft-windows-7
Then Avast not be effective in windows 7 as is in windows xp?
thanks.
@Alwil Lover
Hi Alwil Lover. Glad you use Avast. The ICSA certification has nothing to do with the Windows 7 Certification. W7 certification is from Microsoft and Avast is certified (there is another blog entry here on that). ICSA is apparently doing their own certification; I assume as a way of making more money.
@Yanto Chiang
Hi Yanto. I hear you on MSE and the AV-Comparatives. But they are not new. This part of it is all based on OneCare. As I had said, we are weaker in cleaning up but stronger in detecting. Everything is a series of compromises. I would think people would rather have an AV product that finds the threats.
Hi Vince,
I am not meaning to said that avast have a bad in performance, because i have using avast since i knew your products as well. And so far i really happy with your performance in terms of detection ability.
I think every one have a their strength and weakness.
So as long as we could always improve our selves and to know what is our vulnerability and fixed it.
Good work for you vince and ALWIL team
Well said, Vincent!
Well i have a key logger. I tryed to fin it with norton but it found nothing. so if i were to scan my computer with Avast do you think that it would find it?
Joseph, I would not base the performance of either product on the ability to catch a key logger. They are very hard to detect–Whether a key logger is bad depends on how you use it.
Avast is the best, it detected malwares and viruses that almost didn’t detect(except G-data of course). Im proud that im an avast user and will be an Avast user forever! (if possible)
GREAT JOB AVAST!!!!
Vince,
Another “independent” report has been published from http://www.av-comparatives.org/comparativesreviews/removal-tests. The report highlights eScan, Symantec and MSE as being the most effective at removing malware. Well – removing the FakeAV, Vundo, Rustock and ZBot(Zeus) samples and left over files that is. Not entirely sure whether this was a “paid independent” report, but alas another one featuring Symantec. The report highlighted “none of the AV products” were very good in malware removal or removal of leftovers. My only reservation here has to be the “sample size” which was only 10 samples. Little can be gleaned from this report, other than AV vendors need to work together to continue improving the detection and removal rates. Julian
Hi,
I am fairly new to the world of computers, but being a college student in my 30’s have had to learn how to use them, and am trying to understand the whole concept of AV, malware, spyware, viruses, which is which, what you can do to stop them, what I need to secure my computer, etc. For example, do I need a firewall, an anti-spyware, an antivirus, an virus remover, a worm detecter, a worm remover, HOW MANY THINGS DO I NEED TO INSTALL TO BE SAFE? I got attacked by a virus that was posing as an AV, only I freaked because I did not know what was happening. The only thing that saved me there was they were saying “personal information such as identity, passwords, bank account numbers are being stolen right now: for immediate help press remove right now” so I did, then it came up as saying “for 3 months pay $46.00, and even asked for my credit card #, after it JUST WARNED ME MY ACCOUNT INFO. WAS BEING STOLEN i am not that stupid! A young student I go to Penn State with (I’m 34, and didn’t have all this computer stuff in high school, and started college 3 years ago)told me to use Avast, his mom is a big time computer tech or something like that, even the computer repair guy with 26 years experience didn’t give me anything all that great to use, comodo, and avira, but I still got hit, and the laptop was new, I just bought it 2 months ago, had problems with my other laptop too that I bought 2 years ago, had norton, then symantec which the symantec is free courtesy of Penn State, and I ended up having to buy my 3rd laptop, I am in debt with student loans, and I think people think I am so naive and they can take advantage of me all the time or rip me off just because I am no computer genius, but I am learning alot, and I know human nature, how people in general (MOST) are greedy and all about money, so what is the best way to go since I can’t afford to spend any more on repairs, or buying another computer, I am trying very hard to protect this, it is 2 weeks old, and I am afraid to even use it! Any honest suggestions out there?
@razer So where is this big goof up…seems everything is as stated. Dennis Publishing and all…I love that Avast lets others do their testing…saves me in the long run to as I am not paying for a trumped up review as well!!! AVAST TILL I DIE!!!