avast! blog

After writing the previous entry on vendor-sponsored tests and seeing comments about it, I thought it might be good to explain what AV products do, how they typically gets tested, and how they should be tested. And, I will discuss a bit in a more thorough fashion what is wrong with the testing that was performed by Dennis Technology Lab. I had attempted in the previous posting to illustrate the uselessness of the DTL testing with humor—it did not work with some people (especially at least one Symantec person). So, here is a more straightforward approach.

I thought it would be good to start with a basic tutorial on what antivirus software does…….

The bad guys are trying to get a bad payload (malware) into the user’s computer. This malware/payload could be benign (such as current Conficker) or it could be designed to steal passwords, bank information, open a backdoor for a hacker, etc. The malware can also be a virus, spyware, Trojan horse, etc. It makes no difference in the detection and that is why all good AV/security products include anti-spyware capabilities. Regardless, the purpose of AV is to stop the bad guy from getting the malware into the computer. In the past these attempts were made almost entirely through emails—sometimes SPAM email and sometimes email received from a trusted source with an infected computer.

Currently we see that over 80% of infection routes are via internet browsing. Sometimes the infections come from know bad sites (often hacking sites, sex sites, etc.). But these days, mostly they come for a bad guy infecting a legitimate website. These infections are usually a java script that once downloaded and executed on the user’s computer installs a piece of malware.

Now, there are also other means to infect a user computer (such as over the network, a hack attack, etc.). But as the email and website vectors are the two most common, I will concentrate on them. First, here is a simple diagram showing what an AV product does to provide protection during these activities.

1. Black-listing known bad sites is always the first step in protection. When a site is black-listed, the security product will prevent the user from opening the website. Sometimes the user can override that block; sometimes they can’t. There are a lot of websites in the world whose main purpose is to infect users. Security products find these sites by crawling the web searching for them or by leveraging large installed bases to find them. As they sites may change (or get infected) faster than a crawler can re-visit, it is more effective to use the large installed base. But, most AV/security companies do not have a large enough installed base. This is an advantage of Avast with nearly 100 million users.
2. Even if a site is not on a black list, it can still be infected and harbor malware. This tends to be the most common source of infection these days. Typically these websites are legitimate—but a hacker has broken in and injected some code to infect visitors. It is very important to block these websites from loading. If they are allowed to load, the user can be infected. To block them from loading, one has to either recognize the site is infected (that is have it on a list) or detect the infection while the site is being downloaded but before it is actually in the user’s computer. Avast takes the latter approach and may be unique in that approach. Most try to detect these websites by crawlers similar to the black listing approach above.
3. If a bad site is allowed to load, the next chance is to stop the infected code from loading the payload. This is usually quite difficult and may require the usage of an outbound firewall, sophisticated heuristics, and even behavior detections. We prefer to focus on not allowing these sites to load. Other security companies let the sites load and then try to stop the infection.
4. Infections can also come in via email. The first line of defense against email infections is an anti-spam filter. Many attachments to spam are actually malware. Most security products do not include anti-spam; Avast will have the anti-spam in the Version 5 Internet Security Suite. Norton also just re-added Anti-Spam in their recent Norton 2010. Many ISPs and mail providers (yahoo, Google, etc.) do a reasonable job of blocking spam.
5. So, if all the attempts to keep the malware out of the computer have failed, the signature detection now comes into play. In the past, this was the only component of an AV product and sometimes it is mistakenly believed that this is all AV products do. A signature is basically a mathematical representation of a file. Each file that exists in the world has a unique mathematical representation. So, if we know about a piece of malware we then know its signature. For each file that comes into the computer, we compute the signature, check it against our database and decide if it is known to be malware. Nowadays, this technique is not sufficient as malware changes often—when it changes, its signature changes. Now, this is a very simplistic description of the process and there is actually a lot of complexity behind the scenes—especially in making the signatures generic so that one signature can catch multiple variations of a piece of malware.
6. Heuristics come into play to try and catch malware that we do not yet have a signature for. Heuristics is basically having the AV product do some reasoning about an executable to decide if it is suspicious. There are usually various sensitivity levels of heuristics. The downside is that the more sensitive you make heuristics, the more false positives you get. Norton has added an interesting angle to heuristics in 2010 by factoring in the program usage by the community—if it is used a lot it is probably good. It remains to be seen how well this helps detections but it does come at a price to privacy—Symantec now knows all the executables you load onto your computer.
7. If the AV product has now failed to stop the malware from being downloaded and executed, the next layer of protection is to stop it during execution via behavior blocking. With behavior blocking, the AV product watches what a program does and if it exhibits the behavior of malware (such as making system modifications), it blocks and sometimes kills the malware. This is a very difficult process as some malware is quite smart and can hide itself—some can even disable the antivirus. If the behavior blocking can recognize bad behavior and shut down the malware quickly, you can avoid infection. But, if the malware is too fast or slick for the behavior detection, your computer will get infected and all the AV product can now do is to limit the damage.
8. A new technique in AV products is to execute the browser and other applications within a “sandbox”. This is just like kids playing in a sandbox—everything stays in the sandbox. Basically a sandbox pretends to be a computer. If malware gets executed, it only infects the pretend computer and not the real computer. Thus even if malware makes it past the previous seven protection layers, the sandbox will prevent the malware from causing any damage. To fix the problem, just restart the sandbox. Sandboxes have been around for a number of years but have not been incorporated into AV products until this year. I believe only Avast and Kaspersky have this feature. Avast’s is in the new Version 5 Pro Antivirus and Internet Security Suite. Also, the Avast sandbox works in 32-bit and 64-bit mode while the Kaspersky sandbox only works in 32-bit mode.
9. An outbound firewall can also limit the damage caused by a piece of executing malware. It does this by preventing the malware from communicating to its home/host over the internet. If successful, the firewall can limit or even prevent the damage (depending on the intent of the malware). Outbound firewalls are famously delicate and difficult to manage because you have to define rules for each application that you want to have access to the internet. The newer versions (including Norton 2010 and Avast Version 5 Suite) eliminate this complexity and make the firewall useable by anyone. However, the malware can also outsmart a firewall by piggy backing on other applications that are allowed to use the internet (such as your browser). Thus, it is dangerous to just rely on this last layer of protection—you need the other eight to really be safe.
10. And of course, the last step is to clean up the damage. If malware gets caught at the very beginning, there is nothing to clean up. The further into the computer the malware penetrates, the more there is to clean up. The first step in this is to actually remove the malware. With Avast this can be done with the boot-time scan. In this scan, we reboot the computer and detect viruses before the operating system loads. This allows us to detect viruses that would otherwise hide themselves.

So, it is clear from this that it is best to stop the malware at the outer perimeter—that is the objective of most top antivirus products these days. There is also a lot of hype in the marketplace that AV products—especially free AV products—do not provide these modern capabilities. As I had explained in an earlier posting, they do. The free products (especially avast of course) are even more sophisticated than some paid products. Avast Free Antivirus, V5 incorporates all but anti-spam, firewall, and sandbox. I would bet this is more protection than provided by the paid Antivirus of Norton, McAfee, Kaspersky, or TrendMicro.

Now, there are also some other interesting angles I will talk about in another posting: how cloud computing impacts this, what is the role of the community, how virus data is shared, etc.

So, How do AV Products Get Tested and What is this Controversy Over the Dennis Technology Labs Testing?

While it is clear that it is best to stop malware before it even gets into a computer, that is not what magazines and test labs test. What all the test labs actually test is basically steps 5 and 6—the signature and heuristics detections. Plus, some test the clean up in step 10. They do these tests by literally running millions of known bad and good samples through the competing AV products. While this does not tell you how well an AV product blocks an infection early, it does tell you how well the AV product protects you after an infection has made its way into your system. It also gives you good insight into how effective is the vendor’s ability to collect and process virus samples. And this ability to collect and process samples is crucial. Too many people focus on how many signature updates a vendor puts out every day. The real question should be from how many sensors the vendor is actually collecting samples and how quickly are they processed and turned into signature updates.

To test at the outer perimeter—before an infection gets into your system—is extremely difficult. To properly test one has to:

1. Set up the competing products on different IP addresses (because the host sites often will only serve malware to a single computer on an IP address)
2. Use IP addresses not associated with testing labs (as some malware will not deliver itself to lab IP addresses)
3. Have these computers simultaneously crawl million of websites
4. Then you need to take the results and validate them—there will be false positives and lots of manual analysis. For example, some malware will only deliver itself once. Thus one computer may find it and the rest won’t. Some malware will not deliver itself to certain countries.
5. Figure out how to constantly clean and restart computers that get infected during the crawling

While it would be very worthwhile to do this type of testing, no lab has yet figured how to do this for other than a very small set of suspect sites.

That is exactly what Dennis Technology Labs did—they chose 40 sites. In the last week we detected about 50,000 hijacked websites around the world. It is impossible for 40 sites to be a fair sample of 50,000 diverse sites—the sample size is ludicrous. It is blatantly obvious that one cannot draw any conclusions about overall product abilities and performance when the test is based on so few samples. Symantec paid for the test; hopefully DTL conducted it in a fair manner. But DTL did not make any attempt to explain how they made sure that there was no prior knowledge about Norton’s ability to detect the small sample size of infected websites. Even if there was not prior knowledge, the test is interesting and nothing more. It is no better than the hundreds of YouTube video tests where “experts” test antivirus products against infections and infected sites and pronounce who is best. In both cases, the sample size is just too small to make any conclusions whatsoever.

To illustrate this point, I engaged Vince’s Technology Lab once again. Starting with the list of 50,000 infected websites we know about, I tested Avast Free Antivirus, Version 5 against the latest Norton 360. Additionally, I have PCTools’ AntiSpyware with AntiVirus also running in parallel with Avast 5. (I periodically check out competitor programs by running them alongside Avast. Right now I happen to be doing that with PCTools and Norton Security Scan).

So to clarify, one computer has Avast and PCTools (also Norton Security Scan but that has no real time scanning so its presence is immaterial).  The other had Norton 360. Here are the results (I have deliberately inserted blanks into the website links to make them non-clickable to protect the non-avast users that may be tempted to click on them):

1. http://7 7.fskn.gov.ru/ This is a legitimate Russian website. In fact it is a government website. I chose this website because I know that Symantec/Norton is weak in Russia while Avast has 3.5 million users. So I suspected that Norton 360 would not detect the infection. I was right. Avast alerted to an infection and aborted the connection. Norton 360 did nothing. PCTools also did nothing.
2. http://kings field.edu.sg/ This is a Singapore education firm. I chose it because I used to live in Singapore. Like with the Russian website, Avast alerted to an infection and aborted the connection. Norton 360 did nothing.

So, we have now destroyed the “fact” (or myth) that Dennis Technology Labs says Symantec earns a 100% score in detecting such infections. Statistically my sample size of 2 is probably just as valid as DTL’s sample size of 40. I have a 100% score for Avast and a 0% for Norton. And yes, I know this is ludicrous….the samples are cherry-picked and the sample size is way too small. But, that is exactly my point—this test is as invalid as the one from DTL.

As a caveat, it is possible that the N360 machine did not have current definitions as I had not used it in about a week. But, I do know that these two sites have been infected for well over a month and thus Norton should have been aware.

So, I decide to proceed and check another site; an American site this time. But, just as I start typing the address into the browser, my N360 starts alerting that its antispyware and AV capabilities have been shutdown…..presumably by the infections loaded by one or both of those two dirty websites I had browsed to a minute earlier. So rather than continue the test I must now shutdown the N360 machine and clean it with Avast…..

Now for the serious message if you get this far.  No doubt about it, Norton is a good product.  I would bet it is the best paid product from a traditional vendor on the market these days.  Avast is also a good product.  I personally think it is better than Norton (and it is free). I think its ability to detect infected websites is far superior and that is what is most important these days. I would stack Avast up against Norton in a real test any day.  Avast Free Antivirus vs. Norton Antivirus.  Avast Internet Security Suite (soon to be released) against Norton Internet Security Suite.  It would be a great test if anyone can ever figure out how to do it right.

Not like the testing done in the Vince and Dennis Technology Laboratories.