Protecting over 200 million PCs, Macs, & Mobiles – more than any other antivirus

April 25th, 2010

How to make the Full System Scan 6x faster in 10 days

During the last few weeks, we have been tweaking the avast! 5 engine; and while doing this, we found out that there were some hidden reserves with respect to its performance (namely, the duration of the on-demand scans).

One of the great new features of avast 5 is the persistent cache, a mechanism which allows us to skip rescanning of certain files. In particular, this applies to files which are on our internal whitelists, as well as files which are digitally signed by trusted publishers (we maintain a relatively short list of software publishers that we trust, and we consider any files produced and digitally signed by these publishers as safe).

Previously, we were using the crypto services provided by the operating system (called “wintrust”) to do the actual verification of the digital signatures. We knew this wasn’t ideal though – especially because we realized that in case the underlying system was somehow compromised, any such system API could already be redirected/hijacked by malware and so trusting it was not 100% bulletproof. For this reason, we have been working on our own implementation of the signature verifier. What seemed like an easy task in the beginning actually turned out to be a fairly large project with tens of thousands of lines of code, and many months of work.

The works on this were finished about a month ago, and after some additional reliability testing, we finally released it to the public as part of the April 19th definition update (last Monday). What’s interesting that this change brought us not only increased reliability (the reason why we decided to implement it in the first place), but also significant performance gain. On our test system (a Dell workstation with an Intel Core i7 CPU, 4GB RAM and Windows 7) the duration of the Full System Scan time suddenly went from 39:35 to 16:03 – meaning almost 2.5x speedup!

We haven’t really done a full analysis of what’s actually causing this, but our current hypothesis is that the performance gain is related to checking of the signature catalogs. It is possible that the Wintrust APIs reopen/reread the catalogs every time a file is checked, whereas our implementation only reads them once and keeps them cached in memory for the whole duration of the scan.

Now, this by itself raised a lot of interest in exploring if things could be improved even more. So we revisited the verification code once more, and found out that the code spends most of the time in a function that is responsible for the calculation of SHA-1 hashes. This is no surprise, as pretty much all signing certificates are currently based on the SHA-1 algorithm, and the actual hashing is the most expensive part of the verification process.

So the next logical step was to optimize our implementation of she SHA-1 algorithm. Interestingly enough, one of the engineers on the Intel performance team recently published a nice article describing the possibilities to speed up SHA-1 by means of the SSE2 instructions added in the Pentium 4 processor. Using these ideas, we were able to further optimize the code so that it ran about 30% faster (especially on the latest Core 2 and i7 CPUs).

While doing all these tests, we also noticed one strange thing: the Full System Scan ran pretty much the same time during the first and all subsequent runs. It was not supposed to be like this though – the persistent cache was supposed to let the 2nd and all subsequent scans run faster. Not so dramatically as the Quick Scan (as the Full System Scan is set up so that it does not trust the persistent cache by default), but still quite significantly as we weren’t supposed to be verifying the digital signatures of files during these repeated scans. So we reviewed the relevant code, and were quite surprised to find out that the verification task was indeed performed every time, not just in the first pass. Fixing this (in the yesterday’s engine update, April 24th), we were able to cut down the scan time on that reference machine down to mere 6 minutes 54 seconds – which translates to almost 6x speedup (with no effect on dection rates, of course)!

For us, this was a great exercise which showed the beauty of software engineering. Sometimes, if you try really hard, you can make a heck of a difference.

By the way, I encourage you to  run a Full System Scan and report your findings here in the Comments section below. Of course, your mileage may vary (it all depends on your hardware configuration, but generally the higher-end hardware, the more significant speedups you should expect) but we expect that at least 2-3x speedup should be measurable on pretty much all systems. Also, please keep in mind that the first scan is supposed to take significantly longer so if you have never ran a Full System Scan yet, it’s good to run it twice and compare the results.

Tip: to make the Full System Scan even faster, configure it to actually take advantage of the persistent cache. To do this, open the Full System Scan details, click the Settings button and check the box “Speed up scanning by using the persistent cache” on the Performance page.

Categories: How to, Technology, Uncategorized Tags:
  1. Rednose
    April 25th, 2010 at 15:30 | #1

    Full System Scan is 2.64 x faster now.

  2. bidou
    April 25th, 2010 at 21:10 | #2

    fresh install of avast

    full scan 1:
    00:31:21
    276308 files tested

    full scan 2:
    00:26:59
    276475 files tested

    1.16x faster

    Not so impressive in my case. Other opponent like Kaspersky are doing much better on re-scanning, in my case.

  3. Light Archangel
    April 25th, 2010 at 22:02 | #3

    With The Default Settings and my old PC Pentium 3 and 256MB of RAM
    The full scan takes 47 minutes less than before!! :D

    Thanks for make your product suitable for older machines too!!

    Great Job ALWIL TEAM, simply you’re the best.

    Greetings.

  4. WJ
    April 25th, 2010 at 22:43 | #4

    Full System Scan is 2.88 x faster now.
    143531 files tested.

    Impressive

  5. April 25th, 2010 at 23:30 | #5

    bidou :
    fresh install of avast
    full scan 1:
    00:31:21
    276308 files tested
    full scan 2:
    00:26:59
    276475 files tested
    1.16x faster
    Not so impressive in my case. Other opponent like Kaspersky are doing much better on re-scanning, in my case.

    Two things to note here:
    - the article talks about speedups compare to avast v5 before April 19
    - the Full System Scan isn’t, by default, configured to take advantage of the Persistent cache (i.e. to run much faster on subsequent scans). You can enable this feature, though. Just open the Full System Scan details, click the Settings button and check the box “Speed up scanning by using the persistent cache” on the Performance page.

  6. someguy
    April 26th, 2010 at 01:07 | #6

    With persistant cache enabled, it is 2.21X faster
    -Full System Scan with persistant cache off 0:43:16
    -Full System Scan with persistant cache on 0:19:51
    Also, the quick scan takes 10min faster now :D

    Avast is great, keep up the good work guys ^-^

  7. bidou
    April 26th, 2010 at 01:54 | #7

    @vlk

    Sorry, I misread the part reguarding the full scan with persistant cache. So, with persistant cache enabled for the full scan, I got 14 minutes. Which is around 2x faster than without persistant cache enabled.

    Great improvement. However, from my experience, Kaspersky is still much faster during the rescanning. For me, it uses to take only 2 or 3 minutes when using its own persistent cache.

    I am an Avast user and I am happy with it, but the rescanning time could be even better, as I often tend to make a full scan every night. The faster the better.

    Avast seems to lose a lot of time on packed/archived files (for instance Steam/Valve/Source engine-based games), it seems it is unpacking the entire pak and re-scanning its entire content again, instead of just checking if the packed file changed from the previous scan.

  8. April 26th, 2010 at 08:24 | #8

    Hi Ondrej,

    Initial scanning with Full System Scan around 00:55:56 and second scanning we only need 00:39:02 with tested folders 19489 and amount of data tested : 74.99 GB.

    Well, that is quite nice performance…

    cheers,
    yanto chiang

  9. April 26th, 2010 at 08:26 | #9

    Yanto Chiang :
    Hi Ondrej,
    Initial scanning with Full System Scan around 00:55:56 and second scanning we only need 00:39:02 with tested folders 19489 and amount of data tested : 74.99 GB.
    Well, that is quite nice performance…
    cheers,
    yanto chiang

    Hi Ondrej,

    Sorry mistaken from my site, this happened without persistant chache :)

    cheers,
    yanto chiang

  10. Tommy Miller
    April 26th, 2010 at 10:56 | #10

    Amount of data tested : ~124 GB
    Files : ~234.000
    Folders: ~25.000

    Initial Full System Scan (persistent cache off): 0:57:16
    Second Full System Scan (persistent cache off): 0:34:18
    Third Full System Scan (persistent cache on) : 0:23:55

  11. Jared
    April 26th, 2010 at 20:22 | #11

    Amount of data tested : ~260 GB

    Full System Scan is 2.18 x faster now.

  12. Light Archangel
    April 27th, 2010 at 06:18 | #12

    Light Archangel :
    With The Default Settings and my old PC Pentium 3 and 256MB of RAM
    The full scan takes 47 minutes less than before!!
    Thanks for make your product suitable for older machines too!!
    Great Job ALWIL TEAM, simply you’re the best.
    Greetings.

    Even without the persistant cache
    The improvement is amazing!

    Greetings

  13. brad
    April 28th, 2010 at 05:44 | #13

    Avast you are freaking amazing!!!!! thank you!

    1. scan was 50 min.
    2. is 20Min

    Thank you so much keep improving!!!!!!

  14. Brandon
    April 28th, 2010 at 18:10 | #14

    232k files
    60GB

    1st scan: 20:21
    2nd scan: 10:34

  15. Brandon
    April 28th, 2010 at 18:22 | #15

    3rd scan: 8:50

  16. Mario
    April 29th, 2010 at 14:57 | #16

    Hello,

    Is Avast! going to come out with a toolbar that alerts us about a harmful website.

  17. girl843cp
    April 29th, 2010 at 14:59 | #17

    Hello,

    How do i clear the log on my computer that avast has on the Real time web shield in the report file how do i clear the words and save it so it will not be there next time i look at the report file?

  18. April 29th, 2010 at 16:23 | #18

    girl843cp :

    Hello,

    How do i clear the log on my computer that avast has on the Real time web shield in the report file how do i clear the words and save it so it will not be there next time i look at the report file?

    turn it off in the shield settings

  19. Aethec
    April 29th, 2010 at 16:59 | #19

    1st scan : 0:26:47
    2nd scan : 0:18:02
    3rd scan : 0:14:53
    Files tested : 378k / Folders tested : 39k / Total amount of data : 62.38 GB

    Progressbar shows a weird progress : it stays at 0% while scanning C:\Windows then it jumps to 10% when it begins C:\Users, growing very slowly (when in fact the total
    amount of data tested is ~30%). After that, the progress returns to a normal state.

    It also seems to stay a long time (minutes) in %localappdata%\assembly\ (has something to do with Visual Studio), when in fact the total amount of data in that folder is ~25 MB on my computer – it says the speed is ~45 MB/s when scanning the folder which means it should be scanned in under a second.

  20. lesle
    April 29th, 2010 at 21:16 | #20

    Dear vlk,

    In Full System Scan>Scan Settings>Performance there are two optional settings:
    1) Speed up scanning by using the persistent cache
    2) Store data about scanned files in the persistent cache

    The store data option is checked (=on) by install default.

    QUESTION: What happens when both are checked? Is the following reasoning correct?

    The scan “consults” the persistent cache.
    If it finds the file in the cache, it skips rescanning (thereby saving time).
    If it does not find it in the cache, it scans the file, finds no problems, AND then adds it to the cache?

    Is this correct?

  21. April 29th, 2010 at 21:57 | #21

    Mario :
    Hello,
    Is Avast! going to come out with a toolbar that alerts us about a harmful website.

    This is exactly what the WebShield was designed for, and what it’s been doing for the last ~4 years…

    You don’t need any “toolbars” for that.

  22. April 29th, 2010 at 21:59 | #22

    lesle :
    QUESTION: What happens when both are checked? Is the following reasoning correct?
    The scan “consults” the persistent cache.
    If it finds the file in the cache, it skips rescanning (thereby saving time).
    If it does not find it in the cache, it scans the file, finds no problems, AND then adds it to the cache?
    Is this correct?

    Essentially correct, even though a bit more complicated. Only known good files make it to the cache (we can’t put there all files that are not detected with today’s definitions as their infection status may change in the future).

  23. lesle
    April 29th, 2010 at 22:49 | #23

    @vlk
    Thank you!

  24. spg SCOTT
    April 30th, 2010 at 14:28 | #24

    vlk :

    Mario :
    Hello,
    Is Avast! going to come out with a toolbar that alerts us about a harmful website.

    This is exactly what the WebShield was designed for, and what it’s been doing for the last ~4 years…
    You don’t need any “toolbars” for that.

    Thought so ;)

  25. Mario
    April 30th, 2010 at 14:29 | #25

    Hello,

    When im in the Real Time protection shield looking at them my mouse moves very slowly and its not my computer cause my computer is fast as it can be why does my mouse pointer move slowly?

  26. ramjade
    April 30th, 2010 at 15:40 | #26

    @vik,

    then what is the network shield for? I thought web shield is to block downloads of infected files?

  27. zeni
    April 30th, 2010 at 18:39 | #27

    Hei can sombody help. I have a paid avast It is now telling me that I am not protected so FIX NOW. I have tried this it is not fixing what am I to do next. I tried to call the Europe office there is no answer.Please HELP!

  28. spg SCOTT
    May 1st, 2010 at 15:06 | #28

    @ramjade
    From avast! help file:

    Network shield - monitors all network activity and blocks any threats that are detected on the network. It also blocks access to known malicious websites based on the avast! database of infected URLs.

    I’m sure Vlk can elaborate some more…You could also search the avast! forum, one search result:
    forum.avast.com/index.php?topic=43834.msg366681#msg366681

    @zeni
    @Mario

    I would suggest that you visit the avast! forum, the blog is not really the best place for troubleshooting problems and help. (it will probably be quicker as well.)

    -Scott-

  29. Mulaud
    May 2nd, 2010 at 16:18 | #29

    On my laptop, a scan would usually take an hour or so.
    It now gets done in 20 minutes.

    On my desktop, what usually lasted for about 90 minutes, gets also done in 20 minutes, witch is even faster than on my laptop. (?!?)

    All of this got me more worried than happy for the last week.

  30. May 2nd, 2010 at 19:44 | #30

    bidou :
    @vlk
    Avast seems to lose a lot of time on packed/archived files (for instance Steam/Valve/Source engine-based games), it seems it is unpacking the entire pak and re-scanning its entire content again, instead of just checking if the packed file changed from the previous scan.

    In order to be able to do that, the whole archive would have to be whitelisted. That is, the persistent cache can only work with known good files – not files that it doesn’t know anything about (as their infection status can theoretically change at any time).

  31. May 2nd, 2010 at 19:46 | #31

    Mulaud :
    On my laptop, a scan would usually take an hour or so.
    It now gets done in 20 minutes.
    On my desktop, what usually lasted for about 90 minutes, gets also done in 20 minutes, witch is even faster than on my laptop. (?!?)
    All of this got me more worried than happy for the last week.

    Why would that make you worried?
    Generally, the faster machine, the bigger improvement you can expect (with the recent changes described in the article).

  32. May 2nd, 2010 at 20:14 | #32

    On a new Windows XP Home install:

    Quick
    1st Scan – 5:51
    2nd Scan – 3:30

    Full
    1st Scan – 5:26
    2nd Scan – 4.44

    The faster the quick scans go the better as that is what I run more of.

  33. GloobyGoob
    May 3rd, 2010 at 04:44 | #33

    Hey can anyone help me out? I have been using avast Free for a couple of years now, and I wanted to support the product so I’ve just purchased a license for Avast Internet Security a few minutes ago. The instructions said to just insert the file if you have the free version already so I did. A message popped up and said that the license has been successfully inserted. However, I checked the program and it was still the free version. I even restarted my comp. I bought the license for 2 years (1year free) 3 computers. If I try to insert it again will that count as another time its been used? I already used the license on one computer already and it didn’t work. Can anyone help me? Thanks.

  34. lesle
    May 3rd, 2010 at 17:32 | #34

    @vlk
    I am, to use american idioms, tickled pink, pleased as punch, to report that a thorough scan that in March took 23 hours and 46 minutes now takes 3 hours and 32 minutes.

    This is after turning on the persistent cache and three intermediate, partial scans.

    This reduction in scan time must be tempered by the following: on three hard drives, online and removable, I have 2 primary partitions, 2 daily backup partitions, 2 weekly backup partitions, and two ~monthly backup partitions. I also have an online primary software disk partition and its backup partition.

    This means that after the three primary partitions are scanned and the persistent cache populated, the scan will find the same files in the other partitions.

    I am, to use another american idiom, a happy camper!

    Thanks, Ondrej.

  35. Kurt
    May 4th, 2010 at 10:27 | #35

    @GloobyGoob
    Please contact avast support team via email support at avast dot com.

  36. danny
    May 7th, 2010 at 03:30 | #36

    ya i can find the speed increase if i do the quick scan or full system scan but while i did scan through explorer by rightclicking i dint get the same high speed.i had no viruses in tat drive, so the scan done through wil be equivalent to quick scan till it finds a virus and switch to thorough mode only if virus is found..is confused as to y this scan is alone slow?? use persistence caching option is enabled in explorer scan too

  37. Ivan Samuelson
    May 13th, 2010 at 21:28 | #37

    What about the fast scan? Should I also set it to update the cache in addition to using it? Or just leave it to use the cache only?

  38. Clint
    May 14th, 2010 at 03:35 | #38

    Did full scan on April 26, 2010
    HI The first Full scan
    Run Time is: 0:48:20

    The second full scan I did was on
    May 13, 2010
    Run time was: 0:54:09

    I like that avast free edition is much faster doing a full system scan. Keep up the great work avast team.

Comments are closed.