Every day, 50,000 infected unique URLs of HTTPS-protected websites are detected and blocked. Scientists from the Concordia University in Montreal, Canada, have tested 14 antivirus programs offering HTTPS scanning and found that these programs create more security problems than they actually solve. There was only one exemption from this: Avast. The only issue mentioned in their study is a lack of revoked certificates checking by Avast, which has been in the market since November 2015 and is fixed in 2016 products.

But, let’s take a step back – what is HTTPS scanning? Why is it necessary and why does it make sense if it seems to be adding security problems instead of protecting the user? We wanted to learn more about this, so we asked our colleague Lukas Rypacek, Director of Desktop Platform, to tell us more.

Lukas, HTTPS sites on the web are encrypted, so aren’t they already secure ? Why do antivirus solutions scan HTTPS sites?

Indeed, the “S” in HTTPS stands for “secure”. The fact that a website uses HTTPS means that the connection between the browser and the website’s server is encrypted. It means that your browser and the web server are the only ones that can see your activities and the data that you enter on the website. Think of it as sending a letter inside a closed envelope so it cannot be visible to third parties. While a closed letter cannot be read by the postman, your communication with an HTTPS-encrypted site cannot be seen by hackers who would like to steal your credit card number, for example. HTTPS also helps the browser to verify the identity of the server; it can help you verify that the website you are visiting really belongs to your bank or truly is your favorite blog. This is done with the help of so-called certificates.

However, the encryption does not help much if you don’t connect to your favorite HTTPS-enabled blog but to a new website you’ve just discovered in Google Search. You still can verify that you are communicating with the website via encrypted connection, but this doesn’t tell you much, since you cannot possibly know if the website on the other end of the network is a friendly newspaper or a hacker’s den where a hacker is just waiting for you to enter your data, so he can steal it. It is even more problematic when your favorite website gets hacked. Until the owners find out about the problem and remove it, malware is being served from this otherwise trustworthy domain. In these cases, the encryption doesn’t help you stay safe -- even though your engagement with the website stays invisible to the outside world, it doesn’t mean you are protected from malware.

So you can get infected by a virus if you visit a HTTPS site?

Yes. As more and more online services are moving to HTTPS, attacks are increasingly coming over HTTPS. Today, around 30% of all the web traffic is running on HTTPS. When hackers succeed in hacking into a trustworthy site and inject a malware script or a drive-by-download malware onto the pages, visitors are going to be served with this malware over both HTTP and HTTPS. Furthermore, a few years ago, HTTPS was used mainly by big organizations such as banks. It was somewhat expensive and difficult to obtain a TLS/SSL certificate to encrypt the web traffic. For malware authors, it didn’t pay off to purchase such a certificate to host their website, as users were happy connecting over the easier HTTP. Today, however, anybody who owns a domain can obtain a TLS/SSL certificate, free of charge. Services such as Letsencrypt.org make this process easy for anyone.

This is why this has become interesting for cybercriminals – while it is still difficult for them to pretend a website is a bank, they can build websites that look like harmless download sites but contain malicious content.

How often do HTTPS sites contain malicious content?

Avast detects an average of 50,000 infected and blocked unique HTTPS URLs every day, and each month, Avast protects nearly 3.2 million users from downloading malware or accessing malware distribution sites using HTTPS.

How does antivirus software manage to scan HTTPS connections? How is this possible when the connection is encrypted?

Avast runs on the same machine as the browser. It sits there and waits for the browser to connect to a HTTPS site and start the handshake. The handshake is the first part of HTTPS communication between the browser and the server the website is stored on. The server, and in some cases the client, provide a certificate that allows the browser to determine if the connection should be trusted or not. When this happens, Avast Web Shield takes over the handshake on behalf of the browser and connects itself to the server. It is Avast Web Shield that connects over the Internet with the server and downloads the encrypted traffic. Avast then decrypts it, scans for viruses and hands it over to the original browser like Internet Explorer, Chrome, Firefox or Opera.

Browsers constantly improve the list of checks they perform on HTTPS connections and TLS/SSL certificates. According to the study, this is where many antivirus programs make mistakes. Isn’t the user losing all those security improvements in browsers?

No, not necessarily. This is also one of the important things that researches from  Concordia University tested – how various anti-malware suites implement all the necessary details and if the security is actually improved or worsened. We at Avast have put  great effort into mimicking all the details we see on the connection with the server back to the browser. This gives the browser the opportunity to do most of its checks, despite the fact that Avast has already scanned the data.

When the server sends its certificates, Avast Web Shield verifies them against the Windows System Certificate Store, which is an official list of trusted certificates that browsers like Internet Explorer or Chrome also use. Avast replicates all the attributes in the original certificate back to the browser (such as its expiration date or type of hash algorithm), and it also carefully replicates all the encryption suites and TLS/SSL protocol versions. This is where the study gave credit to Avast and also showed that not all antivirus solutions are consistently careful with this challenge.

The study also mentions that four products are vulnerable to full server impersonation under an active man-in-the-middle (MITM) attack out of the box. What is a full server impersonation and what does this mean for the user’s security?

As I’ve previously mentioned, HTTPS gives the browser the possibility to verify that the other end is truly what it claims to be – simply put, it verifies whether a web page  is really the bank I think I’ve just logged into. This is done by verifying the certificate sent from the server, checking the attributes in it, the name of the website,expiration dates and the certificate authority signing all this. The worst case scenario is when the antivirus product does not do anything of this. Surprisingly, it did happen to some of the tested HTTPS scanners in some special cases (such as when the license was expired). Such an omission allows anyone on the way from the server to the client (e.g. the guy running the Internet café you visit) to pretend to be any website, such as a bank or email server. This renders the whole encryption useless. Going back to our envelope analogy, it’s as if the postman brought you letters that had already been opened and you just reseal them without caring.

How does Avast make sure this cannot happen under the Avast Web Shield?

We simply do not accept letter envelopes that have already been opened, but in reality, it is somewhat more complicated.

Avast Web Shield checks the certificates against the Windows Store of trusted root certificate authorities, which is the absolute minimum. We also never pretend that the connection or certificate is better that it actually is. When the server uses outdated ciphers, we hand over exactly the same ciphers to the browser, so if the server used an old version of the TLS/SSL protocol, we do the same with the browser. With this approach, most of the security checks in modern browsers can still be active and help users discover imposters. Also, we remove and warn against all viruses and malware scripts hidden inside the data, which is the actual purpose of our HTTPS scanning feature.

Moreover, the study shows that several of the reviewed antivirus solutions “also mislead browsers into believing that a TLS connection is more secure than it actually is, by e.g., artificially upgrading a server’s TLS version at the client”. How would this make the solutions more vulnerable and how does Avast solve this problem?

Avast (and other anti-malware suites) run on the same computer as the browser, yet they communicate with each other and they use HTTPS (TLS/SSL). It is implemented this way not as a security measure, but to give the browser a taste of what the actual connection with the real server looks like. When the anti-malware solution sends a better version of the TLS protocol, better cipher or more trusted certificate than what was in the original connection, the browser will mistakenly consider the connection safer and will behave accordingly, when in reality, a proper reaction from the browser would be to display a warning screen.

Are there any problems even with Avast’s solution or things that you would like to improve in the future?

We are constantly trying to improve how the Avast Web Shield HTTPS scanner works. The study pointed out that Avast failed to check for revoked certificates at the time of testing. This was true a year ago, but since then we have fixed the issue in Avast 2016 (which we released in November of last year) and we’re now correctly checking the revocation of certificates using several methods (CRL, OSCP and OSCP stapling). For the users of Chrome and Firefox we have introduced a new, completely unobtrusive way of scanning the traffic that is even more transparent and allows the browser to best put all the built-in security checks to use. This update is also in line with the suggestions from the study, and we continue to prepare more improvements in the next versions of Avast.

Ultimately, it can be said that the recently-published study includes information that pertains to 2015 products. Just as we improved our HTTPS scanner with our Avast 2016 version, it can be that competitors have also optimized their features in the meantime.

Thanks, Lukas, for these interesting insights!