Selecting the right security solution to protect your devices isn’t always an easy decision to make - whether you’re a consumer or a larger organization. Luckily, there are many independent testing laboratories who do their job at testing security products well, so end customers have a way to compare and select a solution that suits their needs best. It’s not often we see a new kid (tester) on the block. 

Yesterday, June 22, 2017, Wikileaks published CIA documents from the agency’s Brutal Kangaroo project. These documents show that the CIA has tested antivirus solutions and they therefore could be recognized as the newest (in)dependent security tester. Sure, the documents are a bit old, but still. I’m personally looking forward to seeing the CIA join the Anti-Malware Testing and Standards Organization (AMTSO) to adopt and collaborate on existing and new testing guidelines and standards.

I’m pleased to inform you that Avast Internet Security received the CIA-graded certification in their test, which was published on WikiLeaks (look for “AIS” = Avast Internet Security). Avast was one of just few antivirus vendors who were able to spot that something suspicious was happening when the CIA tested Avast’s cyber arsenal, to see which antivirus protections they could bypass. The CIA probably prefers you use all the other antivirus solutions, because they either didn’t detect the CIA’s malware or only dumped the execution information into the log file.

 But from the point of testing, I think there’s huge room for the CIA to make improvements. Here are few points and I hope the CIA will consider them for all of their future testing rounds. These suggestions are based on the best practices, AMTSO guidelines, and newly developed standards.

• Security solution vendors should be notified prior to testing and be given the ability to properly set up their products, making sure everything works well (cloud connection, updates and latest definitions installed, ... ). Especially if the CIA runs tests using APT (advanced persistent threat) tools, which they want to use for nation-wide attacks or against larger organisations - it’s important to set up the security products being tested against properly, otherwise the CIA might get wrong results. 
• Developing your own malware for testing might be considered unethical, but some respected testers at least use exploitation toolkits to prove anti-exploit protection. I agree that these are different circumstances, as the CIA eventually uses the malware that they produce and test in the wild. Testing malware you create yourself is just a different kind of test, not a 0-day, not a real world, but a kind of future testing. The CIA isn’t the only one that does this, cybercriminals do this as well.
• We weren’t invited to the CIA’s dispute process and I don’t understand why? Do they not have one, did we miss an email? It’s always necessary for testers to confirm the maliciousness of the samples, logs and malware interactions that were used in the testing environment, with the vendors that were tested. Without this, vendors can’t really be sure the test was performed well.

I think this is enough and I believe the CIA will make some steps to become better tester, at least I hope so ;).