I've just read a press release form AV-Comparatives where they announce a partnership with AntiMalware Test Lab. Long time ago we decided not to participate in AV-Comparatives tests, for a number of reasons. Our opinion is the following:
– The tests should be run by skilled people that must be able to distinguish a malware sample from a goodware one.
– The testers should have a malware test-bed, and of course it should include malware files, with no clean files, no damaged files, etc. It also has to be representative, it does not make sense to test malware that died 15 years ago.
– The testers should check the detection capabilities of each and every product. To do that you will run different samples and see what each product is able to do (behaviour blocking, heuristics, signatures, etc.).
– The testers should be vendor independent, to avoid any bias.
Even though the new alliance won't solve all the mentioned problems, at least we know that now there is someone that knows the difference between a Trojan and a goodware file. We'll be looking forward for new tests run, let's see if we can see something more scientific and serious than a simple "right click on folder, run scan". Anyone can do that at home.
Finally, there is something really serious going on that I have to point out: for the tests in 2008, AV-Comparatives is charging AV companies in order to appear in their tests: some of them are paying 8,000€, another ones just 4,000€, others 2,000€ and even others will be tested for free. Can we be sure that there will not be any bias when testing someone who paid him 8,000€ against another one that is being tested for free? I kindly ask you to have this in mind while reading a magazine comparative when the results are provided by AV-Comparatives.