A Followup About AV Test Reports

Well, my post certainly got a lot of attention. I probably would have put a bit more thought into it had I known it was going to go viral (150K views and counting). I did update it a bit but I see no reason to change its main thrust.

A respondent asked me to comment on the results of some AV product comparison tests that show Microsoft's product, the one I recommended, in a poor light. I had a look at one of the reports they recommended and I think my comments are worth reproducing here.

In that test, MS got 97% (1570 out of 1619) ... lower than most of the other products, but the actual difference is very small.

The major problem with tests like this is that they are designed to fit the strengths of AV products and avoid their weaknesses. The report doesn't say how they acquire their malware samples, but I guess they get them from the same sources AV vendors do. (They're not writing their own since they say they independently verify that their malware samples "work" on an unprotected machine.) So what they're testing here is the ability of AV software to recognize malware that's been in the wild long enough to be recognized and added to someone's database. That's exactly the malware that AV systems detect. But in the real world users will often encounter malware that hasn't had time to be classified yet, and possibly (e.g. in spear-phishing cases) will never be classified. A more realistic test would include malware like that. Testers should cover that by generating a whole lot of their own malware that's not in any database, and see how AV products perform on that. My guess is that that detection rate would be around 0% if they do it realistically, partly because in the real world, malware authors can iterate on their malware until they're sure the major AV products won't detect it. (And no, it doesn't really matter if the AV classifier uses heuristics or neural nets or whatever, except that using neural nets makes it devilishly hard to understand false positives and negatives.)

So for the sake of argument let's suppose 20% of attacks in the real world use new unclassified malware and 80% use old malware and none of the 20% are detected by AV products. In this report, that would be 405 additional malware samples not detected by any product. Now Microsoft scores 77.6% and the best (F-Secure in this case) scores 79.9%. That difference doesn't look as important now.

The other major issue with this whole approach is that it takes no account of the downsides of AV products. If a product slows down your system massively (even more than other products), that doesn't show up here. If a product blocks all kinds of valid content, that doesn't show up here. If a product introduces huge security vulnerabilities --- even if they're broadly known --- that doesn't show up here. If a product spams the user incessantly with annoying messages (that teach them to ignore security warnings altogether, poisoning the human ecosystem), that doesn't show up here.

This limited approach to testing probably does more harm than good, because to the extent AV vendors care about these test results, they'll optimize for them at the expense of those other important factors that aren't being measured.

There are also some issues with this particular test report. For example they say how important it is to test up-to-date software, and then test "Microsoft Windows 7 Home Premium SP1 64-Bit, with updates as of 1st July 2016" and Firefox version 43.0.4. But on 1st July 2016, the latest versions were Windows 10 and Firefox 47.0.1. Another issue I have with this particular report's product comparisons is that I suspect all it's really measuring is how closely their malware sample import pipeline matches the pipelines of other vendors. Maybe F-Secure won that benchmark because they happen to get their malware samples from exactly the same sources as AV-Comparatives and the other products use slightly different sources. The source of malware samples is critical here and I can't find anywhere they say what it is.

Of course there may be research that's better than this report. This just happens to be one recommended to me by an AV sympathizer.

Update Bruce points out that page 6 of the report, second paragraph, does describe a bit more about how they acquired their samples, and they say they scan the Internet themselves for malware samples. I don't know how I missed that! But there's still critical information missing like the lead time between a sample being scanned and then tested. I think the issues I wrote about above still apply.

Tripling Down Against USA Conference Hosting

Well, that escalated quickly!

I wrote "Really, Please Stop Booking International Conferences In The USA" and then the very next day chaos erupted as Trump's executive order on the "seven Muslim countries" appeared and went into effect.

I'm less open-borders than a lot of the people currently freaking out --- but regardless, the order is cruel and capricious, especially how it shut out people with valid visas and even green cards without warning. And I think this reinforces my point about conferences: how can you book a conference with international attendees in a country where this happens?

Administration staff are already talking about other disturbing changes:

Miller also noted on Saturday that Trump administration officials are discussing the possibility of asking foreign visitors to disclose all websites and social media sites they visit, and to share the contacts in their cell phones. If the foreign visitor declines to share such information, he or she could be denied entry. Sources told CNN that the idea is just in the preliminary discussion level.

No thanks :-(.

It has been pointed out that Trump's EO might actually force some organizations to hold more conferences in the USA because some US residents may not be able to return home if they go abroad. That makes some sense but it feels wrong.

On another note, around now the top US universities start their annual drive to recruit foreign grad students. I wonder how that's going to go. The professoriat not being exactly Trump country, they'll have to reconcile their fears with a message that foreign students are going to be OK. Careers depend on it.

Rustbelt Is Hiring

Rustbelt is looking for postdocs.

Most academic CS research projects strike me as somewhere between "could be useful" and "hahahaha no". Rustbelt is one of the very few in the category of "oh please hurry up we need those results yesterday". That's exactly the sort of project researchers should be flocking to.

The long-term goal here is to specify formal semantics for Rust's safe and unsafe code and have a practical, tool-supported methodology for verifying that modules containing unsafe code uphold Rust's safety guarantees. This would lead to an ecosystem where most developers mostly write safe code and that's relatively easy to do, but some developers sometimes write unsafe code and that's a lot harder to do because you have to formally verify that it's OK. In exchange, you know that your unsafe code isn't undermining the benefits of Rust; it's "actually safe". I think this would be a great situation to have. We need the results as soon as possible because they will suggest changes to Rust and/or what is considered "valid unsafe code", and the sooner they happen, the easier that would be.

rr Talk At Auckland C++ Meetup, February 21

I've signed up to give a talk about rr at the Auckland C++ Meetup on February 21, at 6pm. Should be fun!

Really, Please Stop Booking International Conferences In The USA

Last year I opined that international organizations should stop booking conferences in the USA in case Trump became President and followed through on his promise to ban Muslims from entering the country.

That particular risk soared when he was unexpectedly elected, then decreased given he seems to have backtracked on a blanket Muslim ban. Still, it feels like almost anything could happen, and because conference locations have to be booked pretty far ahead, IMHO prudence still means preferring non-USA venues until the future becomes more clear. (The pre-Trump ESTA changes asking for social media account data are also quite troubling. It makes me wonder whether I'm making trouble just posting this .. and I shouldn't have to.)

I especially suggest that if you're one of those people who sees Trump as a clear and present danger to American democracy and social order, it doesn't really make sense to be comfortable with organizing international conferences there.

(To be clear: I'm not calling for some kind of boycott to punish Americans for making "the wrong choice". It's simply about making sure international conferences are as inclusive and congenial as possible.)

Disable Your Antivirus Software (Except Microsoft's)

I was just reading some Tweets and an associated Hackernews thread and it reminded me that, now that I've left Mozilla for a while, it's safe for me to say: antivirus software vendors are terrible; don't buy antivirus software, and uininstall it if you already have it (except, on Windows, for Microsoft's).

Update (Perhaps it should go without saying --- but you also need to your OS to be up-to-date. If you're on Windows 7 or, God forbid, Windows XP, third party AV software might make you slightly less doomed.)

At best, there is negligible evidence that major non-MS AV products give a net improvement in security. More likely, they hurt security significantly; for example, see bugs in AV products listed in Google's Project Zero. These bugs indicate that not only do these products open many attack vectors, but in general their developers do not follow standard security practices. (Microsoft, on the other hand, is generally competent.)

Furthermore, as Justin Schuh pointed out in that Twitter thread, AV products poison the software ecosystem because their invasive and poorly-implemented code makes it difficult for browser vendors and other developers to improve their own security. For example, back when we first made sure ASLR was working for Firefox on Windows, many AV vendors broke it by injecting their own ASLR-disabled DLLs into our processes. Several times AV software blocked Firefox updates, making it impossible for users to receive important security fixes. Major amounts of developer time are soaked up dealing with AV-induced breakage, time that could be spent making actual improvements in security (recent-ish example).

What's really insidious is that it's hard for software vendors to speak out about these problems because they need cooperation from the AV vendors (except for Google, lately, maybe). Users have been fooled into associating AV vendors with security and you don't want AV vendors bad-mouthing your product. AV software is broadly installed and when it breaks your product, you need the cooperation of AV vendors to fix it. (You can't tell users to turn off AV software because if anything bad were to happen that the AV software might have prevented, you'll catch the blame.) When your product crashes on startup due to AV interference, users blame your product, not AV. Worse still, if they make your product incredibly slow and bloated, users just think that's how your product is.

If a rogue developer is tempted to speak out, the PR hammer comes down (and they were probably right to do so!). But now I'm free! Bwahahaha!

Browser Vendors And Business Interests

On Twitter it has been said that "browser-vendors support web-standards when and only when those standards align with their own business interests". That's not always true, and even if it was, "business interests" are broad enough to make surprising results possible in a competitive browser market.

Mozilla, as a nonprofit, isn't entirely driven by "business interests". Mozilla often acts for "the good of the Web" even when that costs them money. (Example: pressing on with their own browser engine instead of switching to Chromium.)

Other vendors perceive (rightly or wrongly) that being seen to "do the right thing" has some business value. There is a PR and marketing effect, but also a recruiting effect; being seen as an evil empire makes it harder to recruit talented staff when other good options are available. To some extent Mozilla's existence has encouraged other vendors to compete on virtue.

Competitive markets can force vendors to implement standards they otherwise might not want to. For example, Apple needs an iOS browser that can render the modern Web or they'll leak market share to Android, so they're forced to implement Web platform improvements that you might think are not in the interests of their App Store business.

Decisions about Web standards and implementations are often made by individuals keen to "do the right thing" even if it might clash with corporate priorities. Everyone's good at rationalizing their decisions to themselves and others.

Browser Vendors Are Responsible For The State Of Web Standards

The W3C and WHATWG are mainly just forums. They have some policies that may help or hurt standards work a little bit, but most responsibility for the state of Web standards rests on the participants in standards groups, especially browser vendors, who drive the development of most Web standards and are responsible for implementing most of them too. Therefore, most of the blame for problems in Web standards should be assigned to the specific browser vendors who generated and implemented those standards, and influenced them in various directions.

For example, the reason media element playback, Web Audio and MediaStreams are all quite different APIs is because when I proposed a MediaStream-based alternative to Web Audio, no other browser vendors were interested. Google already had separate teams working on Web Audio and MediaStreams and was already shipping behind a prefix, Apple was barely engaged at all in the Web Audio working group, and Microsoft was completely disengaged. It's not because of anything specifically wrong with the W3C or WHATWG. (FWIW I'm not saying my proposal was necessarily better than what we got; there are technical reasons why it's hard to unify these APIs.)

In fact, even assigning responsibility to individual browser vendors glosses over important distinctions. For example, even within the Chrome team you've got teams who care a lot about Web standards and teams who care a bit less.

One way to make positive change for Web standards is to single out specific undesirable behavior by specific vendors and call them out on it. I know (from both giving and receiving :-) ) that that has an impact. Assigning blame more generally is less likely to have impact.

Parenting Notes

• My oldest son just got a new phone. By mutual agreement, it's a dumbphone. He's trying to activate it and feels obliged to read the terms and conditions before agreeing to them. Somehow he's even more of a stickler for rules than I am; I've been trying to teach him how you sometimes need to break rules, but it's tricky work without wounding his conscience. He's struggling through the obtuse legalese. It's a real coming-of-age moment.
• My children don't watch much TV and their Internet usage is restricted, but they read a lot of books and I'm supposed to keep track of them. I try hard to get them to read books I've already read and liked, but I still end up having to read a lot of popular child-oriented fiction, much of which is trash. Why aren't more of the ultra-popular teen series better written? There are some good writers, like J.K. Rowling, but so many others just churn out formula with mediocre writing and people lap it up. Right now I'm in Michael Grant's Gone series, which has very average writing but at least combines some good ideas in interesting ways. Could be worse; there's Rick Riordan, who writes similarly but without the ideas. The same rubbish exists in adult fiction, but I don't have to read that.
• My kids don't like movies. I struggle to get them to go and see any movie, Star Wars, whatever, or even watch them at home. I don't understand it.
• On the flip side, they love modern board games. That's excellent because my wife and I do too. It's a bit tough trying to work during the school holidays while they're constantly playing great games like Dominion, Lords of Waterdeep, Settlers, etc.

Cheltenham Beach

Today we drove to Devonport to walk around North Head and Cheltenham Beach. It was a lovely summer day and Cheltenham Beach looked amazing, especially for a beach that's more or less in the heart of Auckland.

How China Can Pressure North Korea

This CNN article claims that there's very little even China can do to influence North Korea. I wonder why that is, because (though I'm relatively uninformed) it seems to me China could have a big impact on North Korea by reforming their policy for handling North Korean escapees.

Currently Chinese policy is that any escapees are returned to North Korea. This is inhumane because those returnees are jailed and often tortured, not to mention the hardships endured by escapees trying to reach South Korea traveling in secret. I understand that China doesn't want to host a flood of North Korean refugees, but I don't know why they couldn't coordinate with South Korea to efficiently ship any and all North Korean escapees to South Korea. (Maybe South Korea doesn't want this, but they could be made to accept it.) This would probably lead to people pouring out of North Korea, which would put significant pressure on the regime. Not only would this give political leverage, but it's also a far more humane policy.

Is CMS Software Generally Really Bad?

I'm helping overhaul our church Website. The old site was built in Joomla 1.5, and the new site is in Joomla 3.6 (I didn't choose the tech). Joomla's supposed to be a pretty popular system so I'm amazed at how much I dislike it and wondering whether there are better options available. There are a lot of small annoyances, some major design issues, and some really bad bugs.

One fundamental issue is that it's very difficult to work backwards from viewing a page on the site to being able to change the content on that page, until you've studied Joomla and the site internals to figure out how the pages are assembled. I had to read a lot of not-particularly-well-organised documentation to grasp the concepts of menus, articles, components, modules and templates, then do hours of spelunking with Firefox devtools, browsing the administrator interface and making educated guesses to figure out which page parts were generated by which Joomla entities. It seems to me that this could be a lot easier, either by offering WYSIWYG tools that show you visually how a page is assembled (and let you edit those parts in-place!), or at least by leaving consistent notes in the generated DOM to indicate where pieces came from. Simplifying the assembly model would also help a lot.

Another fundamental issue is that there's no version control or change preview. As far as I know the only way to figure out what impact a change is going to have is to make it on the live site. It might be possible to set up a staging server, but that looks difficult, and probably impractical for a small-ish project like ours. If you don't like the effects of the change you have to reverse it manually, which is horrible and error-prone. Without version control there's no way to view changes made by others, make experimental branches, etc. The two latter problems would not be improved by a staging server.

Those issues seem so fundamental to me I'm surprised a major CMS fails on them.

Then there are the bugs. My least favourite bug right now is that often, when I edit HTML using the WYSIWYG HTML editor, upon saving the article the targets of all links are replaced with the target of the last link (destroying my work in a non-undoable way, see above). I assume such a terrible bug must be specific to our installation somehow, but the system is complex and opaque enough that debugging it seems impractical.

Our church's needs are not complicated. The site is mostly static and there isn't a huge amount of content. Nevertheless the pages load quite slowly and the generated HTML/CSS/JS is bloated. It's tempting to start over with a completely different approach. Then again, I know there's a plethora of CMSes and Web frameworks and Joomla is very popular (at least in absolute terms) so I feel like there must be something I'm missing.