The EFF SSL Observatory is a project to investigate the certificates used to secure all of the sites encrypted with HTTPS on the Web. We have downloaded a dataset of all of the publicly-visible SSL certificates, and will be making that data available to the research community in the near future.

This is exciting. I knocked together a less ambitious version of this last year, but the EFF guys are doing it like grown-ups, and are getting some interesting data.

Numbers-wise, they’re in the right ballpark, as far as I can tell. Their numbers (1-2m CA-signed certs) coarsely match ones I’ve seen from private sources. I’ve heard from a few CAs that public-crawl estimates tend to err 50-80% low since they miss intranet dark matter, but at least the EFF is tracking other public-crawls. Given that their collection tools and data are going to be made public, that’s a really big deal. Previously, I haven’t been able to get this kind of data without paying for it or collecting it myself. If the database is actively maintained and updated, this will be a great resource for research.

Their analysis of CA certificate usage is also interesting. I’d like to see more work done, here, and in particular I’d like to see how CA usage breaks down between the Mozilla root store and others. We spend considerable effort managing our root store, and recently removed a whole pile of CA certificates that were idle. In some places, the paper seems to make the claim that fully half of trusted CAs are never used, but in other places, the number of active roots they count outnumbers our entire root program. I understand why they blurred the line for the initial analysis, but it would be swell to see it broken out.

As they mention, there are legit reasons for root certs to be idle, particularly for future-proofing. We have several elliptic curve roots, and some large-modulus RSA roots, which are waiting for technology to catch up before they become active issuers while giving CAs a panic switch in the case of an Interesting Mathematical Result — that feels okay to me. On the other hand, if there are certs which are just redundant, it would be great to know, so that we can have that conversation with the relevant CAs, and understand the need to keep the cert active.

This is exactly what I hoped would come of my crawler last year, but they’ve done a much more thorough job. We’ve seen an uptick in research interest in SSL over the last few years. Having a high quality data source to poke when testing a hunch is going to make it easier to spot trends, positive or otherwise. Interesting work, folks; keep it going!

Kathleen Wilson works for the Mozilla Corporation, and manages our queue of incoming certificate authority requests. She coordinates the information we need from the CAs, shepherds them through our public review process and, if approved, files the bugs to get them into the product.

Q: Holy crap! One person does all of that? Is she superhuman?

It has been proven by science. She is 14% unobtainium by volume.

Q: That’s really awesome, but I am a terrible, cynical person and require ever-greater feats of amazing to maintain any kind of excitement.

She came in to a root program with a long backlog and sparse contact information, and has reduced the backlog, completely updated our contact information, and is now collecting updated audit information for every CA, to be renewed yearly.

Q: Hot damn! She’s like some kind of awesome meta-factory that just produces new factories which each, in turn, produce awesome!

I know, right? She has also now removed several CAs that have grown inactive, or for which up to date audits cannot be found. They’ll be gone as of Firefox 3.6.7. They’re already gone on trunk.

Q: Wait, what?

Yeah – you can check out the bug if you like. I’m not positive, but I think this might represent one of the first times that multiple trust anchors have ever been removed from a shipping browser. It’s almost certainly the largest such removal.

Q: I don’t know what to say. Kathleen completes Mozilla. It is inconceivable to me that there could be anything more!

Inconceivable, yes. And yet:

1. She’s also made what I believe to be the first comprehensive listing of our root, with signature algorithms, moduli, expiry dates, &c.
2. In her spare time, she’s coordinating with the CAs in our root program around the retirement of the MD5 hash algorithm, which should be a good practice run for the retirement of 1024-bit RSA (and eventually, in the moderately distant but forseeable future, SHA-1).
3. She has invented a device that turns teenage angst into arable land suitable for agriculture.

Fully 2 of the above statements are true!

Q: All I can do is whimper.

Not true! You can also help! Kathleen ensures that every CA in our program undergoes a public review period where others can pick apart their policy statements or issuing practices and ensure that we are making the best decisions in terms of who to trust, and she’d love you to be a part of that.

• subscribe to the mozilla.dev.security.policy usenet group
• or the mailing list version
• or the google group clone

Q: I’ll do it! Thanks!

No, thank you. That wasn’t a question.

Scam-Detective: A reader has asked me to talk to you about face to face scams. Were you ever involved in meeting a victim, or was all of your contact by email?

John: I never met a victim, but I was involved in a couple of Wash-Wash scams.

Scam-Detective: Wash Wash scams? What does that involve?

John: We would tell the victim that we had a trunk full of money, millions of dollars. One victim met some of my associates in a hotel in Amsterdam, where he was shown a box full of black paper. He was told that the money had been dyed black to get through customs, and that it could be cleaned with a special chemical that was very expensive. My associates showed him how this worked with a couple of $100 bills from the top of the box, which they rinsed with some liquid to remove the black dye. Of course the rest of the bills were only black paper, but the victim saw real money. He handed over $27,000 (about £17,000) to buy the chemicals and was told to return to the hotel later that day to pick up the cash. Of course when he came back, there was nobody there. He couldn’t report it to anybody because if it had been real it would have been illegal, so he would have gotten himself into trouble.

Part 1, Part 2, Part 3.

We build tools in Firefox like stale-plugin warnings and malware blocking to help protect our users, to neuter the technological attacks they may encounter on the web. But we also try, and need to keep trying, to build tools that inform our users so that they can make better decisions. Our phishing warnings and certificate errors try to do this, but mostly by scaring users away from specific attack situations. I hope we’ll continue to build tools like Larry which try to give people some affirmative context as well, to lend some nuance to their sense of place online. I want us to help our users know when they’re on Main Street, and when they’re in an alley.

I know: People get conned in the real world, too, and certainly no browser UI is going to save you from an email-based scam. Stories like this, though, are just specific instances of what I believe to be a more universal principle:

the biggest security risk most people face is misplaced trust

John: Some of the blame has to go to the victims. They wanted the money too because they were greedy. Lots of times I would get emails telling me that they wanted more money than I was offering because of the money they were having to send. They could afford to lose the money.

Scam-Detective: John, I think you have been basically honest with me so far. Please don’t stop that now. You know as well as I do that not all of your victims were motivated by greed. I have seen plenty of scam emails that talk about dying widows who want to give their money to charity, or young people who are in refugee camps and need help to get out. You targetted vulnerable, charitable people as well as greedy businessmen, didn’t you? You didn’t care whether they could afford it or not, did you?

John: Ok, you are right. I am not proud of it but I had to feed my family.

If you have ideas for how we can help users place their trust online more deliberately and carefully: please comment here, or build an addon, or file a bug.

I don’t know if there are people out there who like the way they sound in audio recordings, or look on video. I certainly don’t. I don’t think it’s a self-image issue, either; and I know I’m not alone. My recorded voice lacks the resonance I experience internally, and my recorded image just looks… mouthier (?!) than I imagine myself to be. I don’t even know what that means.

Proposed:

Nightingale’s Corollary to the Uncanny Valley Hypothesis: The depth of one’s psychological attachment to, and familiarity with, one’s own image, amplifies feelings of canny/uncanniness. This can result in greater than average affinity for moderately dissimilar representations (c.f. the popularity of “realistic cartoon avatar” generators, or caricature artists), but also particularly heightened sensitivity to minor dissimilarities.

[Discuss. Cite examples.]

The Point (i.e. Where You Should Have Started Reading)

I bring this up because the inimitable duo of Alix and Rainer recently took some of my scattered ramblings and knit them together into an educational piece on some of the security features in Firefox. I think they did a lovely job:

YouTube

In very much related news, Drew worked with Alix and Rainer to put together a video that talks about some of Firefox’s privacy features. I find it much easier to listen to Drew’s calm, matter of fact, “we did awesome stuff, and want you to know about it” delivery. I suspect you will, as well.

YouTube

In what history will likely judge as a vain attempt to impress him, then, I recently landed bug 513147, deletion of the now antiquated “Properties” dialog that used to be available on right-clicking things like images and links. Not because it was useless (every feature is someone’s baby, and is added for a reason) but because it wasn’t useful enough, to enough people, to justify the cost.

50kb of code in our product that is poorly understood, not often used, and not covered by unit tests is not free. When bugs show up, it takes longer than it should to fix them. If a security bug were to show up (which is always a risk when content mixes with chrome, however remote it may seem) it would be particularly expensive for us to reload that context into our brains to fix it.

Deleting it isn’t free either, of course – there are 4 extensions that build off that dialog that will need to be updated, and there may be some who use it regularly who will be disappointed. But the forces of software (inertia, squeaky wheels, cynicism and inertia) bias so heavily towards keeping code in the tree that we should all try to take clear deletion opportunities when they come up. Not capriciously, not without sensitivity to the impact it can have, but with recognition that the hidden cost to keeping them is also large and… hidden.

It is in the spirit of this sensitivity that we, on the Firefox team, have tagged this bug and others like it: [killthem].  What else do you think should go? (And please, be gentle. Remember, every feature is someone’s baby.)

[Update: Geoff Lankow has taken the code that used to be built in, and made it into an add-on, which is think is fantastic. As I said to him, and as I said above, my assertion has never been that the code was useless, just that it wasn't useful enough to justify its cost in the core product. An add-on is a great place for functionality like that, and I thank Geoff for his work.]

Easily the most often requested improvement was to include intermediates in the database. People wanted to see which issuers had a bunch of subordinate CAs and which issued right from the root. They wanted to see what kind of key sizes and algorithms CAs chose, and how they compared to the key sizes and algorithms used in regular site certs.

I’ve gone and re-crawled to gather that information now, and you can download the zipped db (509M). It’s still an SQLite3 database, though I’ve changed the schema a bit, with certificates now stored in their own table.  Let me know in the comments/email if you need help working with the data.

The schema, if you can call it that, was 100% expediency over forethought, so I would welcome any suggestions on DB organization/performance tweaking. I have done no optimizing so low-hanging fruit abounds, and a complicated query can take more than a day right now, so your suggestions will have visible effects!

Brief Background

Phorm‘s technology hooks in at the ISP level, watches and logs user traffic, and uses it to assemble a comprehensive profile for targeting advertising. While an opt-out mechanism was provided, many users have complained that there was no notice, or that it was insufficiently clear what was going on. NebuAd, another company with a similar product, has apparently used its position at the ISP level to not only observe, but also to inject content into the pages before they reached the user.  It’s hard to get unbiased information here, but this is what I understand of the situation.

Thoughts

1.  Deep packet inspection, in the general case, is a neutral technology. Some technologies are malicious by design (virus code, for instance), but I think DPI has as many positive uses as negative. DPI can let an ISP make better quality of service decisions, and can be done with the full knowledge and support of its users. I don’t think DPI, as a technology, should be treated as insidious.

2. Using deep packet inspection to assemble comprehensive browsing profiles of users without explicit opt-in is substantially more questionable. My browsing history and habits are things I consider private in aggregate, even though any single visit is obviously visible to the site I’m browsing.

It’s possible that I will choose to allow this surveillance in exchange for other things I value, but it must be a deliberate exchange. I would want to have that choice in an explicit way, and not to be opted in by default, even for aggregate data. Moreover, given the complexity of this technology, I would want a great deal of care to go into the quality of the explanation.  Explaining this well to non-technical users might be so difficult as to be impossible, which is why it’s so important that it be opt-in.

3. Using deep packet inspection in conjunction with software that modifies the resultant pages to include, for instance, extra advertising content, is profoundly offensive and undermines the web. The content provider and the user have a reasonable expectation that no one else is modifying the content, and a typical user should not be expected to understand the mechanics of the web sufficiently to be able to anticipate such modifications.

Solutions

As a browser, we do some things to help our users here, but we can’t solve the problem. https resists this kind of surveillance and tampering well, but requires sites to provide 100% of their content over SSL. Technologies like signed http content would prevent tampering, if not surveillance, but once again assume that sites (and browsers!) will support the technology. Ad blockers can turn off any injected ads, tools like NoScript can de-fang any injected javascript but, fundamentally, http content is not tamper-proof, and no plaintext protocol is eavesdropping proof.

People trust their ISPs with a huge amount of very personal data. It’s fine to say that customers should vote with their feet if their ISP is breaking that trust, but in many areas, the list of available ISPs is small, and so the need for prudence on the part of ISPs is large.

That’s what I’m thinking so far, what am I missing?

• What proportion of CA-signed certs are using MD5 signatures?
• What key lengths are being used, with which algorithms?
• Who is issuing which kinds of certificates?

So I decided to go get some of that information, so that I could give it to all of you wonderful people.

What I Did

I put together some python to crawl a list of sites and record the details of their response to an SSL handshake into an sqlite3 database. I don’t know python, and my SQL is extremely rusty, but it works.  The code is here, take it and make it better!  Pay particular attention to the TODO list!

As for a list, I used Alexa’s list of the top 1,000,000 sites, which they quite helpfully make available for free download. Of course we can have all kinds of fun debating whether this is the right list to use, it’s obviously going to have some skews. Anyone with a similar list can either hook me up and I’ll give it a whirl, or download the code and run it themselves.  This list did get me 382,860 certificates from the public internet though, so that’s pretty okay.

What I Can Do With It

For each host in the list, I currently record:

• Whether the connection succeeded or not, and if not, why
• The verification result, using the Mozilla list of CAs
• Various and sundry connection/certificate details (subject, issuer, cipher, keylength)
• The PEM-encoded end-entity cert, for post-analysis

What that means is that I can answer some reasonably relevant questions. For instance, during the recent excitement over MD5 weaknesses, we have been having conversations about retiring MD5 as a supported signature algorithm as, I’m certain, have the other browsers. Making that decision in an informed way requires us to understand how much of the internet still relies on MD5, though, and when those certificates will expire.

What Can You Do With It?

There’s almost 400,000 certificates, all told, in a big SQL-queryable database. Want to see a total breakdown of certs by issuer? By verification code? Want to see the distribution of key lengths?  Or cipher suites?

Maybe you write software that processes certificates – want 400,000 real world examples to test against?  As far as I know, this kind of data hasn’t been available before without paying for it, so I’m actually really interested to know what you can do with it.

How Much of the Internet is this?

Good question! Other estimates I’ve seen for the total population of servers responding to SSL hails out there is about 1-4M.  Based on that, I’d say this is probably about 10-20% of the secure internet, but I wouldn’t try to use this data to make magnitude assessments; it’s better suited to proportions and comparative work, really.  If my estimates are close, then this is a big enough chunk of the total population to produce pretty good data.  Remember, it will exhibit the skews you’d expect from sampling the more popular sites, but that will also serve to weight it towards the certs people are more likely to see. If that’s not the bias you want, use a different list!

The Goods

• The database (gzip’d, sqlite3 format, 367MB) for my Jan 15 crawl.
• The code I used to gather it.  (It needs love, love it.)
• If you want to play with something a little less gargantuan, I’ve also put up a trimmed version with just the top 10,000 (gzip’d, 3MB).

If you find good stuff in here, I hope you’ll leave a comment letting me know.  If you can’t access the database or download the file and want me to run a query against it, let me know.  I only connect to each host once, and all I do is open an SSL connection, so the load on the servers is non-existent.  The load on my machine while running it though, can be heavy.  To keep the interruption to a minimum, I use conservative settings which make a crawl take about 40 hours, so until I have it running in parallel on a rack of excitingly fast machines, please understand that requests for re-crawls will take a while.

What is going on?

Basically for as long as there has been software, there have been nasty people out there who get you to download and install software which turns out to have hidden cargo.  Security folks use names like “virus,” “trojan,” “worm,” and “malware” to describe different types, but the point is that if a person can be tricked into running nasty programs, they can do nasty things.

In this case, rather than wiping your hard drive or turning all your icons upside down, this particular jerk has decided to mess with your Firefox. Once you run the program, it hooks into your Firefox and watches for you to visit certain sites, at which point it will steal your username and password.

How Can I Tell If I Have It?

You can open up your Firefox addons manager (Tools->Add-ons) and go to the “Plugins” section.  If you have a plugin called “Basic Example Plugin for Mozilla” you should disable it.

Original credit to TrustDefender Labs’ blog post on the subject

Does This Mean that Firefox is Insecure?

No, and here’s why:

• This particular malware targets our program, but once you have malicious software running on your system, it can just as easily attack other programs, or harm your computer in other ways.
• This isn’t contracted by just browsing around the web with Firefox 3. In fact, the Malware Protection features in Firefox 3 are designed specifically to prevent sites from being able to attack your computer.

The people getting infected here are either downloading enticing files that have the malware hiding inside (which is why Firefox 3 hands off all downloads to your computer’s virus scanner once downloaded) or, as some sites are reporting, people who have already been infected in the past having their computers forced to download this file as well.

Typical Firefox 3 users who avoid downloading software they don’t trust are unlikely to ever see this, and even the sites reporting it describe its incidence as “rare”.

What’s this I hear about GreaseMonkey?

There are some mentions of greasemonkey in a couple of the early reports based on some analysis of the code used by this malware, but I want to be clear that the (legitimate, and awesome) Greasemonkey Addon is not involved in this malware in any way. It is not involved in the installation or execution of the attack.

As always, the best defense is vigilance.  Use a browser with a solid security record and modern anti-malware defenses built in, and be very careful about downloading and running programs you find online.  If a bad guy is able to get you to run a program on your machine they will be able to do bad things, so we’ll keep trying to stop them and you keep trying to as well.

More details are also available on the official Mozilla security blog.

What Changed?

From a distance, without your glasses on, not much. We still spot situations that could signal a worrying lack of confidence in the integrity of the connection. We still use a content page to inform you of this, and to ask how you’d like to handle it. It sucks that we can’t make the call more automatically – that the attack situations are so hard to tell from the benign ones. What I’ve tried to do is at least make the experience more humane.

Let me mention 4 noteworthy changes:

• Leading with the human text, not the error code. The old pages were implemented using our generic error reporting framework, and that meant that things like technical details got top billing.With this rewrite, I’ve moved certificate errors off on their own, so that our users don’t suffer for our implementation woes. The technical details are still there, but they are now collapsed away until needed.

• Talking about the trust we have in the connection, not in the site. A certificate doesn’t attest to whether a site is trustworthy or not, whether its operators are scrupulous, or whether they have monkeys designing their shopping cart security. The real problem with these expired, mismatched, or self-signed certificates is that we don’t have any confidence that they are the “right” ones for securing your connection to the site.

  Maybe that sounds frivolous to you, but I have spoken with a lot of people who said they supported the action Firefox was taking, but not the way it made them seem unsavoury. I think we now do a much better job here.

• Making the choice clearer. Getting people to read errors is hard, but it’s completely impossible if we teach them that reading just makes things more confusing. I’m convinced that the text here makes the choices, and risks, a lot clearer than they were hiding behind the “Or you can add an exception…” hyperlink.

• If you click the “Add Exception” button on the new page, the dialog will automatically fetch the certificate for you, instead of asking you to manually click “Get Certificate”. This was always possible to set up by changing the browser.ssl_override_behavior preference in about:config, our secret-handshake background config panel, but it was not particularly discoverable back there.

  There were a couple reasons for Firefox 3 not doing that as well. There was some hope that having users interact with the dialog instead of just looking for the “whatever” button might be beneficial, but there was also a concern that pre-fetching the dialog would result in it being full of text as soon as users saw it, making it less likely that they would read it or understand the thing they were asking Firefox to do.

  In the final analysis though, I think the time to try to engender that understanding is before they hit that dialog. So that seeing the report there of the certificates problems is just a final sanity check for a site they already know to expect this from.

What Didn’t Change?

We still show the error pages. People often ask why we don’t just wave self-signed certs through unmolested (possibly hiding the security indicators) instead of blocking them.  As I mentioned when we introduced these pages, that punishes people with safe habits, and lets them get victimized by the man in the middle attacks that are getting much easier to pull off.

We still default to permanent exceptions.  Sometimes people ask why we don’t do temporary exceptions which, being a lesser amount of trust, ought to be a safer default.  The problem is, defaulting to temporary means that users see these error pages more often, which teaches them to ignore them.  It also eliminates the modicum of protection we build up by being able to detect when a trusted self-signed cert changes.  Adding permanent exceptions by default means that a typical user may not see this page more than 2 or 3 times in their life.  Unless, that is, they’re on public wifi and someone starts trying to intercept their traffic, in which case it will pop up surprisingly often.

We still respect the expert pref, for users that deal with hundreds of misbehaving hardware boxes, or otherwise have an atypical need to frequently interact with constantly changing untrusted certs.  If you are one of those people, going into about:config and setting browser.xul.error_pages.expert_bad_cert to true will tell Firefox that you are one of the edge cases.  Combined with the pre-fetch above, this means that a user can add permanent exceptions in the same 2 clicks that it took in Firefox 2, but with significantly more humane language around it.

Ehsan just landed his Private Browsing work, Shawn’s got some awesome site-specific data-clearing cooking, and this week I’m busting my ass a little to get some improvements into our Clear Private Data code as well. When it comes to managing your security and privacy online, Firefox 3.1 is going to be awesome.