SSL Infoquickie (with Bonus Firefox Pro-Tip!)

There is less public information out there about SSL certificate usage than one might like to see. Netcraft has a for-pay report with some interesting figures, and occasionally makes some of that data public, and I’ve blogged about other sources in the past, but in general, it’s pretty sparse. I keep meaning to do something coordinated about that, I have some ideas, but they keep getting back-burnered.

So it came to pass that when someone idly remarked that it would be nice to know what percentage of certs on the top sites were valid, I pounced upon it as a way to quickly release some pent-up info-gathering angst.

It’s profoundly unscientific, but so was the question. Are the Alexa top 500 sites even a good reflection of the most popular SSL sites? Not really. I think it will bias the data towards higher counts of untrusted certs (since the admins aren’t expecting them to be used) and towards lower overall cert counts (since many of those sites won’t answer SSL hails, whereas presumably a list of the top 500 SSL sites all would). Is blindly connecting to their main page on port 443 the best way to harvest their certs? Probably not, lots of them use secure.frobber.tld constructions, so that will also bias the data lower. Let’s just agree that it’s a sort of fun number to have as an order-of-magnitude style signpost.

Of the 500 top sites on Alexa, October 15, 2008:

  • 217 responded to an SSL query on port 443
  • 199 of those replies used valid certs chaining to trusted roots
  • The other 18 were a mix of self-signed, bad chains (likely from trusted roots, though I didn’t investigate), and expired certs.

If you prefer pretty pictures:

SSL Certificate Stats

Any conclusions you want to draw from this data will be only as good as the aforementioned biases within it, but don’t say I never do anything for you in a feeble attempt to vent my own info-lust urges.

Bonus Firefox Pro-Tip: If you are on Firefox 3.1 Nightlies or the upcoming Firefox 3.1 Beta 2, you now have the ability to turn off link-visited colouring.  David Baron recently landed a fix for bug 147777 that adds a new about:config preference to control the behaviour, layout.css.visited_links_enabled.

“Great!” I hear you all saying, “We’ve been hoping for a way to turn off an occasionally useful feature!”

And who hasn’t, really? But the thing of it is that colouring links can give away information to tricky sites about where you’ve been. It’s up to you whether you think that privacy/functionality trade-off is worth making, and the bug is still open while more universal solutions are contemplated, but in the meantime, you have the choice.

Why Did Talos Eat My Face?

Shawn has a great post up about some detective work we’ve been doing this week.  He gives me a bunch of credit, but I am convinced that I did the easy parts – he’s the one making our SQLite happier, so send him the laurels.  Nevertheless, Dave asked if I could write down the basic process used to narrow down the regression, which I agreed to do, because I am a sucker.

Back Story

So Shawn has been trying to update our version of SQLite for some time now.  Updating it fixes half a dozen crasher bugs, and should net us some modest perf wins in some places too.  But three times he’s tried to land it, and three times he’s had to back it out because of a sizeable performance regression on linux.  Specifically, Ts was going from about 1550ms to about 2500ms.  I wouldn’t call that a 65% regression or anything, but I would be pretty comfortable calling it a 61.2% regression so, you know, holy jumping tree squirrels.

The thing is, there was no meaningful hit on Windows or Mac, and even on Linux, Shawn didn’t have much of an idea where to start.  I (as sheriff for the day) didn’t want that sitting in our tree, but understood Shawn not wanting to back it out without any notion of whether the regression was even real, let alone what was causing it.

Enter Standalone Talos

As is so often the case, Alice came to the rescue.  A while ago she put together the standalone talos package, and I keep being happy that she did.  Running talos on your local machine is a bit of a noisy way to gather performance data (depending on the test) but it’s a great way to make your Firefox do exactly the same thing as the real talos does, for the purposes of profiling.  Which is what we did.  And it worked.

How We Dissected The Problem in Six Easy Steps, Only Two of Which Involved Cussin’

  1. Follow the standalone talos set up instructions to get your own local talos setup and pointed to a build without the offending patch.
  2. Edit the sample.config file, drop down to the bottom of the file, and delete every test you don’t care about, since a full talos run is a lot of work.  Tp, in particular, is not a fast test.
  3. Run talos with your profiler of choice.  In our case, the venerable strace. Our command looked like this:
    strace -T -o without-patch-strace.log python run_tests.py sample.config
  4. Swear, because your log is way too short, because you weren’t tracing the child processes, and hence were presumably watching some python crap. Try again with:
    strace -Tf -o without-patch-strace-AGAIN.log python run_tests.py sample.config
  5. Swear, because now you have a gigantic log with everything interleaved stupidly. Read the damned man page this time and come up with the -ff argument to generate a separate trace file for each pid:
    strace -Tff -o without-patch-strace-OMG.log python run_tests.py sample.config
  6. Repeat step 5 after pointing Talos at a build with the patch (and presumably changing the output filename).

This will leave you with a large number of trace files, but it’s pretty easy to recognize the ones you care about.  In Shawn’s case, he just grep‘d out the ones that referenced places.sqlite, and behold, he had before and after traces of precisely the same startup sequence that talos was seeing.

From there it was a short trip to see a massive increase in fsync calls, something we did not expect, but which certainly explains the lag, given the problems we’ve had historically with fsync on Linux.

Total execution time, before and after
Total execution time, before and after

Needless to say, Shawn is on it.

SSL Question Corner

From time to time, in the blogosphere or mailing lists, I will get questions about various security decisions we make in Firefox.  Here’s one that has been popular lately:

Q: I think you are dumb.

It is worded in a variety of ways, of course, but that’s the basic thrust.  A longer version might read:

Q: Why has Firefox started treating self-signed SSL certificates as untrustworthy?  I just want encryption, I don’t care that the cert hasn’t been signed by a certificate authority, and anyhow I don’t want to pay hundreds of dollars just to secure my communications.

There are a couple of implicit assumptions we should dispense with up front, before tackling the meat of the question, to wit:

  1. “Why has Firefox started treating…”  Firefox has been treating self-signed certificates as disconcerting for quite some time.  In Firefox 2, you would get a giant dialog box popping up asking what to do with them.  It was farcically easy to dismiss since just hitting OK would proceed to the site, and since the default was a temporary pass, not a permanent one, you saw the dialog frequently, making it even easier to ignore.  Firefox 3 has absolutely changed that flow — more on that later — but there is nothing new here.
  2. “ … I don’t want to pay hundreds of dollars …” Several CAs accepted by all major browsers sell certificates for less than $20/yr, and StartSSL, in the Firefox 3 root store, offers them for free.

Those concerns are red herrings, the real concern is in the middle:  “Why treat self-signed SSL as untrustworthy?  I just want encryption.”  Let’s explore this.

First of all, this isn’t quite right.  You never *just* want encryption, you want encryption to a particular system.  The whole reason for having encryption is that you don’t want various ill-doers doing ill with your data, so clearly you want encryption that isn’t going to those people.

“So fine, I want encryption to a particular system,” you say, “but I don’t need a CA to prove that my friend’s webmail is trustworthy.  CAs don’t even do that anyhow.  I trust him, Firefox should get out of my way.”

Yes, absolutely – the browser is your agent, and if you trust your friend’s webmail, you should be able to tell Firefox to do so as well.  But how do you know that’s who you’re talking to?

Permit me 3 short digressions…

Digression the First: Ettercap, webmitm, and friends

What if I told you that there were a group of programs out there that made it trivial, brain-dead simple, to intercept your web traffic, log it, and then pass it through without you ever noticing?  These “Man in the Middle” attacks used to be the stuff of scary security fiction, but now they are point-and-click.

If one of these is running on your network (you know, like the packet sniffers you’re protecting against with encryption in the first place) it will poison your network so that all requests go through them.  It will then transparently fetch and pass off any regular web pages without you noticing (after logging anything juicy, of course).  If you request an SSL page, it will generate its own certificate whose human readable details match the real site, same organization name, same domain name, everything, and use that to masquerade as the site in question.  The only difference is, it will be self-signed, since the tool obviously can’t get a CA signature.

Digression the Second: Drive-By Router Reconfig

Do you use one of those home cable-dsl-router/wifi-access-point thingies?  For the last couple years, security folks have gotten giggles out of finding ways to break them, and the number one thing they do is rewrite your network configuration so that your connections go to computers of their choosing.  If your router is subverted in this way, the only hint you might have is that your secure sites have all become self-signed.

Digression the Third: Kaminsky Breaks the Internet

This week I’m at the Black Hat security conference in Vegas, where it is a virtual certainty that Dan Kaminsky is going to outline an attack that lets any site on the internet pretend to be any other site on the internet.  I can pretend to be paypal.com.  You can pretend to be bankofamerica.com.  If your ISP doesn’t fix all of their servers, one aforementioned doer-of-ill can trick them into sending all of their customers to forgeries of the actual sites they seek.  They don’t even have to be on the same network anymore.  This is substantially easier than packet sniffing. The only thing that will tell you whether the sites you are visiting are real is the existence of a trusted certificate, which only the legitimate site can have.

Back to the Plot

The question isn’t whether you trust your buddy’s webmail – of course you do, your buddy’s a good guy – the question is whether that’s even his server at all.  With a CA-signed cert, we trust that it is – CAs are required to maintain third party audits of their issuing criteria, and Mozilla requires verification of domain ownership to be one of them.

With a self-signed certificate, we don’t know whether to trust it or not.  It’s not that these certificates are implicitly evil, it’s that they are implicitly untrusted – no one has vouched for them, so we ask the user.  There is language in the dialogs that talks about how legitimate banks and other public web sites shouldn’t use them, because it is in precisely those cases that we want novice users to feel some trepidation, and exercise some caution. There is a real possibility there, hopefully slim, that they are being attacked, and there is no other way for us to know.

On the other hand – if you visit a server which does have a legitimate need for a self-signed certificate, Firefox basically asks you to say “I know you don’t trust this certificate, but I do.”  You add an exception, and assuming you make it permanent, Firefox will begin trusting that specific cert to identify that specific site.  What’s more, you’ll now get the same protection as a CA signed cert – if you are attacked and someone tries to insert themselves between you and your webmail, the warning will come up again.

I don’t think the approach in Firefox 3 is perfect, I’m not sure any of us do. I have filed bugs, and talked about things I think we could do to continue to enhance our users’ security while at the same time reducing unnecessary annoyances.  You’ll notice that Firefox 3 has fewer “Warning: you are submitting a search to a search engine” dialog boxes than Firefox 2 did, and it’s because of precisely this desire.

I welcome people who want to make constructive progress towards a safer internet and a happier browsing experience. That’s what motivated this change, it’s what motivates everything we do with the browser, really.  So it sure would be nice if we didn’t start from the assumption that changes are motivated by greed, malice, or stupidity.

The Most Important Thing

Microphone by hiddedevries on flickr… or How Mozilla Does Security and What You Can Steal

As promised last week, I have now put my presentation slides for my talk at FIRST2008 online.  I’ve also put up a video I recorded of a dry-run through the slides, in case you want to experience the talk, and not just read it.

Slides (CC-BY-SA):

Video (CC-BY-SA):

Thanks again to Mike Shaver for helping me put these slides together, and to all the people who reviewed them ahead of time.  I really enjoyed this talk, and hope to give it again – as I’ve said many times before, we have learned a lot of lessons the hard way; we should be sharing that experience broadly, since we’re one of the few organizations that can.

I would love any edits or suggestions for the slides themselves, or my presentation of them.  I’ll also accept offers of exciting cash and prizes to give this talk at your campus/company/private island.

How to Make Good (Beer) Bread

A lot of people have asked for this recipe, and I keep promising to write it down, so here goes. You can use any kind of beer, and get a wonderful variety of colours and flavours, but I tend to prefer it with dark ales; they give the bread a darker crumb like a rye bread, and have a really malty, yeasty flavour that I like.  Really hoppy beers do a totally different thing, you should experiment.

I can’t take any credit for this at all – it’s how the French have made bread for 500 years, I suspect (albeit more often with water than beer.)  It takes about 15 minutes of active work (most of it up front), 30 minutes of baking (all of it at the end) and hours in between when you go and do other things.

Ingredients:

  • 1lb flour (white, whole wheat, whatever turns your crank.  I have pretty good success starting with about 10-14oz unbleached white flour, and topping up with whole wheat, but going all white flour is the classic french bread recipe) plus some for working.
  • One 12oz bottle of beer (or, I suppose, 12oz of water).
  • 2tsp salt
  • 1tsp instant yeast (if you buy the traditional yeast instead of instant, I presume you already know how to activate it)

Method in Brief:

  • Combine dry ingredients
  • Stir in beer
  • Knead
  • Cover, let rise for at least 1 hour
  • Punch down mixture, recover.  Let rise for at least 2 hours or refrigerated overnight
  • Pre-heat oven to 450F
  • Form dough into loaf, slash top, dust lightly with flour
  • When dough is placed in oven, spray oven surfaces with water, or throw in two ice cubes
  • Bake for 25-30min or until the crust is nicely browned.  Let stand for at least 15 minutes.

Method with Narrative (and an explanation of the ice cubes):

Mix the flour, salt, and yeast in a bowl that you think is too big.

Pour in the beer.  It helps if the beer is not ice cold, since that will really slow down the yeasties.  If you didn’t think to take the beer out earlier, no worries, just run it under a warm tap until it’s no longer cold to the touch.  This is not an exact science, nor should you treat it as such.

Mix with a spoon until the ingredients are combined.  This will be a really sticky dough.  It will make you sad the first couple times, until you learn to recognize the goodness it portents.

Sprinkle some flour on a countertop or other largish surface, and dump the dough out on to it.  Remove any rings or watches.

Now you’re going to knead the dough.  If you’ve never kneaded before, it’s easy.  Your job is basically to keep mooshing the flour molecules past one another so that there is ample opportunity for them to link up into chains called “gluten”.  Gluten is what gives bread its elasticity.  If you are not accustomed to thinking of bread as “elastic”, think about how a slice of bread deals with mashing and stretching (i.e. by mashing and stretching) vs. how a slice of cake does (i.e. by crumbling).  Flour mixtures all tend to form gluten.  Things like kneading help it out (which you do it with bread doughs and not cake doughs), things like fat hinder it (which is why you add things like shortening to cake – so-called because it “shortens” the dough – breaks up the gluten chains).

Kneading is also a nice analog process for getting the flour:beer ratio right, since your natural stickiness aversion will tend to have you adding sprinkles of flour as you work the dough and the surface becomes sticky again and again.  To knead, use your fingers, knuckles, or palms to stretch the dough out along one axis, then fold it over on itself and repeat.  When it gets too sticky to work with (try to keep it as sticky as you can stand) add more flour.  You don’t have to do this for long, 5-10 minutes is probably fine.  When you’re done, you’ll see a difference in the dough ball: if you stretch it a little, it will bounce back mostly into shape.

You’re basically done the hard work.

Throw the dough ball into a big bowl.  If you’re clever, you will have really lightly oiled the bowl first (not like a muffin pan or anything, just a shot of Pam, or a dab of canola oil swooshed around on a paper towel) because it will make the dough easier to remove later.  Throw a dishcloth over the bowl and leave it somewhere warm in your kitchen to think about life.  If your oven is a newer one with a “Proof” setting, now is when you can use it.

You don’t want to cook the dough here, you just want the yeasties to be at a happy temperature.  In case you weren’t clear, yeast are little, edible, live fungi that eat sugars in flour (among other things) and leave alcohol and carbon dioxide gas as well as a host of mostly nice-tasting things in their wake.  The alcohol is mostly incidental for our purposes here, though there is a delightful symmetry in the fact that we’re mixing beer (grains + yeast + water = alcohol and incidentally CO2) and bread (grains + yeast + water = CO2 and incidentally alcohol).  Point is, these guys will work through the flour making little bubbles as they go, which give our bread the ability to rise.  Very exciting.

After an hour or two, your dough will have doubled in size.  The only problem with cooking it right now is that your bubbles will not be evenly distributed.  You’ll probably actually have a couple giant bubbles that will lead to silly looking bread.  And anyhow, that’s not actually the only reason – giving the yeast more time also lets them develop more interesting flavours.

What you CAN do after an hour (or two, or three, this is not an exact science) is what bakers call “punching down.”  You can leave the dough in the bowl, but basically what you want to do is re-distribute the bubbles through the dough, and bust up any big ones.  Just rotate the bowl around, folding the edge back towards the center until you’ve got a ball again without any obvious giant bubbles.  It will lose some of its newfound volume too, that’s okay.  The yeasties still have plenty to work with.

What you do now is up to you.  You could wait another hour and bake it and have tasty bread.  You could go out for the afternoon and then bake it for dinner and have very tasty bread.  Personally, I do this on a Saturday, with an eye to baking it on Sunday, so I give it the rest of the day to rise, and then I’ll generally punch it down a second time and put it in the fridge overnight.  By Sunday dinner, I have me some outstandingly tasty bread.  Again, I’m not taking credit, the recipe is as old as the hills.  But ask around, my Sunday night bread kicks ass.  Anyhow, time passes.

Your dough should be room temperature when you go to bake it which is trivial unless you’ve gone for the (highly recommended!) overnight rise in the fridge.  Why not just leave it to rise overnight, out of the fridge, I hear you ask?  By all means, give it a shot.  It will rise a lot, and the surface will feel like silk, and it will be nearly impossible to handle (think about trying to form a loaf out of steam, say).  But I admire your moxie.

Every time you manipulate your dough, some of the bubbles will collapse.  They’ll come back as long as there is anything else for your yeast to eat (which there will be), so don’t worry, but it does mean that you don’t really want to be doing any dough manipulation immediately before baking.  I deal with this by forming the loaf and putting it on a sheet of parchment for the final rise, since parchment can go directly into the oven.  Pro-tip: wax paper is not parchment.

This is a French bread recipe, so you want to bake it on a stone or, failing that, a cookie sheet.  You don’t put this in a loaf pan.  That means that you have considerable flexibility on the size and shape of your loaf.  You can sort of just plop your bread ball down and make a round “boule” loaf, or you can form it into the standard french-stick ellipsoid.  The only trick here is to try to ensure that the “skin” of the loaf, the soon-to-be-crust, is stretched nicely instead of lying slack.  To do that, as you’re shaping it, curl the sides of the loaf down under itself, so that the top skin stretches.  This is easier to do than to describe – you’ll get the hang of it.

Heat your oven to 450F (convection ovens are yay, and will generally be smart enough to re-interpret that as 425F since they do a better job of baking.  If yours doesn’t, help it out.)  If you have a baking stone, it should obviously be in there too.  When the oven is hot, right before putting the bread in, slash the top a few times and dust it with a little more flour.  Not only does this give it a classic look, but it also lets the bread rise more evenly as the bubbles expand in the heat.

Right after you put it in the oven, throw in a few ice cubes.  Not on the bread, just anywhere in the oven.  The ice cubes are key. They will form steam, and the steam will condense on the dough, since it is cooler than the air in the oven.  In the process, they will give up some heat to the surface of the dough.  This makes for very happy crusts, and is the difference between people thinking you cooked bread, and people thinking you are super-awesome.

5 minutes in, you can throw a few more ice cubes in, to finish the job. Be careful when you open the oven door, that steam is going to try to have a party on your face.

Total baking time is about 25 minutes, though I generally have to give whole wheat dough a little bit longer to get a good crust going.

Take it out when it’s ready, put it on some kind of rack to cool.  If you put your ear up next to it, you will hear the signature sound of bread – a crackling as it cools, that makes my mouth water just thinking about it.

Cut thick slices.  Use real butter.  Marvel at how something so easy can taste so good.  I know there’s a lot of words there, but by the third time you do this, it is easier and more rewarding than just about anything else you’ll do in your kitchen.

Security Screencast(s)

As Alix mentions, I recently put together a quick screencast of some of the new security features in Firefox 3. Of course, beltzner promptly scooped me with his own inimitable screencast, and what with the launch, it’s only now that I’m getting around to posting mine.

What’s interesting to me, though, is the difference between what I originally recorded, and what Alix published. I recorded the raw screencast using Jing, which is a simple, free screencasting tool for Mac and Windows. It caps you at 5 minutes, and records as flash, but it’s super easy to use, and screencast.com will host the resultant video for you. You can see what I recorded here:

http://content.screencast.com/bootstrap.swf

But then I handed it off to Alix and David and Rainer, and they turned my 5 minutes of low production values into 2 minutes of edited, titled video, with helpful visuals! See if you notice the difference…


Firefox 3: Security from Mozilla Firefox on Vimeo.

As promised in my last post, I’ll soon be posting yet another video, this time an hour long talk I gave at FIRST. And then, I think, no more blatant self-promotion for a couple weeks, eh?

Have you installed Firefox 3 yet?

Hello Vancouver! Briefly!

A quick note, to any Vancouverites that may be interested, that I will be in town on Wednesday to speak at the FIRST 2008 conference. The title of the talk is “The Most Important Thing – How Mozilla Does Security, and What You Can Steal.” If you’re attending the conference, I hope I’ll see you there. Once the conference is over, I’ll post my slides and a video of a presentation dry-run, in case anyone is interested.

I had a lot of help from several people, most notably Shaver, in putting this presentation together; my goal is to keep adapting it and ideally get other people giving it as well. Security is something that the Mozilla project has a lot of experience with, and a lot to be proud of. It is important to our mission that we share that expertise. Even when what we’re saying isn’t new (“have unit tests”), the fact that we have achieved the success we have lets us be a proof point for people trying to make change in their own projects (“Mozilla didn’t think code review was too time-intensive.”)

I may not be an official member of the evangelism team, but I will do whatever I can to encourage more people in our community to take their knowledge outbound. We are doing crazy awesome stuff here (how many IT people, on the planet, have dealt with what Justin‘s team has?) and we should consider it an obligation to spread that knowledge around. Heck, that’s actually sort of what my talk is about.

Firing Up Browser Security

Low Flying Dogs on FlickrWindow and I recently did a joint interview for Federico Biancuzzi at SecurityFocus about many of the security changes we’ve made in Firefox 3. It covers both front-end and back-end information, and mentions several changes that I haven’t had a chance to mention here in the past.

If you’re interested, check it out.

[PS – Full props to r80o on flickr – this is a pretty excellent photo for “caution”, and CC too!]

It’s Ready.

Victory PosterFirefox 3 will be released Tuesday, June 17th.  This is indescribably exciting for us here at Mozilla, we are all quite giddy in our excitement places.  If you would like to share in this joy, I recommend the following 5 step regimen:

  1. Go read the Deb Richardson’s unbelievable Field Guide to Firefox 3, so you can see what all the hubbub is about.
  2. Go watch Mike Beltzner’s magnificent-yet-brief Intro to Firefox 3 features screencast, to set your pots a boilin’.
  3. Go view John Slater’s profoundly triumphant post of the new Firefox 3 Victory Poster, to adorn your walls and windshields.
  4. Go sign up yourself to participate in the life-alteringly spectacular Download Day, you one best chance to be part of a Guinness World Record that doesn’t involve putting bees up your nose.
  5. Go find yourself a party full of people every bit as frothy as we are.

Can I get an “Amen!” people?