Commentary


26
Aug 11

Rapidity

[This is a re-post of a post that originally appeared on the Future of Firefox blog]

Last week we released a new version of Firefox. We shipped on time, 6 weeks after the last update, making it our first true rapid release milestone. There was cake. Now that we know that we’re capable of this velocity, I’d like to revisit the reasons why it’s important, and the lessons we’ve already learned.

Mission drives Mozilla. People sometimes forget that we’re a non-profit, that our only job is to make the Web a better place. Rapid release advances our mission in important ways. We get features and improvements to users faster. We get new APIs and standards out to web developers faster. We are delivering on the promise of the web at web speed.

Small, frequent releases improve quality, too. Engineers in the Mozilla community regularly say things now like “I don’t like not understanding this piece, let’s back it out and I’ll catch the next train.” We move deliberately. We don’t rush. And, even though it sounds like a contradiction, when we take our time we go faster.

There’s a great deal for us to be proud of, but we also need to be humble. This change was hard for us to make, and it’s been hard for some of our supporters, too. We have been glib or dismissive in the way we’ve communicated about parts of it. We live rapid release daily, and that makes it easier for us to see past the problems. We are also tenacious about the necessity of our new schedule, and tenacity can be mistaken for obstinacy.

We, everyone in the Mozilla community, all of us, need to communicate with clarity and sensitivity. We need to help the people who support our mission to understand why these changes are essential. We need to keep listening, and adjusting as we learn. We need to, and we will.

The push to ship faster isn’t some kind of software machismo. We push ourselves to ship faster because the web is under threat. Amazing and innovative people are doing amazing and innovative things and right now they have a choice: build for the web, or build for the walled gardens. The web can win that fight.

The open web is the most amazing, universal communication and distribution platform ever built. To win, the web needs to be agile and responsive. To help it, we need to be agile and responsive, too. That’s why rapid release matters.


18
Jul 11

Every Six Weeks

It’s astounding to me, but we’ve been living rapid release for a few months now. We’re moving faster. A new feature implemented today and landed on mozilla-central can be delivered to our users in 12 to 18 weeks, not months or years. Incredibly, the same process that gives us that agility is giving us greater robustness, too. Testing and stabilization of each release across progressively larger audiences helps us find and fix bugs early, and build confidence in the quality of each release.

I want to clarify an important part of the process, though, that I think many people haven’t yet understood. Remember, an individual release train is 6 weeks of development time followed by 12 weeks of stabilization:

New work doesn’t land on Aurora and Beta. Instead, those channels focus exclusively on working with our heroic and growing community of testers to spot any unexpected issues introduced during development, and then resolve them. Looking at this diagram, you might well conclude that we’d have a release ready every 18 weeks.

Aurora and Beta are so single-minded in their focus on stabilization and testing, though, that many engineers can move on to new work. If we take a step back and look at the broader picture, this is what actually happens:

During the 12 weeks that a release spends on Aurora and Beta, the Mozilla community is not sitting idle. They are already working on features and fixes for the next release, and the release after that. Every 6 weeks their work is picked up into the next Aurora, the next Beta, and the next release. When you look at this broader picture, you notice an important point:

There can be a new release of Firefox every 6 weeks, not every 12 or 18.

I’ll say it again, because it’s important: most of the time, we’ll release a new Firefox every 6 weeks.

Many people are surprised by this fact, though it’s been part of the process all along. When Firefox 4 came out, we committed to ship the next release of Firefox within 3 months. We did it, and when we did I think many people concluded that we have moved to a 3 month cycle. In truth, though, the only reason it took us 3 months was that our Aurora and Beta channels started off empty; they had to wait for the new release to make it through the process. The next Firefox is already in Beta, and is scheduled to come out 6 weeks after the last one. When that happens, yet another Firefox will enter Beta, and so on.

We’re studying the effects of the process carefully; it’s a big change and we will be flexible in our approach as new information comes in. We may decide that 6 weeks is the wrong interval, for instance, though it’s worth remembering that Firefox maintenance releases have been released on 6-8 week intervals for years, and sometimes included major changes. We’re also paying close attention to the impacts this cycle has on our ecosystem of add-ons, plugins, and other 3rd party software that interacts with Firefox. We’re working with large organizations, too, to understand how rapid release can fit into their software deployment systems.

Whatever adjustments we make, it’s clear that rapid release is a major improvement in our ability to respond to the needs of our users and the web. Every 6 weeks we have a new Firefox to evaluate and, unless some surprising and irreconcilable breakage is discovered, release to the world. No one will have to wait a year for the developer scratchpad now in Beta, or the massive memory and performance improvements already on Aurora, or the slick tab management animations soon to land on Nightly. Rapid release is already paying dividends, and we’re just getting started.

[This post originally appeared on the Channels blog]


22
Nov 10

First Impressions from China

Great Wall in FogChina is different.

When I got back from my recent trip to visit Mozilla Online in Beijing, I heard myself saying that often, but it’s very nearly a content-free statement. Of course China is different. A better, albeit clumsier, way to express things is:

The Chinese web is not the web we are used to.

“We” Mozilla, “We” the Western tech world, “We” the builders of the web. China is going about things differently, and they’re bringing more than a billion people online with them. The folks at Mozilla online understand this and were exceedingly patient and generous with their time helping me begin to do so as well.

Here’s one way of thinking about that difference: Continue reading →


5
Aug 10

The SSL Observatory

Oh ho, lookit what the EFF went and did!

The EFF SSL Observatory is a project to investigate the certificates used to secure all of the sites encrypted with HTTPS on the Web. We have downloaded a dataset of all of the publicly-visible SSL certificates, and will be making that data available to the research community in the near future.

This is exciting. I knocked together a less ambitious version of this last year, but the EFF guys are doing it like grown-ups, and are getting some interesting data.

Numbers-wise, they’re in the right ballpark, as far as I can tell. Their numbers (1-2m CA-signed certs) coarsely match ones I’ve seen from private sources. I’ve heard from a few CAs that public-crawl estimates tend to err 50-80% low since they miss intranet dark matter, but at least the EFF is tracking other public-crawls. Given that their collection tools and data are going to be made public, that’s a really big deal. Previously, I haven’t been able to get this kind of data without paying for it or collecting it myself. If the database is actively maintained and updated, this will be a great resource for research.

Their analysis of CA certificate usage is also interesting. I’d like to see more work done, here, and in particular I’d like to see how CA usage breaks down between the Mozilla root store and others. We spend considerable effort managing our root store, and recently removed a whole pile of CA certificates that were idle. In some places, the paper seems to make the claim that fully half of trusted CAs are never used, but in other places, the number of active roots they count outnumbers our entire root program. I understand why they blurred the line for the initial analysis, but it would be swell to see it broken out.

As they mention, there are legit reasons for root certs to be idle, particularly for future-proofing. We have several elliptic curve roots, and some large-modulus RSA roots, which are waiting for technology to catch up before they become active issuers while giving CAs a panic switch in the case of an Interesting Mathematical Result — that feels okay to me. On the other hand, if there are certs which are just redundant, it would be great to know, so that we can have that conversation with the relevant CAs, and understand the need to keep the cert active.

This is exactly what I hoped would come of my crawler last year, but they’ve done a much more thorough job. We’ve seen an uptick in research interest in SSL over the last few years. Having a high quality data source to poke when testing a hunch is going to make it easier to spot trends, positive or otherwise. Interesting work, folks; keep it going!


13
Jul 10

Kathleen, a FAQ

Q: Kathleen who?

Kathleen Wilson works for the Mozilla Corporation, and manages our queue of incoming certificate authority requests. She coordinates the information we need from the CAs, shepherds them through our public review process and, if approved, files the bugs to get them into the product.

Q: Holy crap! One person does all of that? Is she superhuman?

It has been proven by science. She is 14% unobtainium by volume.

Q: That’s really awesome, but I am a terrible, cynical person and require ever-greater feats of amazing to maintain any kind of excitement.

She came in to a root program with a long backlog and sparse contact information, and has reduced the backlog, completely updated our contact information, and is now collecting updated audit information for every CA, to be renewed yearly.

Q: Hot damn! She’s like some kind of awesome meta-factory that just produces new factories which each, in turn, produce awesome!

I know, right? She has also now removed several CAs that have grown inactive, or for which up to date audits cannot be found. They’ll be gone as of Firefox 3.6.7. They’re already gone on trunk.

Q: Wait, what?

Yeah – you can check out the bug if you like. I’m not positive, but I think this might represent one of the first times that multiple trust anchors have ever been removed from a shipping browser. It’s almost certainly the largest such removal.

Q: I don’t know what to say. Kathleen completes Mozilla. It is inconceivable to me that there could be anything more!

Inconceivable, yes. And yet:

  1. She’s also made what I believe to be the first comprehensive listing of our root, with signature algorithms, moduli, expiry dates, &c.
  2. In her spare time, she’s coordinating with the CAs in our root program around the retirement of the MD5 hash algorithm, which should be a good practice run for the retirement of 1024-bit RSA (and eventually, in the moderately distant but forseeable future, SHA-1).
  3. She has invented a device that turns teenage angst into arable land suitable for agriculture.

Fully 2 of the above statements are true!

Q: All I can do is whimper.

Not true! You can also help! Kathleen ensures that every CA in our program undergoes a public review period where others can pick apart their policy statements or issuing practices and ensure that we are making the best decisions in terms of who to trust, and she’d love you to be a part of that.

Q: I’ll do it! Thanks!

No, thank you. That wasn’t a question.


11
Feb 10

Interview with a 419 Scammer

For those who haven’t seen it, scam-detectives.co.uk has a really interesting 3-part interview with a former Nigerian scammer.

Scam-Detective: A reader has asked me to talk to you about face to face scams. Were you ever involved in meeting a victim, or was all of your contact by email?

John: I never met a victim, but I was involved in a couple of Wash-Wash scams.

Scam-Detective: Wash Wash scams? What does that involve?

John: We would tell the victim that we had a trunk full of money, millions of dollars. One victim met some of my associates in a hotel in Amsterdam, where he was shown a box full of black paper. He was told that the money had been dyed black to get through customs, and that it could be cleaned with a special chemical that was very expensive. My associates showed him how this worked with a couple of $100 bills from the top of the box, which they rinsed with some liquid to remove the black dye. Of course the rest of the bills were only black paper, but the victim saw real money. He handed over $27,000 (about £17,000) to buy the chemicals and was told to return to the hotel later that day to pick up the cash. Of course when he came back, there was nobody there. He couldn’t report it to anybody because if it had been real it would have been illegal, so he would have gotten himself into trouble.

Part 1, Part 2, Part 3.

We build tools in Firefox like stale-plugin warnings and malware blocking to help protect our users, to neuter the technological attacks they may encounter on the web. But we also try, and need to keep trying, to build tools that inform our users so that they can make better decisions. Our phishing warnings and certificate errors try to do this, but mostly by scaring users away from specific attack situations. I hope we’ll continue to build tools like Larry which try to give people some affirmative context as well, to lend some nuance to their sense of place online. I want us to help our users know when they’re on Main Street, and when they’re in an alley.

I know: People get conned in the real world, too, and certainly no browser UI is going to save you from an email-based scam. Stories like this, though, are just specific instances of what I believe to be a more universal principle:

the biggest security risk most people face is misplaced trust

John: Some of the blame has to go to the victims. They wanted the money too because they were greedy. Lots of times I would get emails telling me that they wanted more money than I was offering because of the money they were having to send. They could afford to lose the money.

Scam-Detective: John, I think you have been basically honest with me so far. Please don’t stop that now. You know as well as I do that not all of your victims were motivated by greed. I have seen plenty of scam emails that talk about dying widows who want to give their money to charity, or young people who are in refugee camps and need help to get out. You targetted vulnerable, charitable people as well as greedy businessmen, didn’t you? You didn’t care whether they could afford it or not, did you?

John: Ok, you are right. I am not proud of it but I had to feed my family.

If you have ideas for how we can help users place their trust online more deliberately and carefully: please comment here, or build an addon, or file a bug.


27
Oct 09

Videos – Firefox Privacy & Security Features

Preamble (with Discussion Question)

I don’t know if there are people out there who like the way they sound in audio recordings, or look on video. I certainly don’t. I don’t think it’s a self-image issue, either; and I know I’m not alone. My recorded voice lacks the resonance I experience internally, and my recorded image just looks… mouthier (?!) than I imagine myself to be. I don’t even know what that means.

Proposed:

Nightingale’s Corollary to the Uncanny Valley Hypothesis: The depth of one’s psychological attachment to, and familiarity with, one’s own image, amplifies feelings of canny/uncanniness. This can result in greater than average affinity for moderately dissimilar representations (c.f. the popularity of “realistic cartoon avatar” generators, or caricature artists), but also particularly heightened sensitivity to minor dissimilarities.

[Discuss. Cite examples.]

The Point (i.e. Where You Should Have Started Reading)

I bring this up because the inimitable duo of Alix and Rainer recently took some of my scattered ramblings and knit them together into an educational piece on some of the security features in Firefox. I think they did a lovely job:


YouTube

In very much related news, Drew worked with Alix and Rainer to put together a video that talks about some of Firefox’s privacy features. I find it much easier to listen to Drew’s calm, matter of fact, “we did awesome stuff, and want you to know about it” delivery. I suspect you will, as well.


YouTube


4
Sep 09

Deletion

To a first approximation, I think you can gauge how much people think about software quality by how highly they value deletion. While most rookie developers are chiefly interested in building rather than in tearing down (for what I hope are obvious reasons), great throbbing brains like Graydon speak about deletion with the kind of reverence that I presume cardinals reserve for only the coolest of popes.

In what history will likely judge as a vain attempt to impress him, then, I recently landed bug 513147, deletion of the now antiquated “Properties” dialog that used to be available on right-clicking things like images and links. Not because it was useless (every feature is someone’s baby, and is added for a reason) but because it wasn’t useful enough, to enough people, to justify the cost.

50kb of code in our product that is poorly understood, not often used, and not covered by unit tests is not free. When bugs show up, it takes longer than it should to fix them. If a security bug were to show up (which is always a risk when content mixes with chrome, however remote it may seem) it would be particularly expensive for us to reload that context into our brains to fix it.

Deleting it isn’t free either, of course – there are 4 extensions that build off that dialog that will need to be updated, and there may be some who use it regularly who will be disappointed. But the forces of software (inertia, squeaky wheels, cynicism and inertia) bias so heavily towards keeping code in the tree that we should all try to take clear deletion opportunities when they come up. Not capriciously, not without sensitivity to the impact it can have, but with recognition that the hidden cost to keeping them is also large and… hidden.

It is in the spirit of this sensitivity that we, on the Firefox team, have tagged this bug and others like it: [killthem].  What else do you think should go? (And please, be gentle. Remember, every feature is someone’s baby.)

[Update: Geoff Lankow has taken the code that used to be built in, and made it into an add-on, which is think is fantastic. As I said to him, and as I said above, my assertion has never been that the code was useless, just that it wasn't useful enough to justify its cost in the core product. An add-on is a great place for functionality like that, and I thank Geoff for his work.]