However, when we decided to rebuild Firefox for Android using Native UI, we recognized that the first release couldn’t ride the trains. The iterative release model that serves us so well with Firefox works best when most changes are incremental and independent. Building a new high-performance front end for Firefox on Android, by contrast, involves many interconnected pieces being rebuilt in tandem.

Right now, the engineering team is focused on building an amazing browser for Android phones, and we’ll have a beta to show you in the coming weeks. It might coincide with one of our regular 6 week trains, but it’s quite possible it won’t. If it doesn’t, don’t worry. It’s cool. Firefox for Android will get back on the trains once the native UI rebuild is finished, but for a change this major we have extra work we want to do before we send it out the door. We’ll only ship it once we’re happy with its quality and performance. If you can’t wait that long, check us out on tablets or try our early release Aurora builds. I think you’ll be pleased.

[This post originally appeared on the Future of Firefox blog]

Last week we released a new version of Firefox. We shipped on time, 6 weeks after the last update, making it our first true rapid release milestone. There was cake. Now that we know that we’re capable of this velocity, I’d like to revisit the reasons why it’s important, and the lessons we’ve already learned.

Mission drives Mozilla. People sometimes forget that we’re a non-profit, that our only job is to make the Web a better place. Rapid release advances our mission in important ways. We get features and improvements to users faster. We get new APIs and standards out to web developers faster. We are delivering on the promise of the web at web speed.

Small, frequent releases improve quality, too. Engineers in the Mozilla community regularly say things now like “I don’t like not understanding this piece, let’s back it out and I’ll catch the next train.” We move deliberately. We don’t rush. And, even though it sounds like a contradiction, when we take our time we go faster.

There’s a great deal for us to be proud of, but we also need to be humble. This change was hard for us to make, and it’s been hard for some of our supporters, too. We have been glib or dismissive in the way we’ve communicated about parts of it. We live rapid release daily, and that makes it easier for us to see past the problems. We are also tenacious about the necessity of our new schedule, and tenacity can be mistaken for obstinacy.

We, everyone in the Mozilla community, all of us, need to communicate with clarity and sensitivity. We need to help the people who support our mission to understand why these changes are essential. We need to keep listening, and adjusting as we learn. We need to, and we will.

The push to ship faster isn’t some kind of software machismo. We push ourselves to ship faster because the web is under threat. Amazing and innovative people are doing amazing and innovative things and right now they have a choice: build for the web, or build for the walled gardens. The web can win that fight.

The open web is the most amazing, universal communication and distribution platform ever built. To win, the web needs to be agile and responsive. To help it, we need to be agile and responsive, too. That’s why rapid release matters.

I want to clarify an important part of the process, though, that I think many people haven’t yet understood. Remember, an individual release train is 6 weeks of development time followed by 12 weeks of stabilization:

New work doesn’t land on Aurora and Beta. Instead, those channels focus exclusively on working with our heroic and growing community of testers to spot any unexpected issues introduced during development, and then resolve them. Looking at this diagram, you might well conclude that we’d have a release ready every 18 weeks.

Aurora and Beta are so single-minded in their focus on stabilization and testing, though, that many engineers can move on to new work. If we take a step back and look at the broader picture, this is what actually happens:

During the 12 weeks that a release spends on Aurora and Beta, the Mozilla community is not sitting idle. They are already working on features and fixes for the next release, and the release after that. Every 6 weeks their work is picked up into the next Aurora, the next Beta, and the next release. When you look at this broader picture, you notice an important point:

There can be a new release of Firefox every 6 weeks, not every 12 or 18.

I’ll say it again, because it’s important: most of the time, we’ll release a new Firefox every 6 weeks.

Many people are surprised by this fact, though it’s been part of the process all along. When Firefox 4 came out, we committed to ship the next release of Firefox within 3 months. We did it, and when we did I think many people concluded that we have moved to a 3 month cycle. In truth, though, the only reason it took us 3 months was that our Aurora and Beta channels started off empty; they had to wait for the new release to make it through the process. The next Firefox is already in Beta, and is scheduled to come out 6 weeks after the last one. When that happens, yet another Firefox will enter Beta, and so on.

We’re studying the effects of the process carefully; it’s a big change and we will be flexible in our approach as new information comes in. We may decide that 6 weeks is the wrong interval, for instance, though it’s worth remembering that Firefox maintenance releases have been released on 6-8 week intervals for years, and sometimes included major changes. We’re also paying close attention to the impacts this cycle has on our ecosystem of add-ons, plugins, and other 3rd party software that interacts with Firefox. We’re working with large organizations, too, to understand how rapid release can fit into their software deployment systems.

Whatever adjustments we make, it’s clear that rapid release is a major improvement in our ability to respond to the needs of our users and the web. Every 6 weeks we have a new Firefox to evaluate and, unless some surprising and irreconcilable breakage is discovered, release to the world. No one will have to wait a year for the developer scratchpad now in Beta, or the massive memory and performance improvements already on Aurora, or the slick tab management animations soon to land on Nightly. Rapid release is already paying dividends, and we’re just getting started.

[This post originally appeared on the Channels blog]

We’re succeeding because we’re acting deliberately. We’re doing it on purpose. We know what we need our release process to do and we’re building forward from that, instead of shooting first and calling whatever we hit the target.

I’m glad that we’re being deliberate about how we build Firefox. We need to be deliberate about what we build, too. I’ll tell you how I think that should go, after a brief digression on how it has gone up until now.

(If you have no time for that, deb’s written a more concise introduction.)

Prehistory: Bugs

Historically, if you wanted to get a feature built, you filed a bug. Maybe you started a newsgroup thread, or an irc conversation but it wasn’t real until there was a bug. This is a great instinct. It makes the work concrete, it lets people opt in to tracking its progress, and it makes Bugzilla into a single source of truth about active work in the project.

Recent History: Prioritization

Bugs are good for tracking atomic pieces of work, but they are not good for overall project direction. As Mozilla grows, with more going on at once, we hit contention for scarce resources like code review, UX support, even permission to land at all, late in cycle.

Contention can be virtuous. It forces us to focus, but that focus is nearsighted. Managers ask their teams to work on high priority items, but priorities are often ad hoc (or post hoc!) Our releases have been driven by extremely smart people and they’ve been phenomenally successful, but the lack of a centralized, up-front plan and prioritization has lead to wasted effort and missed opportunities. It also bleeds those smart people dry, and we can do better.

The Future, In 4 Parts

We’re still going to use bugs to track work. We’re still going to need ideas from everywhere and we’re still going to need brilliant people, close to the code, letting us know what’s most important. What’s changing is that we’re going to bring some order to those ideas. It all starts with the feature page.

Part 1: The Feature Page

A feature page is a basic one-page summary of a piece of work. It’s written before implementation work begins, and brings all the information about a feature together, including:

• A summary of the feature’s motivation, goals and non-goals
• Bugs tracking work items
• Dependencies on other groups
• The feature team (Developer, UX, Security, QA, Project Management, &c) and overall owner
• Any UI designs, behaviour specs, test plans, and references necessary to make the implementation right, and usefully estimate its size

A feature page is more flexible than a bug for tracking these soft pieces. It introduces some maintenance cost for the feature owner, but the benefit is that every feature has a source of truth. Beyond the benefits this brings to engineering, it gives teams like QA, Security, Marketing, and Support a point source for the work they do for every feature we ship. Not having that source in the past has hurt us. Repeatedly, and significantly.

Part 2: Feature Tracking

When a new feature is picked up for work, the owner (not necessarily the main developer!) links to the feature from the tracking page. The tracking page transcludes each feature’s status line into an overall summary.

A given feature might move between releases on the tracking page as the realities of the feature’s size change; that’s fine. Having things on one page still lets us see at a glance what’s likely to make the next train, and the train after that. It helps us spot possible conflicts or danger spots earlier and lets people who care about different stages of the release focus their work. As this system starts to work, I can well imagine our development meetings re-orienting themselves to track this list.

Part 3: Feature Prioritization

Looking for a feature to work on? The product team is building a list with all of them. The feature list is, in a sense, the product team’s central deliverable: it maps our overall vision and priorities to a ranked list of feature work that embodies them. Start at the top of the list, work down until you find a good match for your skills and interests, poke the owner, and spin it up.

Part 4: Adding Features

The features on the list come from everywhere in Mozilla, as they always have. What’s new is that we collect them in one page; new features go into the inbox. Have an idea? Write up a feature page, add it to the inbox, and the product team will triage it into the right spot.

Of course, you can just do the work in a bug without any of this, but you do so at your own peril. Unlisted features have no priority, which means you’re back to contending for scarce resources. Writing the feature page is also an exercise in clarifying your thoughts, I don’t think the bar there is too high.

So to sum up…

I know, it’s a lot of words. And not every piece of work rises to the level of “Feature” and requires this overhead – straight up bug fixes rarely will, for instance. Nevertheless, I think the change is worth it. For me, ensuring we work on the right things, to say nothing of the benefits in terms of communication & coordination of work, are easily worth the cost. What do you think?

When someone leaves the Mozilla Corp, it’s tradition for them to send a note to our global alias saying so. Mike sent his late last week, and this was my reply.

Don’t you hate it when someone leaves and people start replying to all? Very spammy. Well listen here, folks: I’m the only other one here with “Director of Firefox” at the start of his business cards, I’m the other half of the travelling Canadian roadshow, and I’ve got something to say so I’ma say it, and you’re gonna read it and that’s just how that’s gonna go.

Johnathan

Ze Frank is a pretty creative guy, but what makes him really interesting to me is his ability to make other people creative. It’s what he does. He catalyzes creativity, frequently among those who don’t consider themselves creative. And when he talks about how he does it, he talks about the value of constraint.

Asked to go and “be creative,” he notes, most people shut down. So, instead, he asks for something more specific. He asked them to make a whole earth sandwich; they made a few. He asked people to send in pictures of vacuum cleaners dressed as people. He got 215. Constraining people, forcing them to solve a smaller problem, made them better at it.

Creativity isn’t the only thing that benefits from constraint. Asking engineers (or, really, anyone) for “an estimate” is basically akin to asking them to “be creative.” They know what examples of the thing in question look like, they understand that it’s a reasonable request, they just don’t actually know how to get there from here, much less how to be accurate about it.

Back in the sixties, NASA and the US DoD were spending a great deal of money on engineering. They therefore took a keen interest in improving planning and estimation, not unlike the interest you might take if someone was setting all of your money on fire. Out of this interest sprung the mellifluously titled “PERT/COST SYSTEMS DESIGN” which, on the subject of estimation, made this central observation:

If you ask engineers for 3 estimates (Best Case, Most Likely, Worst Case) instead of 1, you get different answers.

That’s pretty exciting! Constraints get us different answers, and different answers mean more bits of information. If you’re not convinced that this is brilliant, though, here comes some next level awesome: A (weighted) average of these 3 estimates is a better predictor of actual completion time than any one of them. Specifically

(Best + 4*Most Likely + Worst) / 6

turns out to work pretty well in the general case. These so-called “PERT Estimates” or “3-point Estimates” give engineers credit for their assessment of “most likely” by weighting it heavily, but still allow optimism and pessimism to pull the average. I dare you to argue with this graph:

Likelihood of project completion date vs estimates

Having 3 data points actually helps in other ways, too. It means you can more clearly quantify the uncertainty of a project by comparing best and worst case estimates, and watching to see if the distance between them shrinks over time. It means you can produce “optimistic” and “pessimistic” schedules. And, most importantly, it means that everyone is saying the same thing when they estimate.

Best, Worst, Most Likely. Try it for your next project, and see how it works. As we finish Firefox 4 and start looking at what comes next, there will be plenty of estimation happening, and I’m keen to see us bringing more science to the table. This may not be the right model for us, or we may discover that the coefficients need changing in our version of the equation; that’s fine. That would actually be a great result. My interest isn’t in pushing a particular tool, my interest is in getting better at planning, getting more awesome out to our users faster. I think we do that by looking for systems that have worked for others, and seeing how well they adapt to us.

And then we dress up the vacuum cleaners.

Problem: We have meetings.

What’s worse, we persist in having them every week. Being the kind of project we are, we keep agendas and notes from those meetings publicly and invite the community to participate (does your browser? Great!)

What you want, then, is for each week’s meeting notes to link to next week’s and last week’s, like such:

And so, we do. But those links have to be hand-edited every week. Indeed, the pages for various meeting notes have earnest, heart-wrenching pleas in HTML comments, like

<!– REPLACE YYYY-MM-DD with the previous and next week’s year-month-date –>

No one should have to live like that.

Solution: ParserFunctions

Our mediawiki install includes the ParserFunctions extension, which has a whole bag of tricks. One of these tricks is {{#time}}. #time lets you produce various kinds of time/date strings, to wit:

{{#time: Y-m-d }}

Particularly nice, though, is that you can specify relative times, e.g.

{{#time: Y-m-d|+1 week}}

The relative syntax is so flexible, in fact, that I can utter this monstrosity:

[[Platform/{{#time: Y-m-d|tuesday last week}}|« previous week]]

to link to last week’s notes from a given page!

Still with me? Because there’s one snag left. The above works for people who have a static front page with this week’s info, and only ever want to link one week back. But those relative dates are relative to now — what if I want each link in the chain to link to the week prior?

No problem — our pages are named according to their dates, so just make the link relative to that, instead:

[[Platform/{{#time: Y-m-d | {{SUBPAGENAME}} -1 week}}|« previous week]]

Presto.

The things you learn while waiting for a phone call. If you want to get really exciting, you can do all this in transclusion tags, to have last week’s notes automatically added to this week, but that’s left as a terrifyingly-recursive exercise for the reader.

What’s your favourite mediawiki hack?

(PS – Full credit to Melissa for giving me the idea in the first place. I am naught but the implementor.)

“When it’s ready” doesn’t mean we can take our time, though. Firefox 4 is good for the web, good for our users, and puts the heat on other vendors to up their own game. We need to ship it ASAP – we want release candidates in weeks, not months. And that means a hard look at our blocker list.

Blocker bugs have a rank order. If you can’t have all of them, there are some you’d want more than others, even though every single one of them is a bug we want to fix. That’s healthy. Building software means making those calls. Each bug is evaluated against whether it’s worth holding back the thousands of fixes that have already made it into the Firefox 4 tree. At this point, very few bugs are worth holding back that much awesome.

Hard vs. Soft Blocking1

To that end, then, if you watch bugzilla, you’ve seen blocker bugs sprouting one of two new whiteboard labels:

• [hardblocker] – These bugs prevent us from shipping. We’ll hold the release for the very last one of them. A hard blocker is a failure of a core part of our release criteria, e.g. a crash, a memory leak, a performance hit, a security issue, a UI breakage that can’t be recovered from, an incompatibility we can’t stomach.
• [softblocker] – These bugs are things we want to fix as soon as possible, but can ship with if the hard blockers are done. They can be fixed in maintenance releases if needed, or in Firefox 5 which, remember, is not so very far away. Soft blockers might include visual polish, strange edge cases, optional aspects of new specs, or opportunistic performance wins.

Hard blockers trump everything. That doesn’t mean they are the only things that will get fixed – indeed we hope and expect many of our soft blockers to make it in as well. We didn’t clear their blocking flags, they are still legit work items and have landing pre-approval. Soft blockers are what beltzner calls the “opportunity space” – the work that lifts the quality and delight of the product. But we have to make the hard calls, and soft blockers are second priority to shipping. People paid to work on Firefox will be focusing exclusively on hard blockers, first.

The hard blocker list is currently at 143. When it hits 0, we can ship. Let’s kill it dead.

[1] Inevitably, when we do a pass like this, someone will want to digress into a thread about nomenclature. “Why are they blockers if they don’t block?” “Are there hard soft blockers, or soft hard blockers?” I love the creativity of our community, but I think it’s a distraction right now, and I’d suggest to you that we have more interesting problems to solve in the next little while!

When I got back from my recent trip to visit Mozilla Online in Beijing, I heard myself saying that often, but it’s very nearly a content-free statement. Of course China is different. A better, albeit clumsier, way to express things is:

The Chinese web is not the web we are used to.

“We” Mozilla, “We” the Western tech world, “We” the builders of the web. China is going about things differently, and they’re bringing more than a billion people online with them. The folks at Mozilla online understand this and were exceedingly patient and generous with their time helping me begin to do so as well.

Here’s one way of thinking about that difference:

Piracy

To a first approximation, all commercial software in China is pirated. Pirated applications running on pirated Windows installs. In fact, massive cottage industries have sprung up around packaging, distributing, and even managing updates for pirated software.

User expectations

The state of pirated software and rampancy of malware is such that, as Mi Jia from Mozilla Online put it,

“Even novice computer users know how to reinstall their operating system. It’s something everyone does.”

There is no expectation of quality or stability, nor even a shared set of assumptions about what a browser should be.

The Wild East

These two factors conspire to create an anything-goes market for browsers in China. Here are a few examples:

• Last year a browser emerged that went from nothing to 16% market share in a year largely through auto-installs (the company has a popular update-management product, and drops a green IE-like icon on the desktop automatically.)
• Browsers are partnering with CDN-like entities to improve latencies and dodge expensive bandwidth costs and caps, particularly for international content.
• Many browsers are now “hybrids”, shipping with multiple rendering engines so that they can advertise modern features and performance, while retaining compatibility with IE-only sites. The trident engine they ship is often IE6.

How does Mozilla succeed on a web like this? With browsers seeing 20% marketshare swings from year to year? With a significant IE6 legacy and all the pain that suggests? With a new user class who aren’t accustomed to being able to demand high quality software of any kind, much less expect it to defend their interests and respect their choices?

In the short term, we use the freedom from expectation to innovate and experiment. We won’t win by finding our own short term tricks to force installs, but we can try out new features, and learn from the experimentation of others. In the long term, we do it the way we’ve always done. China is a very different web than the one Firefox was born into, but people are still people. Web users in China deserve the same web as the rest of the world. They deserve to browse safely, and to be able to trust their browser to act as their agent. We should build software that helps them use a free and open web, and we should teach them why those things matter.

It wasn’t so very long ago that IE was the only game in town, and we changed that one person at a time. We built a community of people who learned what it could mean to have a web browser built to protect them and their interests, and then told their family and friends about it. That shouldn’t have worked, it should have been somewhere between crazy and impossible, but we did it. We’re now the majority browser in some countries, and the web as a whole is better for it.

We need to do it again, folks. We need to bring a new, Chinese web generation into our community. We need to tell them what we think the web can be, and what they should demand from their browser, whichever one they choose. People are people, so how do we reach them? How did we reach you?

The EFF SSL Observatory is a project to investigate the certificates used to secure all of the sites encrypted with HTTPS on the Web. We have downloaded a dataset of all of the publicly-visible SSL certificates, and will be making that data available to the research community in the near future.

This is exciting. I knocked together a less ambitious version of this last year, but the EFF guys are doing it like grown-ups, and are getting some interesting data.

Numbers-wise, they’re in the right ballpark, as far as I can tell. Their numbers (1-2m CA-signed certs) coarsely match ones I’ve seen from private sources. I’ve heard from a few CAs that public-crawl estimates tend to err 50-80% low since they miss intranet dark matter, but at least the EFF is tracking other public-crawls. Given that their collection tools and data are going to be made public, that’s a really big deal. Previously, I haven’t been able to get this kind of data without paying for it or collecting it myself. If the database is actively maintained and updated, this will be a great resource for research.

Their analysis of CA certificate usage is also interesting. I’d like to see more work done, here, and in particular I’d like to see how CA usage breaks down between the Mozilla root store and others. We spend considerable effort managing our root store, and recently removed a whole pile of CA certificates that were idle. In some places, the paper seems to make the claim that fully half of trusted CAs are never used, but in other places, the number of active roots they count outnumbers our entire root program. I understand why they blurred the line for the initial analysis, but it would be swell to see it broken out.

As they mention, there are legit reasons for root certs to be idle, particularly for future-proofing. We have several elliptic curve roots, and some large-modulus RSA roots, which are waiting for technology to catch up before they become active issuers while giving CAs a panic switch in the case of an Interesting Mathematical Result — that feels okay to me. On the other hand, if there are certs which are just redundant, it would be great to know, so that we can have that conversation with the relevant CAs, and understand the need to keep the cert active.

This is exactly what I hoped would come of my crawler last year, but they’ve done a much more thorough job. We’ve seen an uptick in research interest in SSL over the last few years. Having a high quality data source to poke when testing a hunch is going to make it easier to spot trends, positive or otherwise. Interesting work, folks; keep it going!