9 months of rewrite. In web time— in mobile web time— that’s years. Rewrites are almost always the wrong call. You get to throw away bad code, but you throw away good code, too. Hard won, battle-hardened fixes are expensive to throw away, and the code is never cleanly labelled “baby” and “bathwater.” The decision to rewrite Firefox was one of the loudest weeks I’ve had. Even once we’d decided, my hand hovered over the send key for a long time.

3.5 stars. We didn’t pull the old version from the market during the rewrite. We wanted to keep those users safe with regular security updates; that meant keeping the product online. It meant taking black eyes every day from users who tried us, who loved us for our desktop product, and who were disappointed. And because the Android rating system rounds, our already painful 3.7-star rating rounded down to 3.5. We were dead last.

7:1. If you want to maintain a 4.5 star rating, it takes seven 5-star reviews to counteract each 1-star review. Star ratings are deeply flawed for all kinds of reasons: selection bias, survivorship bias, false dilemmas, unidimensionality, reporting bias, et cetera. I know this. And still I watched them. When you try to reach people through an app store, your star rating is the first assessment of your product they’ll see. And when you try to make something great, reviews are real pieces of feedback from real human beings and they are painful and they are frustrating and they are golden.

1181 reviews in the first week. 588 of them were 5-stars. 105 of them were 1-star. I argued with my screen over 1-star comments about bugs we had already fixed. I swooned at 5-star reviews that said they were reversing earlier 1-star reviews.

6 weeks after beta, we pushed to release. By this point beta had climbed to 3.8, which rounds to 4 stars. Fists were pumped. Release sat at 3.6.

Time passed.

200,000 reviews and 6 releases later, this week, today, Firefox and Beta both show 4.5 (rounded) stars in the market and the team is still going strong. I’m immensely proud of the work they’ve done. It’s made me reflective and maybe a bit wordy. I want to have some profound and pithy lesson to separate then and now. Something that I can package up and hand to you. We certainly learned lessons, profound lessons, but in repeating them they sound trite:

Listen. Care about what people think. Be hungry for feedback. Don’t work forward from the tech you have to the product you can build; work backward from the product your users deserve to the tech you’ll need to get there. Ask for help, and accept it even when it hurts to admit that you need it. Don’t throw things away lightly, but be able to throw them away in those rare cases where it’s necessary. Surround yourself with the most excellent people you can find. That one helps a lot.

The haters will say I over-focus on star ratings. There are certainly lots of other good things to measure, and lots of bad things to say about stars. But the stars abide. They are inescapable. They sit front and center in your primary distribution channel. Those five little stars. And I believe they do say something about what we’ve built.

Building good things is hard and my hat is off to anyone who earnestly tries. If the good thing you build reaches people through an app store, I want you to know that I know how you feel. And I’m rooting for you.

[I posted a version of this post on Medium, because it's fun to try new things.]

For Christmas last year I bought my brother a horse head and, while I was at it, picked one up for the office. It felt like the kind of thing that Mozilla Toronto would enjoy.

Ah, Monday. #mocoto twitter.com/johnath/status…

— J Nightingale (@johnath) December 17, 2012

Good times.

Between meetings we needed a place to store it; the sad, flat way it sort of collapsed when left on a desk was unsatisfying. So Madhava and I went out in search of a head on which to mount it. Steps from our office is a mannequin supply store, which helped.

Cut back to midday, @robcee, when the plan was still forming. /cc @johnath twitter.com/madhava/status…

— madhava (@madhava) January 22, 2013

I put the horse head on its styrofoam mount and, for giggles, set it up in the window. Facing outwards. Staring at the office in the next building.

For 2 weeks it sat there and I was pretty happy with it. And then something happened. Something wonderful and magical happened. I wasn’t there for it, but lmandel and overholt were.

The other office responded.

They taped a note to the glass.

“What’s with the horse?”

We responded.

“Who you callin’ a horse?”

They responded.

“Why you, of course!”

Ongoing inter-building horse head-related discussion @mocotooh #Mozilla #Toronto twitter.com/overholt/statu…

— Andrew Overholt (@overholt) February 15, 2013

And so it went. Back and forth.

Stick with me, I’m getting to the world changing.

So last week we bought them their own horse head. And yesterday mconley delivered it to their office. They invited him in. They brought the box over to the window where we could see. They gathered around their window and we gathered around ours. And when they opened it and realised what it was they actually jumped up and down, and applauded, and mouthed “thank you” through the window at us.

In Bowling Alone, Robert Putnam’s utterly definitive work on society and community, he writes that the best predictor of a school’s success is the activity of its PTA, and that the activity level of a PTA can be changed dramatically by one or two committed parents. Derek Sivers’ TED talk is all about the powerful change that happens when one person being silly becomes two people being silly, and cognitive scientists have been talking about the power of allies for 50 years.

The horse head(s) didn’t change the world; I’m not that pompous. But changing the world is hard work and it’s worth getting some practice in. So go start something. And, this is crucial, if you see someone else starting something: play along. Maybe the thing you’re playing with draws a great big crowd and changes the world. Maybe it never amounts to more than an office of strangers 50ft away silently jumping up and down and saying thanks.

It’s not such a terrible downside.

UPDATE: The world keeps getting more awesome. Denise from the other office just sent me email saying that the 4th floor office in our building saw their “best day ever” sign, and posted a reply. They don’t even know about the horses. They just wanted to share in the happy.

and tomorrow will be even better (photo credit: denise)

You’re three. You can count, you can sound out words with help, and you can sing. Thanks to your mom you’re a big fan of Mary Poppins and thanks to me you’re a big fan of O Brother Where Art Thou but I think you still sing mostly to Raffi and One Direction. For One Direction you also dance, which involves more hip motion than I’m really comfortable with.

You seemed to understand Halloween and Christmas much better this year, and you definitely understand your birthday. You want 3 star-shaped candles on your cake. You want star candles because your mom put glow-in-the-dark stars on the ceiling of your room. Every night when I put you to bed we turn off the lights and we sit in the rocking chair and we stare at the stars and talk about which one is your favourite. It’s basically the best thing ever.

You’re developing complex thoughts and emotions now at a dizzying pace. Like “awkwardness of forgetting someone’s name.” You try to remember, wrestle with it visibly, and then call them, “that one,” quietly, half-embarrassed. “Patience with daddy clearly being an idiot” is another one that’s coming along nicely.

The one that stopped my heart, though, was a story from day care. You’ve been in day care for 6 months, and I wondered how it would go since it was your first real socialization with other kids outside of occasional visits. It’s a big day care with a different groups for different ages. A month or two ago, a new boy came in to the group next to yours in the facility. He was pretty sad on his first day and, during naptime, he was lying on his mat and crying. Your mat was nearby, on the other side of a half-height dividing wall. When your daycare supervisor walked over, she saw that you were awake, and speaking softly to him through the wall saying,

It’s okay. You don’t need to cry. Your mommy will come back. Mommies and daddies always come back.

You break my heart, kiddo, in all the very best ways. Happy birthday. I love you I love you I love you.

A brief pause.

She: Why? Because some guy on the internet told you to?
He: He… no. I mean. That’s how I found out about it. But that’s not why.
She: Why then? Doesn’t it itch?
He: It itches for a while. That’s part of the whole thing.
She: Part of what whole thing? Why put yourself through that?
He: Just because. Don’t worry about it.
She: No. No way. If have to live with it for a month, I want to know why.
He: No. Anyhow, you’ll think it’s silly.
She: Probably.

A brief pause.

He: I do it to be free.
She: You do it to be free.
He: I do it to be free. I do it because the world finds a hundred ways each day to silence the songs of my heart. I do it because I need a counterpoint to neckties and filing cabinets and oxford commas. I do it because so much of my life is multivitamins and single-file lines and carpet tiles and checklists and parallel parking and sugar free sweetener and appointment reminders and hand sanitizer and whiteboards that the idea of running a razor across my neck every day in an effort to fit in better is simply more than I will accept. I do it to push back that cloud of oppressive, pervasive, repressive cleanliness by half an inch; to give myself space to breathe, and to be.

A brief pause.

She: Multivitamins. Is that what the guy on the internet told you? Multivitamins?
He: We also get to tweet about it and post photos.
She: Oh good.

I’m writing this one from an airport. Tomorrow we’ll probably video chat, because you are a kid for whom the world has just always included video chat.

I’m at the airport so that I can fly to Boston. I’m going to meet up with a bunch of other people from Mozilla who are working to make the web awesome for you. I find it difficult to remember life before the web, and I didn’t even really get it until my teens. It changes quickly. I wonder what it will look like by the time you read this. I hope it’s excellent. I want you to know that a lot of incredible people worked really hard to build it. We think it’s just about the best thing we can do to make the world a better, fairer, friendlier place. I hope you agree.

Since my last note, you’ve become a city girl. You ride streetcars and subways. When I ask you what you want to do on Sunday morning, you answer, “Go to the café. Then go to the Farmer’s market.” You’ve become a hippy. You have skinned knees. You fell and cut your lip last month, and you have two slivers in your hand that you won’t let me remove.

And yet you run. You run and you laugh while you’re running. You pull all the pillows off the couch and then fall blindly back into them. You climb onto the railing of your crib, stand on the back of couches, and generally terrify your mother and me. And when you (inevitably) damage yourself, you cry, and you run to one of us, and you make us kiss it better and then, through sobs, you say, “feel better now.”

Last week I took my 5,000th picture of you. That’s one every four and a half hours since the moment you were born. I don’t think I’ll keep up the rate but thanks for your patience, regardless.

You are just the best of me, little girl. Thank you for everything you bring into my life. Thank you for kisses and hugs and being clingy after your nap. Thank you for helping me make pancakes, and for sharing your chocolate. And thank you for video calls when I’m far from home. We’re boarding soon, Lil. I love you.

Daddy

I’ve included my answer below even though, re-reading it a few days later, there’s so much more I want to add (I can’t believe I didn’t mention working with our worldwide community of employees and volunteers, or the impact of video conferencing, or the miracle of california tacos, or qdb, or mozillamemes…)

What’s your experience?

== SNIP ==

These days I’m administrative overhead at Mozilla, but I started as an engineer.

4 things that are exactly as you’d expect:

1. The people you work with are outstanding. Prior employers of mine claimed to “hire well” and they probably did by industry standards. Mozilla folk are just from another league altogether. Jaw-droppingly amazing – smart, creative, motivated, fearless.
2. You have immense direct scope to change the web. Teams are generally small and have clear focus areas, so your day to day work has an impact immediately. There are also pretty low barriers to cross-pollination. If wandering down a rabbit hole leads you to a bug elsewhere in the code: fix it. Barriers to “hacking in other people’s turf” are low.
3. It’s really, honestly out in the open. Engineers tend to expect this since many of them have interacted with the project as volunteer contributors before coming to work here. People from other disciplines are often surprised to see that the openness isn’t a front. There are some corp-confidential things, usually around partnerships with other companies that we haven’t announced yet, and some things are restricted to our security group (which includes non-employees) but by and large the project runs, and is governed, through public meetings, mailing lists, and bugs.
4. People are really mission-driven. We all remind each other regularly that we’re a non-profit – our job is to make the web better. That mission is invoked regularly as an acid test for the work we do and the decisions we’re making. Far, far moreso than I ever experienced from the mission statement of a traditional for profit shop. We still need to compete hard, we still need to win, so a lot of the time we *sound* like a hungry startup, but it comes from love.

5 things that often surprise people:

1. Everything you do gets attention. Lots of attention. If you flame someone in a bug, you should expect The Register to write an article about it. If you blog about our update process, maybe Forbes. Pixel-level UI changes, user agent strings, support for niche image formats, background colour when viewing images… You are in a fishbowl, and the world is watching.
2. We have a massive amount build and test infrastructure. This shouldn’t be surprising, but engineers are often coming from cultures that pay lip service to automation and testing. We’re swimming in it.
3. Basically all engineering work requires code review. Again, this is as it should be everywhere, but is rarely actually enforced.
4. There is profound heterogeneity of tool stack for development. People develop on any platform, have custom build systems, and use whatever tool is convenient/available/known to them. Coming from IBM, where eclipse was the universal IDE, it surprised me to see how different one person’s set up was from the next.
   
5. There are twin hierarchies for engineering decision making – the corporate management chain and the project’s module ownership system. This is a work in progress and sometimes the lines blur (leaders will sometimes end up with status in both trees) but by and large technical ownership is driven from merit and work in the code, while organizational direction and prioritization is owned by formal “management.” I manage the teams that work on Firefox desktop, Firefox android, toolkit, and developer tools, but I’m not the module owner for any of those. We’ve had VPEs who don’t have commit rights.

We’ve grown a lot, very quickly, so while there’s never just one version of “what it’s like,” now is a particularly change-heavy time. I’m as curious as anyone else about the other answers that show up here. But I continue to be astonished that I’m permitted to work with this incredible sample of humanity. Astonished and humbled.

Daily.

== /SNIP ==

Updates:

• Mike Ratcliffe has posted his own thoughts
• Brian Bondy has answered in the quora thread

When I read (and re-read) the posts, I see 3 main points:

1. The constant interruption of updates is toxic to the usability of any piece of software, especially one as important as your web browser.
2. Our reasons for frequent updates were arbitrary, and based on the wrong priorities.
3. We take our users for granted.

To be honest, if it weren’t for the third point, I wouldn’t be writing this. Anytime you do something that impacts lots of people, especially lots of impassioned and vocal people, you’re gonna get criticism. Listening to that is essential, but fanning the flames can consume all your energy and even still some people won’t be convinced. The third point, though, made by two people who know and love Mozilla even if they haven’t been close to the release process, isn’t something I want to leave sitting. I understand how it can fit a narrative, but it’s just not true.

Since I’m writing anyhow, though, let’s take them in order.

Interruptions Suck

Yes. They do. One criticism that I think we should openly accept is that the move to regular releases was irritating. The first releases on the new schedule had noisy prompts (both ours and the operating systems’). They broke extensions. Motives aside, our early execution was lacking and we heard about it. Plenty.

Today our updates are quiet. Addons have been compatible by default since Firefox 10 back in January. But that was a mountain of work that would have been much nicer to have in hand up front. As Jono says, hindsight is 20/20, but we should have done better with foresight there.

Motivations

It was hard for me to read the misapprehension of motives in these posts. Hard because I think Mozilla’s earned more credit than that, and hard because it means I haven’t done a good job articulating them.

Let me be clear here because I’m one of the guys who actually sits in these conversations: when we get together to talk about a change like this, concepts like “gotta chase the other guys” are nowhere in the conversation. When we get together and draw on whiteboards, and pound on the table, and push each other to be better, it is for one unifying purpose: to do right by our users and the web.

I wrote about this a while back, but it bears repeating. We can’t afford to wait a year between releases ever again; we can’t afford to wait 6 months. Think how much the web changes in a year, how different your experience is. Firefox 4 was 14 months in the making. A Firefox that updates once every 14 months is not moving at the speed of the web; we can’t go back there. Every Firefox release contains security, compatibility, technology and usability improvements; they should not sit on the shelf.

There’s nothing inviolate about a 6 week cycle, but it’s not arbitrary either. It is motivated directly from our earnest belief that it is the best way for us to serve our users, and the web at large.

And so the hardest thing for me to read was the suggestion that…

We Take Our Users For Granted

Nonsense. I don’t know how else to say it. In a very literal way, it just doesn’t make sense for a non-profit organization devoted to user choice and empowerment on the web to take users for granted. The impact of these changes on our users was a topic of daily conversation (and indeed, clearly, remains one).

To watch a Mozilla conversation unfold, in newsgroups or in blogs, in bugzilla or in a pub, is an inspiring thing because of how passionately everyone, on every side of an issue, is speaking in terms of the people of the web and how we can do right by them. We are at our most excellent then.

There’s beauty in the fact that this is another of those conversations. It is not lost on me, nor on Jono and Atul, I’d wager. They are Mozillians. And I believe they care deeply about Firefox users. I hope they realize how much the rest of us do, too.

The combined mobile and graphics teams cut the beta blocker list for Fennec Native in half. The desktop team banged together a working prototype of sign in to the browser. The firefox tech leads worked with product and project management to nail down the kilimanjaro bug list for desktop. Madhava gave a great talk about the future of Firefox UX. I would have scored it as a strong success based on those outcomes alone.

And then this happened:

Hacking

• Scratchpad backending to github gists
• Jetpacks in scratchpad
• Task-specific icons in Windows 7 jumplists
• Finalized interactions and designs for the new downloads manager
• Sync meet-up and discussion to evaluate how to move forward with a smaller core, by moving data handling to components and allowing the Sync team to concentrate on the real Sync functionality
• Session Store meet-up and discussion on how to move forward with “Speedy Session Restore” as well as SS2 (rewriting SS).
• Initial explorations of a Fennec reading mode (post-1.0)
• Implemented Windows 8 appbar
• Roll your own one-off, bespoke debuggers in the scratchpad.
• Responsive Design tool for web developers

Lightning Talks

• Honza made Firebug’s HTTP Monitor a standalone component, and it can now remote-monitor fennec, too
• Gavin gave a great talk about code review
• Tim Abraldes demoed the Web App runtime
• Margaret walked us through Hacking Fennec
• Brian Bondy updated the status of our Windows 8 work
• Felipe found and fixed memory leaks with about:cc
• Dietrich walked through the state of the browser components team
• David Dahl went meta, and talked about giving Firefox talks in our community

I know I’ve missed things. I know some of it is still being written up. In fact, I’m not even the first to write a round up post. Here’s Finkle doing the same, and dcamp, and cwiiis.

FX-Team is big enough these days that it’s quite an undertaking to get us all together in one place. But man, it’s phenomenal when we do.

“I’m going to blindly refuse to listen to contrary opinions. I’ve already made up my mind, and will invent reasons why alternatives won’t work. Most importantly, I’m going to get this done my way, regardless of whether it’s actually the best decision, or even a good idea.”

You’ve never approached a problem that way. No one has.

But you’ve probably told yourself that story about someone else. You’ve been on the receiving end of one of these mindless and petty tyrants, in a bug or a mailing list or a standards body, and you’ve decided that you were seeing a rigid idiot in action. I know I have.

My philosophy of science prof used to talk about how the two important tests of a scientific model are whether it allows you to make accurate predictions, and how well it helps you discover new things. This matters more than its elegance or its intuitive appeal, though a really nice model has those, too.

The Rigid Idiot model does, for better or for worse, predict. It predicts more rigid idiocy, and people using that model to inform their interactions are likely to get precisely that. But it’s a pretty hollow model for generativity; it doesn’t help you make progress.

Here’s an alternate model:

Stress response pre-dates our neocortex, and outranks it. It is wired more deeply into us than language, much less rational discussion. And it has predictable effects. A person under stress (personal, professional, social, physical) will lose patience more quickly, anger easily, resist change, and consider fewer alternatives before making decisions. It’s an ancient, optimized cognitive path: less waffling when there are lions nearby. That it impairs our ability to function in this 10,000 year old thing called ‘civilization’ is evolutionary postscript.

You get to choose which model you bring to a conversation. When you assume that the person you’re dealing with is acting atypically, from point-in-time stress instead of born-in idiocy, you give yourself follow-up questions to ask about timelines, or conflicting pressures, or hidden assumptions. You give yourself ways to understand motivations, and implicit guidance about tone.

Not every asshole is a stress response waiting to be defused, but I swear to you that the single greatest improvement you can make to your success rate with these conversations is to switch models. I have seen people turn on a dime once their stressors are addressed. Suddenly there are lots of solutions, and confrontation turns to collaboration. It’s like a god damned secret decoder ring, to be honest.

With practice, you may even start to recognize the descent into idiocy in your own interactions, though it won’t make you immune. This is old, lizard-brain stuff. Like drunkenness, you can get better at detecting it, but you can’t think your way out of it. And, as with drink, my hope is that if you see someone a little worse for wear, you remember that it’s fleeting. Give them some time to sober up before assuming that’s who they really are.

Let’s go back to first principles for a minute. Releasing every 6 weeks is a cadence we set for ourselves to satisfy several constraints. Constraints like:

• Delivering security and stability fixes on a regular basis.
• Getting new features out to our users promptly, and being able to iterate on the feedback we receive.
• Containing the amount of code change and change-interaction that happens per release.
• Giving ourselves time to react to problems discovered before release (on Nightly, Aurora, or Beta)

Releasing daily wouldn’t work very well; it runs afoul of the last constraint. Releasing yearly would hurt us on the first 3. But the constraints are just about as well satisfied by 40 days or 44 days as they are by 42 days.

We derive great benefit from our current schedule. It satisfies these constraints much better than the old, monolithic release model did. But that is not to say that we should treat 42-day cycles as inviolate. We will adjust. We will add or drop days, or add or drop weeks when we need to. We’ll be respectful of the fact that other people build plans around our plans, and try not to alter schedules without notice, but at the end of the day we’ll do right by our users first and foremost.

Right now we’re not looking at moving the release to another day of the week (Tuesdays do have some nice properties), or adding a skip week somewhere to take us off-cycle for the next patch Tuesday, but those discussions are absolutely on the table when they make sense.

No religion here. 6 weeks is a nice spot in the constraint space, not a law. The first time we miss it, people will talk about us “slipping,” but the tail doesn’t wag the dog here. Firefox still ships when it’s ready.