Category: Commentary

Wherein The Author talks about things and his feelings thereupon.

The Co-pour

logo-4webI’ve started writing again.

I suspect most of you know this already, but for those who don’t: Melissa and I have started writing down our thoughts about management and leadership in tech in a new Medium publication called The Co-pour. We’ve got a few dozen posts up already, covering promotions, firing, and several other areas we’ve seen new leaders struggle.

We have more in the works but here’s the current list, in case you’re curious:

I’m not shutting this place down, it’s still where I’ll write my letters to the girls, and anything else that feels right. But the organizational leadership stuff feels like a chunk of work that can live in its own place. And the collaboration with Melissa makes it important to me that it’s a place we built together.

Anyhow if that’s a thing you’re interested in following, and you haven’t found us over there yet, I hope you’ll take a look. Leading people is hard work, and we want to help where we can.

[PS to the Mozillians out there: Yes, it’s hosted at mfbt.ca. I’ve had to domain for years, for obvious reasons. This felt like an appropriate use of it.]

Stargazing

1 year ago, I started a spreadsheet. For 9 months we’d been rewriting Firefox for Android from the ground up, and on May 16, 2012 we released the first beta version to the Android market. I started a spreadsheet to track our star rating. I was nervous.

9 months of rewrite. In web time— in mobile web time— that’s years. Rewrites are almost always the wrong call. You get to throw away bad code, but you throw away good code, too. Hard won, battle-hardened fixes are expensive to throw away, and the code is never cleanly labelled “baby” and “bathwater.” The decision to rewrite Firefox was one of the loudest weeks I’ve had. Even once we’d decided, my hand hovered over the send key for a long time.

3.5 stars. We didn’t pull the old version from the market during the rewrite. We wanted to keep those users safe with regular security updates; that meant keeping the product online. It meant taking black eyes every day from users who tried us, who loved us for our desktop product, and who were disappointed. And because the Android rating system rounds, our already painful 3.7-star rating rounded down to 3.5. We were dead last.

7:1. If you want to maintain a 4.5 star rating, it takes seven 5-star reviews to counteract each 1-star review. Star ratings are deeply flawed for all kinds of reasons: selection bias, survivorship bias, false dilemmas, unidimensionality, reporting bias, et cetera. I know this. And still I watched them. When you try to reach people through an app store, your star rating is the first assessment of your product they’ll see. And when you try to make something great, reviews are real pieces of feedback from real human beings and they are painful and they are frustrating and they are golden.

1181 reviews in the first week. 588 of them were 5-stars. 105 of them were 1-star. I argued with my screen over 1-star comments about bugs we had already fixed. I swooned at 5-star reviews that said they were reversing earlier 1-star reviews.

6 weeks after beta, we pushed to release. By this point beta had climbed to 3.8, which rounds to 4 stars. Fists were pumped. Release sat at 3.6.

Time passed.

200,000 reviews and 6 releases later, this week, today, Firefox and Beta both show 4.5 (rounded) stars in the market and the team is still going strong. I’m immensely proud of the work they’ve done. It’s made me reflective and maybe a bit wordy. I want to have some profound and pithy lesson to separate then and now. Something that I can package up and hand to you. We certainly learned lessons, profound lessons, but in repeating them they sound trite:

Listen. Care about what people think. Be hungry for feedback. Don’t work forward from the tech you have to the product you can build; work backward from the product your users deserve to the tech you’ll need to get there. Ask for help, and accept it even when it hurts to admit that you need it. Don’t throw things away lightly, but be able to throw them away in those rare cases where it’s necessary. Surround yourself with the most excellent people you can find. That one helps a lot.

The haters will say I over-focus on star ratings. There are certainly lots of other good things to measure, and lots of bad things to say about stars. But the stars abide. They are inescapable. They sit front and center in your primary distribution channel. Those five little stars. And I believe they do say something about what we’ve built.

Building good things is hard and my hat is off to anyone who earnestly tries. If the good thing you build reaches people through an app store, I want you to know that I know how you feel. And I’m rooting for you.

[I posted a version of this post on Medium, because it’s fun to try new things.]

Horseplay

I want to make a point about changing the world, but first we need to talk about the horse head.

For Christmas last year I bought my brother a horse head and, while I was at it, picked one up for the office. It felt like the kind of thing that Mozilla Toronto would enjoy.

//platform.twitter.com/widgets.js
Good times.

Between meetings we needed a place to store it; the sad, flat way it sort of collapsed when left on a desk was unsatisfying. So Madhava and I went out in search of a head on which to mount it. Steps from our office is a mannequin supply store, which helped.

I put the horse head on its styrofoam mount and, for giggles, set it up in the window. Facing outwards. Staring at the office in the next building.

For 2 weeks it sat there and I was pretty happy with it. And then something happened. Something wonderful and magical happened. I wasn’t there for it, but lmandel and overholt were.

The other office responded.

They taped a note to the glass.

“What’s with the horse?”

We responded.

“Who you callin’ a horse?”

They responded.

“Why you, of course!”

And so it went. Back and forth.

Stick with me, I’m getting to the world changing.

So last week we bought them their own horse head. And yesterday mconley delivered it to their office. They invited him in. They brought the box over to the window where we could see. They gathered around their window and we gathered around ours. And when they opened it and realised what it was they actually jumped up and down, and applauded, and mouthed “thank you” through the window at us.

In Bowling Alone, Robert Putnam’s utterly definitive work on society and community, he writes that the best predictor of a school’s success is the activity of its PTA, and that the activity level of a PTA can be changed dramatically by one or two committed parents. Derek Sivers’ TED talk is all about the powerful change that happens when one person being silly becomes two people being silly, and cognitive scientists have been talking about the power of allies for 50 years.

The horse head(s) didn’t change the world; I’m not that pompous. But changing the world is hard work and it’s worth getting some practice in. So go start something. And, this is crucial, if you see someone else starting something: play along. Maybe the thing you’re playing with draws a great big crowd and changes the world. Maybe it never amounts to more than an office of strangers 50ft away silently jumping up and down and saying thanks.

It’s not such a terrible downside.

 

UPDATE: The world keeps getting more awesome. Denise from the other office just sent me email saying that the 4th floor office in our building saw their “best day ever” sign, and posted a reply. They don’t even know about the horses. They just wanted to share in the happy.

and tomorrow will be even better (photo credit: denise)

At Our Most Excellent

Jono recently wrote a blog post about Firefox updates, and Atul wrote a follow up. They are two of the brightest usability thinkers I know. When they talk about users, I listen. I listen, even though some of the things they say sound confused to me, and some are plain wrong. And I listen because if people as bright and in tune with Mozilla as them think these things, I bet others do, too.

When I read (and re-read) the posts, I see 3 main points:

  1. The constant interruption of updates is toxic to the usability of any piece of software, especially one as important as your web browser.
  2. Our reasons for frequent updates were arbitrary, and based on the wrong priorities.
  3. We take our users for granted.

To be honest, if it weren’t for the third point, I wouldn’t be writing this. Anytime you do something that impacts lots of people, especially lots of impassioned and vocal people, you’re gonna get criticism. Listening to that is essential, but fanning the flames can consume all your energy and even still some people won’t be convinced. The third point, though, made by two people who know and love Mozilla even if they haven’t been close to the release process, isn’t something I want to leave sitting. I understand how it can fit a narrative, but it’s just not true.

Since I’m writing anyhow, though, let’s take them in order.

Interruptions Suck

Yes. They do. One criticism that I think we should openly accept is that the move to regular releases was irritating. The first releases on the new schedule had noisy prompts (both ours and the operating systems’). They broke extensions. Motives aside, our early execution was lacking and we heard about it. Plenty.

Today our updates are quiet. Addons have been compatible by default since Firefox 10 back in January. But that was a mountain of work that would have been much nicer to have in hand up front. As Jono says, hindsight is 20/20, but we should have done better with foresight there.

Motivations

It was hard for me to read the misapprehension of motives in these posts. Hard because I think Mozilla’s earned more credit than that, and hard because it means I haven’t done a good job articulating them.

Let me be clear here because I’m one of the guys who actually sits in these conversations: when we get together to talk about a change like this, concepts like “gotta chase the other guys” are nowhere in the conversation. When we get together and draw on whiteboards, and pound on the table, and push each other to be better, it is for one unifying purpose: to do right by our users and the web.

I wrote about this a while back, but it bears repeating. We can’t afford to wait a year between releases ever again; we can’t afford to wait 6 months. Think how much the web changes in a year, how different your experience is. Firefox 4 was 14 months in the making. A Firefox that updates once every 14 months is not moving at the speed of the web; we can’t go back there. Every Firefox release contains security, compatibility, technology and usability improvements; they should not sit on the shelf.

There’s nothing inviolate about a 6 week cycle, but it’s not arbitrary either. It is motivated directly from our earnest belief that it is the best way for us to serve our users, and the web at large.

And so the hardest thing for me to read was the suggestion that…

We Take Our Users For Granted

Nonsense. I don’t know how else to say it. In a very literal way, it just doesn’t make sense for a non-profit organization devoted to user choice and empowerment on the web to take users for granted. The impact of these changes on our users was a topic of daily conversation (and indeed, clearly, remains one).

To watch a Mozilla conversation unfold, in newsgroups or in blogs, in bugzilla or in a pub, is an inspiring thing because of how passionately everyone, on every side of an issue, is speaking in terms of the people of the web and how we can do right by them. We are at our most excellent then.

There’s beauty in the fact that this is another of those conversations. It is not lost on me, nor on Jono and Atul, I’d wager. They are Mozillians. And I believe they care deeply about Firefox users. I hope they realize how much the rest of us do, too.

The Cognitive Science of SHUT UP

“I’m going to be a shrill and rigid idiot.

“I’m going to blindly refuse to listen to contrary opinions. I’ve already made up my mind, and will invent reasons why alternatives won’t work. Most importantly, I’m going to get this done my way, regardless of whether it’s actually the best decision, or even a good idea.”

You’ve never approached a problem that way. No one has.

But you’ve probably told yourself that story about someone else. You’ve been on the receiving end of one of these mindless and petty tyrants, in a bug or a mailing list or a standards body, and you’ve decided that you were seeing a rigid idiot in action. I know I have.

My philosophy of science prof used to talk about how the two important tests of a scientific model are whether it allows you to make accurate predictions, and how well it helps you discover new things. This matters more than its elegance or its intuitive appeal, though a really nice model has those, too.

The Rigid Idiot model does, for better or for worse, predict. It predicts more rigid idiocy, and people using that model to inform their interactions are likely to get precisely that. But it’s a pretty hollow model for generativity; it doesn’t help you make progress.

Here’s an alternate model:

Stress response pre-dates our neocortex, and outranks it. It is wired more deeply into us than language, much less rational discussion. And it has predictable effects. A person under stress (personal, professional, social, physical) will lose patience more quickly, anger easily, resist change, and consider fewer alternatives before making decisions. It’s an ancient, optimized cognitive path: less waffling when there are lions nearby. That it impairs our ability to function in this 10,000 year old thing called ‘civilization’ is evolutionary postscript.

You get to choose which model you bring to a conversation. When you assume that the person you’re dealing with is acting atypically, from point-in-time stress instead of born-in idiocy, you give yourself follow-up questions to ask about timelines, or conflicting pressures, or hidden assumptions. You give yourself ways to understand motivations, and implicit guidance about tone.

Not every asshole is a stress response waiting to be defused, but I swear to you that the single greatest improvement you can make to your success rate with these conversations is to switch models. I have seen people turn on a dime once their stressors are addressed. Suddenly there are lots of solutions, and confrontation turns to collaboration. It’s like a god damned secret decoder ring, to be honest.

With practice, you may even start to recognize the descent into idiocy in your own interactions, though it won’t make you immune. This is old, lizard-brain stuff. Like drunkenness, you can get better at detecting it, but you can’t think your way out of it. And, as with drink, my hope is that if you see someone a little worse for wear, you remember that it’s fleeting. Give them some time to sober up before assuming that’s who they really are.

Rapidity

[This is a re-post of a post that originally appeared on the Future of Firefox blog]

Last week we released a new version of Firefox. We shipped on time, 6 weeks after the last update, making it our first true rapid release milestone. There was cake. Now that we know that we’re capable of this velocity, I’d like to revisit the reasons why it’s important, and the lessons we’ve already learned.

Mission drives Mozilla. People sometimes forget that we’re a non-profit, that our only job is to make the Web a better place. Rapid release advances our mission in important ways. We get features and improvements to users faster. We get new APIs and standards out to web developers faster. We are delivering on the promise of the web at web speed.

Small, frequent releases improve quality, too. Engineers in the Mozilla community regularly say things now like “I don’t like not understanding this piece, let’s back it out and I’ll catch the next train.” We move deliberately. We don’t rush. And, even though it sounds like a contradiction, when we take our time we go faster.

There’s a great deal for us to be proud of, but we also need to be humble. This change was hard for us to make, and it’s been hard for some of our supporters, too. We have been glib or dismissive in the way we’ve communicated about parts of it. We live rapid release daily, and that makes it easier for us to see past the problems. We are also tenacious about the necessity of our new schedule, and tenacity can be mistaken for obstinacy.

We, everyone in the Mozilla community, all of us, need to communicate with clarity and sensitivity. We need to help the people who support our mission to understand why these changes are essential. We need to keep listening, and adjusting as we learn. We need to, and we will.

The push to ship faster isn’t some kind of software machismo. We push ourselves to ship faster because the web is under threat. Amazing and innovative people are doing amazing and innovative things and right now they have a choice: build for the web, or build for the walled gardens. The web can win that fight.

The open web is the most amazing, universal communication and distribution platform ever built. To win, the web needs to be agile and responsive. To help it, we need to be agile and responsive, too. That’s why rapid release matters.

Every Six Weeks

It’s astounding to me, but we’ve been living rapid release for a few months now. We’re moving faster. A new feature implemented today and landed on mozilla-central can be delivered to our users in 12 to 18 weeks, not months or years. Incredibly, the same process that gives us that agility is giving us greater robustness, too. Testing and stabilization of each release across progressively larger audiences helps us find and fix bugs early, and build confidence in the quality of each release.

I want to clarify an important part of the process, though, that I think many people haven’t yet understood. Remember, an individual release train is 6 weeks of development time followed by 12 weeks of stabilization:

New work doesn’t land on Aurora and Beta. Instead, those channels focus exclusively on working with our heroic and growing community of testers to spot any unexpected issues introduced during development, and then resolve them. Looking at this diagram, you might well conclude that we’d have a release ready every 18 weeks.

Aurora and Beta are so single-minded in their focus on stabilization and testing, though, that many engineers can move on to new work. If we take a step back and look at the broader picture, this is what actually happens:

During the 12 weeks that a release spends on Aurora and Beta, the Mozilla community is not sitting idle. They are already working on features and fixes for the next release, and the release after that. Every 6 weeks their work is picked up into the next Aurora, the next Beta, and the next release. When you look at this broader picture, you notice an important point:

There can be a new release of Firefox every 6 weeks, not every 12 or 18.

I’ll say it again, because it’s important: most of the time, we’ll release a new Firefox every 6 weeks.

Many people are surprised by this fact, though it’s been part of the process all along. When Firefox 4 came out, we committed to ship the next release of Firefox within 3 months. We did it, and when we did I think many people concluded that we have moved to a 3 month cycle. In truth, though, the only reason it took us 3 months was that our Aurora and Beta channels started off empty; they had to wait for the new release to make it through the process. The next Firefox is already in Beta, and is scheduled to come out 6 weeks after the last one. When that happens, yet another Firefox will enter Beta, and so on.

We’re studying the effects of the process carefully; it’s a big change and we will be flexible in our approach as new information comes in. We may decide that 6 weeks is the wrong interval, for instance, though it’s worth remembering that Firefox maintenance releases have been released on 6-8 week intervals for years, and sometimes included major changes. We’re also paying close attention to the impacts this cycle has on our ecosystem of add-ons, plugins, and other 3rd party software that interacts with Firefox. We’re working with large organizations, too, to understand how rapid release can fit into their software deployment systems.

Whatever adjustments we make, it’s clear that rapid release is a major improvement in our ability to respond to the needs of our users and the web. Every 6 weeks we have a new Firefox to evaluate and, unless some surprising and irreconcilable breakage is discovered, release to the world. No one will have to wait a year for the developer scratchpad now in Beta, or the massive memory and performance improvements already on Aurora, or the slick tab management animations soon to land on Nightly. Rapid release is already paying dividends, and we’re just getting started.

[This post originally appeared on the Channels blog]

First Impressions from China

Great Wall in FogChina is different.

When I got back from my recent trip to visit Mozilla Online in Beijing, I heard myself saying that often, but it’s very nearly a content-free statement. Of course China is different. A better, albeit clumsier, way to express things is:

The Chinese web is not the web we are used to.

“We” Mozilla, “We” the Western tech world, “We” the builders of the web. China is going about things differently, and they’re bringing more than a billion people online with them. The folks at Mozilla online understand this and were exceedingly patient and generous with their time helping me begin to do so as well.

Here’s one way of thinking about that difference: Continue reading “First Impressions from China” →

The SSL Observatory

Oh ho, lookit what the EFF went and did!

The EFF SSL Observatory is a project to investigate the certificates used to secure all of the sites encrypted with HTTPS on the Web. We have downloaded a dataset of all of the publicly-visible SSL certificates, and will be making that data available to the research community in the near future.

This is exciting. I knocked together a less ambitious version of this last year, but the EFF guys are doing it like grown-ups, and are getting some interesting data.

Numbers-wise, they’re in the right ballpark, as far as I can tell. Their numbers (1-2m CA-signed certs) coarsely match ones I’ve seen from private sources. I’ve heard from a few CAs that public-crawl estimates tend to err 50-80% low since they miss intranet dark matter, but at least the EFF is tracking other public-crawls. Given that their collection tools and data are going to be made public, that’s a really big deal. Previously, I haven’t been able to get this kind of data without paying for it or collecting it myself. If the database is actively maintained and updated, this will be a great resource for research.

Their analysis of CA certificate usage is also interesting. I’d like to see more work done, here, and in particular I’d like to see how CA usage breaks down between the Mozilla root store and others. We spend considerable effort managing our root store, and recently removed a whole pile of CA certificates that were idle. In some places, the paper seems to make the claim that fully half of trusted CAs are never used, but in other places, the number of active roots they count outnumbers our entire root program. I understand why they blurred the line for the initial analysis, but it would be swell to see it broken out.

As they mention, there are legit reasons for root certs to be idle, particularly for future-proofing. We have several elliptic curve roots, and some large-modulus RSA roots, which are waiting for technology to catch up before they become active issuers while giving CAs a panic switch in the case of an Interesting Mathematical Result — that feels okay to me. On the other hand, if there are certs which are just redundant, it would be great to know, so that we can have that conversation with the relevant CAs, and understand the need to keep the cert active.

This is exactly what I hoped would come of my crawler last year, but they’ve done a much more thorough job. We’ve seen an uptick in research interest in SSL over the last few years. Having a high quality data source to poke when testing a hunch is going to make it easier to spot trends, positive or otherwise. Interesting work, folks; keep it going!

Kathleen, a FAQ

Q: Kathleen who?

Kathleen Wilson works for the Mozilla Corporation, and manages our queue of incoming certificate authority requests. She coordinates the information we need from the CAs, shepherds them through our public review process and, if approved, files the bugs to get them into the product.

Q: Holy crap! One person does all of that? Is she superhuman?

It has been proven by science. She is 14% unobtainium by volume.

Q: That’s really awesome, but I am a terrible, cynical person and require ever-greater feats of amazing to maintain any kind of excitement.

She came in to a root program with a long backlog and sparse contact information, and has reduced the backlog, completely updated our contact information, and is now collecting updated audit information for every CA, to be renewed yearly.

Q: Hot damn! She’s like some kind of awesome meta-factory that just produces new factories which each, in turn, produce awesome!

I know, right? She has also now removed several CAs that have grown inactive, or for which up to date audits cannot be found. They’ll be gone as of Firefox 3.6.7. They’re already gone on trunk.

Q: Wait, what?

Yeah – you can check out the bug if you like. I’m not positive, but I think this might represent one of the first times that multiple trust anchors have ever been removed from a shipping browser. It’s almost certainly the largest such removal.

Q: I don’t know what to say. Kathleen completes Mozilla. It is inconceivable to me that there could be anything more!

Inconceivable, yes. And yet:

  1. She’s also made what I believe to be the first comprehensive listing of our root, with signature algorithms, moduli, expiry dates, &c.
  2. In her spare time, she’s coordinating with the CAs in our root program around the retirement of the MD5 hash algorithm, which should be a good practice run for the retirement of 1024-bit RSA (and eventually, in the moderately distant but forseeable future, SHA-1).
  3. She has invented a device that turns teenage angst into arable land suitable for agriculture.

Fully 2 of the above statements are true!

Q: All I can do is whimper.

Not true! You can also help! Kathleen ensures that every CA in our program undergoes a public review period where others can pick apart their policy statements or issuing practices and ensure that we are making the best decisions in terms of who to trust, and she’d love you to be a part of that.

Q: I’ll do it! Thanks!

No, thank you. That wasn’t a question.