07
Sep 07

Gaming

Dice by OlivanderWhen I was coding at IBM, we had pretty clear quality metrics that had to be met before a product went out the door.  We had to execute all of our tests, and pass 95%, for instance.  No, not 100%, because good developers ought to write tests even if they know the current code won’t be able to pass them – that’s far better than not writing the test, and someone at IBM got that.  We also couldn’t ship with any P1 defects, and all P2 defects had to have a “disposition” – a workaround, or at least clear documentation on alternatives.  We were, after all, IBM.

I remember one product cycle where things were particularly tight.  Maybe they’re all “particularly tight.”  In this case anyhow, some teams had fallen far behind, to the point that our team was being brought in to do triage and QA on their code as well.  It was a stressful time for the product managers, for the whole department.

We were also not meeting our quality goals.  There were significant P1s that still didn’t have fixes, and our pass rate on tests was mid-80s.  We were asked to “focus.”

Whether it was encouraging “focus” per se, or just competent, dedicated people trying to do their job, we made some headway.  Tests-passed got into the high-80s, not many P1s got fixed but a couple more P2s had workarounds written.  Not enough, but better.  Still, we were about to run out of time.  That’s when we got an email.

“We test our code to make sure that the intended functionality succeeds,” it started (or words to that effect.)  “Obviously, it wouldn’t make sense to test functionality we never expected to have.  If we were releasing a word processor, and wanted to get inline spellcheck in, but just couldn’t do it, well then it would hardly be sensible to wring our hands about failing the inline spellcheck tests, would it?”

Oh…kaaaaay… we thought, all of us together.

“So if there are tests failing that we know we can’t fix in time, then that’s functionality we don’t intend to ship.  So it doesn’t make sense to include those in our tests.”

With those tests removed, of course, our pass rate went way up.  Ahem.

There was still the matter of the wayward P1s and P2s, but every developer in the room knows how those were fixed.  One morning we all came in to a bunch of bugmail saying that our P2s were now, coincidentally and en masse, P3s; our P1s were all either P2s or P3s depending on how plausibly a workaround could be written.

And the product shipped.  And customers complained.  And tech sales wept.  And a year after shipping we had no active, deployed, reference customers.  And we did that thing, where we taught our customers not to trust our X.0 software, to wait for at least two service packs before trusting us.  I hate doing that thing.

This isn’t about me throwing stones at IBM, it’s about underscoring how hard metrics are to get right, and how prone people are to gaming them when their incentives are misaligned.  I bet the product managers got congratulated for shipping Another On-Time Release. I’m sure, too, that the blame for the market failures was spread broadly enough to be much less impactful, so it’s hardly surprising that PMs would act this way.  I know that’s not novel insight, but I’ve always held on to that story as one of my own favourite examples.

The Mozilla community has amazed and impressed me with its active awareness of, and resistance to, these kinds of games, but it’s a never-ending battle.  We, too, will second-guess our decision to mark some feature as P1 when we get down to it, or our decision to mark some bug as blocking.  But I feel like there’s a cultural difference in game-awareness that’s important; those decisions generally seem to have “Are we gaming things here?” as part of the discussion.  Can anyone tell me how we get there?  IBM is not full of idiots nor of self-serving cycnics.  If someone can tell me how to bottle that awareness, and cultivate it in software companies, and make it stick, I’ll write the book and give you a cut.


28
Aug 07

Airport Security 2 for 1!

nedrichards' playmobil photoTwo interesting (if longish) articles lately on airport/airplane security:

1. A pilot on airline security

2. An interview between Bruce Schneier (Security Dude) and Kip Hawley (Head TSA Dude)

Both are, I think, interesting reading; and both avoid the Designated Stupid Zones (“Airport security is useless” and “Whatever it takes to Fight Terror”) at the polar ends of the debate.

Neither of these articles is directly related to Mozilla, but enough of my co-workers travel regularly that I’m gonna tag it that way anyhow, so that it shows up on planet – where our blogs all hang out and play together while we’re at work.

[Special thanks to nedrichards for the photo – I’m keeping this one around.]


20
Aug 07

SSL Infoporn

mac_steve infoporn600,000.  According to Netcraft, there are about 600,000 SSL sites out there on the public internet, and we just recently tipped over that arbitrary, but pleasantly round, number.

I’m not sure why, but when I tell people this (people, that is, who have any hope of being interested in such things; a small, biased, statistically indefensible sample,) they are surprised.  I think mostly they expect the number to be higher.  And in actual fact, it probably is, at least a little bit.  I am reasonably certain, without even looking into them, that Netcraft’s methods are more prone to type-2 errors – false negatives – than they are to false positives.  Nevertheless, it’s probably the right order of magnitude.  There are almost certainly less than a million, for instance.

Netcraft doesn’t publish any numbers it may gather about the ratio, in that group, between DV, OV, and EV certs, but the informal vibe I get leads me to believe that there are around 2000 EV certs out there at the moment.  Given that several of these have gone to extremely high traffic domains, though, that number probably under-represents their network significance.

I bring these numbers up here because they seem to surprise people, and surprises are generally more instructive than confirmations.  In the last couple weeks, a fair number of surprising numbers have flitted across my radar, so I figured I would rehash a couple here, with no particular (conscious) effort to weave a narrative into them beyond, “hey look, infoporn!” Continue reading →


09
Aug 07

There goes that analogy?

So Medeco Locks, often cited as the unpickable-in-practice lock, can be picked.  Not just picked, bump keyed.  I guess that’s sad if you’re Medeco, though I suspect that in their heart of hearts, they know as well as I do that lockpicking thieves are rarely the high-probability threat.

I don’t know if there are vendors out there calling their solution the “Medeco of internet security” but I suppose they’ll want to stop, if so.  The nice thing, though, is that the whole fracas is a delicious example of General Security Maxim #6:

If your product is unbreakable, you are wrong.  Also, here comes the breaking.

If you suffer from this tendency to overstate security claims, I’ve created a motivational poster to help you remember.

(Thank you johpan for the ostrich, and flickr toys for the insta-motivate.)


25
Jul 07

Beyond the Padlock: OSCON Talk Slides

PadlockI’m about to go on at OSCON. My talk is titled “Beyond the Padlock: Security UI for the Distracted.”  Meanwhile, behind me in the speakers’ lounge, people are teaching one another to juggle.  So all in all, so far so good.

For those who couldn’t be here, or for those who could, and want another chance to critique my slides, or for those who just like babies with tinfoil on their heads, I’ve uploaded a copy in PDF format.

Wish me luck!  And if you were one of the extremely helpful people who provided reviews and suggestions, thankyouthankyou.  I attribute 95% of any success I may enjoy to your help.


14
Jun 07

Blatant Self-Promotion

PeacockThe Society of Technical Communication has published my latest article in the June edition of Intercom. I wrote it back at IBM, with my coworker Rick Goldberg, and it’s a pretty short piece, but because of the timing of submission and my job change, it’s the first article in print that identifies me as a Mozilla employee. Which is sort of cool.

As a happy coincidence, it happens to be one of the articles they chose for free online distribution, so you can get a full copy of the text in PDF format, if you’re interested.

Kicking and Screaming: Modernizing Today’s Help Systems

Please note, we had no role in choosing the photo to accompany the article. What’s the deal there? Two small CRTs, and a television? With an optical wheel mouse? Aroo?

Also, while trumpeting, I wanted to mention to anyone visiting OSCON 2007 that I (or a person with a similar, but misspelled version of my name) will be giving a talk on Wednesday the 25th about Security UI in general, and Firefox 3 security UI in particular. It would be really keen if I had an audience! Astute readers will note that phrases like “rogues’ gallery” are outside of my normal lexicon. The description was written by Gerv who, in addition to being British and using phrases like “spend the readies” as though they have semantic content, was going to give the talk before I showed up, but graciously bowed out so that I could sink or swim on my own two feet, as it were.

[Photo Courtesy of Billy Brown]


04
Jun 07

Will Firefox have a Green Bar?

Green Bar (Ha!) - From flickrThe number one question I get asked, in my capacity as Human Shield at Mozilla, is how we make any money.  People ask it with a sort of knowing grin, as though they already know we get it from leprechauns, but they want to hear me admit it.  That’s not what this blog post is about.

The second most frequent question I get asked, and the one I’m more directly positioned to answer, is whether Firefox 3 will have an IE7-style Green Bar.  I’m going to try to answer that here by offering my opinion on the matter, and an update on my coding progress to that end.

The short answer to that question is: no.

Continue reading →


21
Mar 07

Revisiting Security UI – Part 2

So we need to get better. We need to start fixing our messages to users so that we are more accurately communicating security information, while being mindful to not bury them in technicalities they neither want nor need. We need cues that are persistent (not relying on people to notice their absence), that are difficult to spoof, and that don’t mix metaphors.

We also, difficult as it is, need to get out of the “safety” game. We can’t tell users “this site is safe” because we don’t know that. Even ignoring the liabilities that might come with such a claim, there isn’t a good technological way to tell, right now, whether a particular site is safe in the way users care about. Do they handle credit card information properly? Do they ignore angry customers? Are they a front for stolen goods? These kinds of naughty people could get SSL certificates (and accompanying padlocks) and even the extended validation practices being discussed wouldn’t really stop them.

What we can do is equip people to make the safety decision for themselves, just as they often have to in the physical world, because we do have some information. It’s like putting ingredients labels on food. What we can do is change the conversation to be about identity instead of safety. This is important, so pay attention:

We need to change the conversation to be one about identity, not safety.

Identity is something we can verify. The padlock conflated identity with other things like encryption status and security, and while that conflation is almost natural to PKI-veterans, it has proven misleading for users.

So what might identity look like?

Continue reading →