Speaking to Lords – FAQ

People seem quite interested in how the trip went. Since I’m too sleepy to have anything qualifying as a coherent, synthesized opinion, FAQ format seems like the strongest play.

How Did It Go?

I think it went quite well. Of course, it’s hard to nail down short term success criteria for conversations with parliamentarians. A meeting like that is not going to end with a legislator standing up and saying “I agree. Let’s go pass a law.” Things like this are an exercise in advocacy: “Here is my opinion of the situation and the options under discussion for its remedy,” followed by others giving their versions of the same thing.

I do feel, though, that my opinion was listened to, understood, and amplified by others. The room included, in addition to invited experts and press, at least half a dozen Lords, and 3 or 4 MPs, so I am also confident that I was heard by people in a position to act on what they hear.

What Did You Say?

A couple of things. I said that this kind of data collection is not something users can be expected to understand and, if they did understand it, not something they have much ability to avoid.

I said that in many markets, even developed ones like Canada and Britain, there isn’t enough choice in ISPs to make “voting with your wallet” a realistic option for people who find this kind of surveillance invasive.

I said that the technological mechanisms for preventing this are prohibitively expensive (in the case of things like “universal SSL deployment”), largely ineffective (since traffic analysis would still be possible), and brittle (opt-out cookies assume you never switch computers or browsers, that you never reinstall or move houses, that you won’t be worn down to the point of surrender by the Nth attempt to opt out).

I said that, historically, anonymized data isn’t. The AOL data was blown wide open, for instance, and that was just search terms, not browsing history. I said that however ironclad Phorm’s current processes may be, this kind of data collection being done by multiple companies over any interesting period of time will almost certainly result in anonymity failures.

I said that the collection of this information is insidious, that however noble and scoped the initial goals, it tends towards exploitation because it is too valuable not to.

After saying a chunk of that in a single burst, I got some applause from some of the people in attendance which felt odd, but certainly seemed to suggest that I had struck a chord.

What are the Lords Like?

Parliamentarians, really, since there were MPs there, but in any event I was impressed, particularly by Baroness Miller, who organized the event.  She was exceptionally good at running a room – at ensuring that legislators’ questions were answered, at bringing digressions back around to the central themes, and ensuring that multiple voices were heard. As a group, they were forthright but unapologetic about their lack of technical knowledge (that’s not their job), and asked clear questions aimed at understanding the legislative implications of various details.

Were there Swords? Powdered Wigs? Snuff Boxes?

People from the UK know that their legislators are basically like other legislators, albeit with more exciting titles. To the rest of us though, the whole thing sounds very romantic, and we entertain positively ridiculous notions like this. No swords, no wigs, no “Yes, your exalted worshipfulness.”  The houses of parliament are guarded by perfectly normal police officers with perfectly normal frowns and perfectly normal assault rifles, but very little pomp.

What about the Building?

Imagine that your great great great grandparents and their friends had all the money in the country, and decided to build a place to hang out. Imagine that since then, it’s where everyone decided to put their cool stuff.  Imagine walking through rooms, separated by wooden doors older than calculus. Imagine those rooms are alternately filled with statues, murals, statues in front of murals, framed masterworks, and leather bound books about anything that could matter. Imagine that there are entirely different paths, staircases and elevators for peers of the realm than for everyone else.  Imagine that you could fit your current house inside the Queen’s entrance and have room to fly a kite from the roof.

It is a nice building.

Would you do it again?

Yes.  Yes I would.

I still think that legislating technology is fraught with peril. The way to mitigate that peril is not to run away from it, though, but to be a voice for the kind of change we want, and against the kind of change we don’t.

Is the Bowmore 17 you brought back tasty?

Yes.

Deep Packet Inspection Considered Harmful?

I was recently asked, in the context of the ongoing Phorm debacle, and with other interested parties, to meet with members of the UK government and discuss deep packet inspection technologies, and their impact on the web.  I’m still organizing my thoughts on the subject – I’ve put some here, but I’d love to know where else you think I should look to ensure I have considered the relevant angles.

Brief Background

Phorm‘s technology hooks in at the ISP level, watches and logs user traffic, and uses it to assemble a comprehensive profile for targeting advertising. While an opt-out mechanism was provided, many users have complained that there was no notice, or that it was insufficiently clear what was going on. NebuAd, another company with a similar product, has apparently used its position at the ISP level to not only observe, but also to inject content into the pages before they reached the user.  It’s hard to get unbiased information here, but this is what I understand of the situation.

Thoughts

1.  Deep packet inspection, in the general case, is a neutral technology. Some technologies are malicious by design (virus code, for instance), but I think DPI has as many positive uses as negative. DPI can let an ISP make better quality of service decisions, and can be done with the full knowledge and support of its users. I don’t think DPI, as a technology, should be treated as insidious.

2. Using deep packet inspection to assemble comprehensive browsing profiles of users without explicit opt-in is substantially more questionable. My browsing history and habits are things I consider private in aggregate, even though any single visit is obviously visible to the site I’m browsing.

It’s possible that I will choose to allow this surveillance in exchange for other things I value, but it must be a deliberate exchange. I would want to have that choice in an explicit way, and not to be opted in by default, even for aggregate data. Moreover, given the complexity of this technology, I would want a great deal of care to go into the quality of the explanation.  Explaining this well to non-technical users might be so difficult as to be impossible, which is why it’s so important that it be opt-in.

3. Using deep packet inspection in conjunction with software that modifies the resultant pages to include, for instance, extra advertising content, is profoundly offensive and undermines the web. The content provider and the user have a reasonable expectation that no one else is modifying the content, and a typical user should not be expected to understand the mechanics of the web sufficiently to be able to anticipate such modifications.

Solutions

As a browser, we do some things to help our users here, but we can’t solve the problem. https resists this kind of surveillance and tampering well, but requires sites to provide 100% of their content over SSL. Technologies like signed http content would prevent tampering, if not surveillance, but once again assume that sites (and browsers!) will support the technology. Ad blockers can turn off any injected ads, tools like NoScript can de-fang any injected javascript but, fundamentally, http content is not tamper-proof, and no plaintext protocol is eavesdropping proof.

People trust their ISPs with a huge amount of very personal data. It’s fine to say that customers should vote with their feet if their ISP is breaking that trust, but in many areas, the list of available ISPs is small, and so the need for prudence on the part of ISPs is large.

That’s what I’m thinking so far, what am I missing?

How Markus Made the World Better Today

Markus Stange did a pretty awesome thing for those of us who work with the Mozilla trees and tinderboxen:

First of all, bookmark that action.

As for how it works, a couple minutes of playing around should explain it.  Basically it’s a running changelog with associated builds and status, plus an at-a-glance view of the tree in the bottom right, and details of a selected run in the bottom left.

If it were covered in shaved black truffle and a velvety 25 year old balsamic, it would be only marginally more delicious.

If you see mstange on IRC, give him your love.  If you have suggestions, give him your patches.

[Update: Why yes, it does require querySelectorAll support.  I trust you have a browser that does that, right?]

SSL Information Wants to be Free

Recent events have really thrown light onto something I’ve been feeling for a while now: we need better public information about the state of the secure internet.  We need to be able to answer questions like:

  • What proportion of CA-signed certs are using MD5 signatures?
  • What key lengths are being used, with which algorithms?
  • Who is issuing which kinds of certificates?

So I decided to go get some of that information, so that I could give it to all of you wonderful people.

Continue reading “SSL Information Wants to be Free”

Firefox Malware?

A crappy thing happened last week – someone wrote some malware that infects Firefox. We obviously don’t like that very much at all, but I wanted to at least make it clear what is and isn’t happening, since there’s some confusion out there.

What is going on?

Basically for as long as there has been software, there have been nasty people out there who get you to download and install software which turns out to have hidden cargo.  Security folks use names like “virus,” “trojan,” “worm,” and “malware” to describe different types, but the point is that if a person can be tricked into running nasty programs, they can do nasty things.

In this case, rather than wiping your hard drive or turning all your icons upside down, this particular jerk has decided to mess with your Firefox. Once you run the program, it hooks into your Firefox and watches for you to visit certain sites, at which point it will steal your username and password.

How Can I Tell If I Have It?

You can open up your Firefox addons manager (Tools->Add-ons) and go to the “Plugins” section.  If you have a plugin called “Basic Example Plugin for Mozilla” you should disable it.


Original credit to TrustDefender Labs’ blog post on the subject

Does This Mean that Firefox is Insecure?

No, and here’s why:

  • This particular malware targets our program, but once you have malicious software running on your system, it can just as easily attack other programs, or harm your computer in other ways.
  • This isn’t contracted by just browsing around the web with Firefox 3. In fact, the Malware Protection features in Firefox 3 are designed specifically to prevent sites from being able to attack your computer.

The people getting infected here are either downloading enticing files that have the malware hiding inside (which is why Firefox 3 hands off all downloads to your computer’s virus scanner once downloaded) or, as some sites are reporting, people who have already been infected in the past having their computers forced to download this file as well.

Typical Firefox 3 users who avoid downloading software they don’t trust are unlikely to ever see this, and even the sites reporting it describe its incidence as “rare”.

What’s this I hear about GreaseMonkey?

There are some mentions of greasemonkey in a couple of the early reports based on some analysis of the code used by this malware, but I want to be clear that the (legitimate, and awesome) Greasemonkey Addon is not involved in this malware in any way. It is not involved in the installation or execution of the attack.

As always, the best defense is vigilance.  Use a browser with a solid security record and modern anti-malware defenses built in, and be very careful about downloading and running programs you find online.  If a bad guy is able to get you to run a program on your machine they will be able to do bad things, so we’ll keep trying to stop them and you keep trying to as well.

More details are also available on the official Mozilla security blog.

Performance Dashboard (v2)

Way back when, (almost exactly a year ago, actually) I built a dashboard for getting at-a-glance views of our performance metrics, to make it easier to spot regressions and assess the state of the tree.

And so, of course, days later we decommissioned those boxes and that whole way of reporting performance, and the dashboard fell into disrepair.

A couple days ago I rebuilt it.

It’s in its infancy right now.  It only pulls data for the 1.9.1/Firefox3.1 branch, and it only pulls a couple tests thus far, but those are easy to add.  It has no fun widgets or user-preference memory or any of that, but patches are accepted.

The code is in a public hg repo here in case you want to beat me to any particular feature.  To run your own copy, just clone the repo, run the scrapedata.py script to get some up to date stats, and then open index.html in a suitably awesome browser.

The graphs are built with google’s really excellent charting API.  It’s reasonably flexible, and great for quick stuff BUT I’m not looking to replace our existing graph server.  That thing has all kinds of charting goodness that I absolutely don’t aim to reinvent.

This is a quick, dumb dashboard; not an immersive data navigation environment.  It isn’t complicated, it’s just something I thought would be useful.  How would you make it better?

[UPDATE: It’s not just coincidence that Rob has been thinking about these issues too, but it is kind of funny to me that we posted within hours of each other.  Clearly it was an idea whose time had come. ]

New in Firefox 3.1: Linkified View Source

Look what Curtis just did:

Linky!

Curtis Bartley is the newest member of the Firefox front end team and, to get his feet wet, he made the world a better place by fixing a very old bug. And its 7 duplicate bugs.

Specifically, he set it up so that resources which are referenced in source are now clickable links.  Want to know what that external javascript does?  Click the link, and it will be loaded in the source viewer.  Likewise CSS.  Maybe you clicked “View Source” only to discover you were looking at a frame set, and actually wanted the source for a frame – that works too.

And yes, back and forward keyboard shortcuts work. And yes, both relative and absolute links work. And yes, you can have this in a tab instead of a separate window, either by sticking view-source: on to the front of your URLs (see?), or by finding one of the addons that does it for you.

Way to go Curtis, keep ’em coming!

SSL Infoquickie (with Bonus Firefox Pro-Tip!)

There is less public information out there about SSL certificate usage than one might like to see. Netcraft has a for-pay report with some interesting figures, and occasionally makes some of that data public, and I’ve blogged about other sources in the past, but in general, it’s pretty sparse. I keep meaning to do something coordinated about that, I have some ideas, but they keep getting back-burnered.

So it came to pass that when someone idly remarked that it would be nice to know what percentage of certs on the top sites were valid, I pounced upon it as a way to quickly release some pent-up info-gathering angst.

It’s profoundly unscientific, but so was the question. Are the Alexa top 500 sites even a good reflection of the most popular SSL sites? Not really. I think it will bias the data towards higher counts of untrusted certs (since the admins aren’t expecting them to be used) and towards lower overall cert counts (since many of those sites won’t answer SSL hails, whereas presumably a list of the top 500 SSL sites all would). Is blindly connecting to their main page on port 443 the best way to harvest their certs? Probably not, lots of them use secure.frobber.tld constructions, so that will also bias the data lower. Let’s just agree that it’s a sort of fun number to have as an order-of-magnitude style signpost.

Of the 500 top sites on Alexa, October 15, 2008:

  • 217 responded to an SSL query on port 443
  • 199 of those replies used valid certs chaining to trusted roots
  • The other 18 were a mix of self-signed, bad chains (likely from trusted roots, though I didn’t investigate), and expired certs.

If you prefer pretty pictures:

SSL Certificate Stats

Any conclusions you want to draw from this data will be only as good as the aforementioned biases within it, but don’t say I never do anything for you in a feeble attempt to vent my own info-lust urges.

Bonus Firefox Pro-Tip: If you are on Firefox 3.1 Nightlies or the upcoming Firefox 3.1 Beta 2, you now have the ability to turn off link-visited colouring.  David Baron recently landed a fix for bug 147777 that adds a new about:config preference to control the behaviour, layout.css.visited_links_enabled.

“Great!” I hear you all saying, “We’ve been hoping for a way to turn off an occasionally useful feature!”

And who hasn’t, really? But the thing of it is that colouring links can give away information to tricky sites about where you’ve been. It’s up to you whether you think that privacy/functionality trade-off is worth making, and the bug is still open while more universal solutions are contemplated, but in the meantime, you have the choice.

Why Did Talos Eat My Face?

Shawn has a great post up about some detective work we’ve been doing this week.  He gives me a bunch of credit, but I am convinced that I did the easy parts – he’s the one making our SQLite happier, so send him the laurels.  Nevertheless, Dave asked if I could write down the basic process used to narrow down the regression, which I agreed to do, because I am a sucker.

Back Story

So Shawn has been trying to update our version of SQLite for some time now.  Updating it fixes half a dozen crasher bugs, and should net us some modest perf wins in some places too.  But three times he’s tried to land it, and three times he’s had to back it out because of a sizeable performance regression on linux.  Specifically, Ts was going from about 1550ms to about 2500ms.  I wouldn’t call that a 65% regression or anything, but I would be pretty comfortable calling it a 61.2% regression so, you know, holy jumping tree squirrels.

The thing is, there was no meaningful hit on Windows or Mac, and even on Linux, Shawn didn’t have much of an idea where to start.  I (as sheriff for the day) didn’t want that sitting in our tree, but understood Shawn not wanting to back it out without any notion of whether the regression was even real, let alone what was causing it.

Enter Standalone Talos

As is so often the case, Alice came to the rescue.  A while ago she put together the standalone talos package, and I keep being happy that she did.  Running talos on your local machine is a bit of a noisy way to gather performance data (depending on the test) but it’s a great way to make your Firefox do exactly the same thing as the real talos does, for the purposes of profiling.  Which is what we did.  And it worked.

How We Dissected The Problem in Six Easy Steps, Only Two of Which Involved Cussin’

  1. Follow the standalone talos set up instructions to get your own local talos setup and pointed to a build without the offending patch.
  2. Edit the sample.config file, drop down to the bottom of the file, and delete every test you don’t care about, since a full talos run is a lot of work.  Tp, in particular, is not a fast test.
  3. Run talos with your profiler of choice.  In our case, the venerable strace. Our command looked like this:
    strace -T -o without-patch-strace.log python run_tests.py sample.config
  4. Swear, because your log is way too short, because you weren’t tracing the child processes, and hence were presumably watching some python crap. Try again with:
    strace -Tf -o without-patch-strace-AGAIN.log python run_tests.py sample.config
  5. Swear, because now you have a gigantic log with everything interleaved stupidly. Read the damned man page this time and come up with the -ff argument to generate a separate trace file for each pid:
    strace -Tff -o without-patch-strace-OMG.log python run_tests.py sample.config
  6. Repeat step 5 after pointing Talos at a build with the patch (and presumably changing the output filename).

This will leave you with a large number of trace files, but it’s pretty easy to recognize the ones you care about.  In Shawn’s case, he just grep‘d out the ones that referenced places.sqlite, and behold, he had before and after traces of precisely the same startup sequence that talos was seeing.

From there it was a short trip to see a massive increase in fsync calls, something we did not expect, but which certainly explains the lag, given the problems we’ve had historically with fsync on Linux.

Total execution time, before and after
Total execution time, before and after

Needless to say, Shawn is on it.