One feature I’ve been talking a lot about is our support for Geolocation. I think that once Firefox 3.5 gets out there and sites realize they have a (privacy- and user-control-respecting) way to ask their users for their location in the world, all kinds of great services will show up. Flickr already has a photos-near-you feature, for instance, and I imagine mapping sites, restaurant reviews, and others are hot on their tails.

So I’m sure, in short order, that this won’t be necessary. In the meantime, if you’re running one of the Firefox 3.5 Release Candidates, you can use this bookmarklet to inject your current location into the google maps search box, so that you can base searches off your current location:

javascript:function sv(s){document.querySelector("#q_d").value=s};sv("Checking...");navigator.geolocation.getCurrentPosition(function(a){c=a.coords;sv(c.latitude+"%20"+c.longitude);document.forms.q_form.submit();},function(){sv("Rejected!")});

If you haven’t used a bookmarklet before, it’s easy. Open up your bookmark manager, decide where you want to put this (I like to have them on my bookmarks toolbar, since I use them a lot), and create a new bookmark. When it asks for a location, put in the code pasted above. Now, when you’re on the google maps site, click the bookmark to jump to your current location (after, of course, giving your consent).

This bookmarklet is specific to google maps (but I bet you can hack it!), and it certainly requires you to be using a modern browser with support for these features.  If you don’t have the latest Firefox yet, you can become part of our early testing community by downloading a copy now.

[Update: Changed the bookmarklet code a little to give some feedback immediately by letting you know it's checking. I bet someone out there has already made a version of this that's half as long, and twice as powerful. Comment!]

She was making a point about how Google gives users control over the kind of ads they see, and she mentioned this:

I think I always knew that the “Ads by Google” text at the bottom of ads was clickable – I’ve probably even clicked it. Historically though, it’s just been a sales pitch for would-be advertisers and content authors.  Now, when you click on it (go on, there’s one at the bottom of this post), there’s a link to your very own “Ad Preferences Manager.”

This page tells you what Google thinks you’re interested in based on the browsing habits it’s observed, and hence what kinds of ads it wants to show you (seriously, go check it out).  It also gives you the option to add/remove interests, or opt out entirely.

Betsy, from Google, was talking about how they had been trying to really get the word out to people about this interface, so that people could control their ad experience. I wasn’t sure whether that message was reaching people – even people who might care about the information advertisers collect.

A couple of questions, then:

1. Did you know about this page?
2. Do the contents there surprise you?  How accurate are they?
3. How does it all make you feel? Are you more comfortable, knowing that you have some control? Or are you less comfortable, seeing the profile laid out like that?
4. Did you make any changes while you were there?

Stick that in your perspective and smoke it. It’s worth clicking through to the HD version.

Easily the most often requested improvement was to include intermediates in the database. People wanted to see which issuers had a bunch of subordinate CAs and which issued right from the root. They wanted to see what kind of key sizes and algorithms CAs chose, and how they compared to the key sizes and algorithms used in regular site certs.

I’ve gone and re-crawled to gather that information now, and you can download the zipped db (509M). It’s still an SQLite3 database, though I’ve changed the schema a bit, with certificates now stored in their own table.  Let me know in the comments/email if you need help working with the data.

The schema, if you can call it that, was 100% expediency over forethought, so I would welcome any suggestions on DB organization/performance tweaking. I have done no optimizing so low-hanging fruit abounds, and a complicated query can take more than a day right now, so your suggestions will have visible effects!

How Did It Go?

I think it went quite well. Of course, it’s hard to nail down short term success criteria for conversations with parliamentarians. A meeting like that is not going to end with a legislator standing up and saying “I agree. Let’s go pass a law.” Things like this are an exercise in advocacy: “Here is my opinion of the situation and the options under discussion for its remedy,” followed by others giving their versions of the same thing.

I do feel, though, that my opinion was listened to, understood, and amplified by others. The room included, in addition to invited experts and press, at least half a dozen Lords, and 3 or 4 MPs, so I am also confident that I was heard by people in a position to act on what they hear.

What Did You Say?

A couple of things. I said that this kind of data collection is not something users can be expected to understand and, if they did understand it, not something they have much ability to avoid.

I said that in many markets, even developed ones like Canada and Britain, there isn’t enough choice in ISPs to make “voting with your wallet” a realistic option for people who find this kind of surveillance invasive.

I said that the technological mechanisms for preventing this are prohibitively expensive (in the case of things like “universal SSL deployment”), largely ineffective (since traffic analysis would still be possible), and brittle (opt-out cookies assume you never switch computers or browsers, that you never reinstall or move houses, that you won’t be worn down to the point of surrender by the Nth attempt to opt out).

I said that, historically, anonymized data isn’t. The AOL data was blown wide open, for instance, and that was just search terms, not browsing history. I said that however ironclad Phorm’s current processes may be, this kind of data collection being done by multiple companies over any interesting period of time will almost certainly result in anonymity failures.

I said that the collection of this information is insidious, that however noble and scoped the initial goals, it tends towards exploitation because it is too valuable not to.

After saying a chunk of that in a single burst, I got some applause from some of the people in attendance which felt odd, but certainly seemed to suggest that I had struck a chord.

What are the Lords Like?

Parliamentarians, really, since there were MPs there, but in any event I was impressed, particularly by Baroness Miller, who organized the event.  She was exceptionally good at running a room – at ensuring that legislators’ questions were answered, at bringing digressions back around to the central themes, and ensuring that multiple voices were heard. As a group, they were forthright but unapologetic about their lack of technical knowledge (that’s not their job), and asked clear questions aimed at understanding the legislative implications of various details.

Were there Swords? Powdered Wigs? Snuff Boxes?

People from the UK know that their legislators are basically like other legislators, albeit with more exciting titles. To the rest of us though, the whole thing sounds very romantic, and we entertain positively ridiculous notions like this. No swords, no wigs, no “Yes, your exalted worshipfulness.”  The houses of parliament are guarded by perfectly normal police officers with perfectly normal frowns and perfectly normal assault rifles, but very little pomp.

What about the Building?

Imagine that your great great great grandparents and their friends had all the money in the country, and decided to build a place to hang out. Imagine that since then, it’s where everyone decided to put their cool stuff.  Imagine walking through rooms, separated by wooden doors older than calculus. Imagine those rooms are alternately filled with statues, murals, statues in front of murals, framed masterworks, and leather bound books about anything that could matter. Imagine that there are entirely different paths, staircases and elevators for peers of the realm than for everyone else.  Imagine that you could fit your current house inside the Queen’s entrance and have room to fly a kite from the roof.

It is a nice building.

Would you do it again?

Yes.  Yes I would.

I still think that legislating technology is fraught with peril. The way to mitigate that peril is not to run away from it, though, but to be a voice for the kind of change we want, and against the kind of change we don’t.

Is the Bowmore 17 you brought back tasty?

Yes.

Brief Background

Phorm’s technology hooks in at the ISP level, watches and logs user traffic, and uses it to assemble a comprehensive profile for targeting advertising. While an opt-out mechanism was provided, many users have complained that there was no notice, or that it was insufficiently clear what was going on. NebuAd, another company with a similar product, has apparently used its position at the ISP level to not only observe, but also to inject content into the pages before they reached the user.  It’s hard to get unbiased information here, but this is what I understand of the situation.

Thoughts

1.  Deep packet inspection, in the general case, is a neutral technology. Some technologies are malicious by design (virus code, for instance), but I think DPI has as many positive uses as negative. DPI can let an ISP make better quality of service decisions, and can be done with the full knowledge and support of its users. I don’t think DPI, as a technology, should be treated as insidious.

2. Using deep packet inspection to assemble comprehensive browsing profiles of users without explicit opt-in is substantially more questionable. My browsing history and habits are things I consider private in aggregate, even though any single visit is obviously visible to the site I’m browsing.

It’s possible that I will choose to allow this surveillance in exchange for other things I value, but it must be a deliberate exchange. I would want to have that choice in an explicit way, and not to be opted in by default, even for aggregate data. Moreover, given the complexity of this technology, I would want a great deal of care to go into the quality of the explanation.  Explaining this well to non-technical users might be so difficult as to be impossible, which is why it’s so important that it be opt-in.

3. Using deep packet inspection in conjunction with software that modifies the resultant pages to include, for instance, extra advertising content, is profoundly offensive and undermines the web. The content provider and the user have a reasonable expectation that no one else is modifying the content, and a typical user should not be expected to understand the mechanics of the web sufficiently to be able to anticipate such modifications.

Solutions

As a browser, we do some things to help our users here, but we can’t solve the problem. https resists this kind of surveillance and tampering well, but requires sites to provide 100% of their content over SSL. Technologies like signed http content would prevent tampering, if not surveillance, but once again assume that sites (and browsers!) will support the technology. Ad blockers can turn off any injected ads, tools like NoScript can de-fang any injected javascript but, fundamentally, http content is not tamper-proof, and no plaintext protocol is eavesdropping proof.

People trust their ISPs with a huge amount of very personal data. It’s fine to say that customers should vote with their feet if their ISP is breaking that trust, but in many areas, the list of available ISPs is small, and so the need for prudence on the part of ISPs is large.

That’s what I’m thinking so far, what am I missing?

First of all, bookmark that action.

As for how it works, a couple minutes of playing around should explain it.  Basically it’s a running changelog with associated builds and status, plus an at-a-glance view of the tree in the bottom right, and details of a selected run in the bottom left.

If it were covered in shaved black truffle and a velvety 25 year old balsamic, it would be only marginally more delicious.

If you see mstange on IRC, give him your love.  If you have suggestions, give him your patches.

[Update: Why yes, it does require querySelectorAll support.  I trust you have a browser that does that, right?]

• What proportion of CA-signed certs are using MD5 signatures?
• What key lengths are being used, with which algorithms?
• Who is issuing which kinds of certificates?

So I decided to go get some of that information, so that I could give it to all of you wonderful people.

What I Did

I put together some python to crawl a list of sites and record the details of their response to an SSL handshake into an sqlite3 database. I don’t know python, and my SQL is extremely rusty, but it works.  The code is here, take it and make it better!  Pay particular attention to the TODO list!

As for a list, I used Alexa’s list of the top 1,000,000 sites, which they quite helpfully make available for free download. Of course we can have all kinds of fun debating whether this is the right list to use, it’s obviously going to have some skews. Anyone with a similar list can either hook me up and I’ll give it a whirl, or download the code and run it themselves.  This list did get me 382,860 certificates from the public internet though, so that’s pretty okay.

What I Can Do With It

For each host in the list, I currently record:

• Whether the connection succeeded or not, and if not, why
• The verification result, using the Mozilla list of CAs
• Various and sundry connection/certificate details (subject, issuer, cipher, keylength)
• The PEM-encoded end-entity cert, for post-analysis

What that means is that I can answer some reasonably relevant questions. For instance, during the recent excitement over MD5 weaknesses, we have been having conversations about retiring MD5 as a supported signature algorithm as, I’m certain, have the other browsers. Making that decision in an informed way requires us to understand how much of the internet still relies on MD5, though, and when those certificates will expire.

What Can You Do With It?

There’s almost 400,000 certificates, all told, in a big SQL-queryable database. Want to see a total breakdown of certs by issuer? By verification code? Want to see the distribution of key lengths?  Or cipher suites?

Maybe you write software that processes certificates – want 400,000 real world examples to test against?  As far as I know, this kind of data hasn’t been available before without paying for it, so I’m actually really interested to know what you can do with it.

How Much of the Internet is this?

Good question! Other estimates I’ve seen for the total population of servers responding to SSL hails out there is about 1-4M.  Based on that, I’d say this is probably about 10-20% of the secure internet, but I wouldn’t try to use this data to make magnitude assessments; it’s better suited to proportions and comparative work, really.  If my estimates are close, then this is a big enough chunk of the total population to produce pretty good data.  Remember, it will exhibit the skews you’d expect from sampling the more popular sites, but that will also serve to weight it towards the certs people are more likely to see. If that’s not the bias you want, use a different list!

The Goods

• The database (gzip’d, sqlite3 format, 367MB) for my Jan 15 crawl.
• The code I used to gather it.  (It needs love, love it.)
• If you want to play with something a little less gargantuan, I’ve also put up a trimmed version with just the top 10,000 (gzip’d, 3MB).

If you find good stuff in here, I hope you’ll leave a comment letting me know.  If you can’t access the database or download the file and want me to run a query against it, let me know.  I only connect to each host once, and all I do is open an SSL connection, so the load on the servers is non-existent.  The load on my machine while running it though, can be heavy.  To keep the interruption to a minimum, I use conservative settings which make a crawl take about 40 hours, so until I have it running in parallel on a rack of excitingly fast machines, please understand that requests for re-crawls will take a while.

I think “memes are fun” and “I hate memes but this one’s okay” and “I’m a grumpy buzzkill” have all been taken as reactions to this thing already.  I guess “we are social little monkeys whether we acknowledge it or not, and it’s okay to do silly things which feed that need” is what I’ll go with, though I believe it overlaps with some existing responses too.  I do think I’m the first one on planet to mention monkeys in this context, though, so I’ll cling to that shred of originality.

The Rules

• Link to your original tagger(s) and list these rules in your post. I have the distinct honour of having been pinged by veritable titans of the internet age. You may know them as Beltzner, Campbell, Finette, and Slater, but to me they’ll always be Mike, Rob, Jane and John.
• Share seven facts about yourself in the post. Yes well, there are sections for that, you know.
• Tag seven people at the end of your post by leaving their names and the links to their blogs. Likewise.
• Let them know they’ve been tagged. Quite.

The 7

1. I used to have my own television show.  It was called The Voice on Rogers Cable 10, Brampton.  It was a teen affairs show and I was the host.  I actually had no particular interest in it, but I ran with a crew that included a lot of television nerds, who took courses in television production, and since all the fun was behind the camera, they shoved me out in front.  This is where I first learned that in television, “The Talent” has a meaning opposite to the one you’d expect.  We did 3 shows, the last featuring a live performance from a local grunge band.  Awesome.
2. I have a reasonably excellent keloid on my right arm as the result of a scald from a cup of tea. I was probably 10 or 11 at the time, and I flomped down on the couch one evening when a cup of tea, erstwhile sitting pleasantly on the armrest, leapt from its resting place in the ensuing compression wave and came to rest on my arm and stomach getting what I can only describe as my undivided attention. I remember rushing to the shower to douse it with water. I remember being in shock, shuddering about every 5 seconds; vibrating, really. No hospital, no skin graft, although I did wear a Jobst™ compression sleeve for most of grade 6 to keep the scarring down. These days, it’s just a gnarly scar, but in high school I had about half my friends convinced that I’d tried to bite my arm off, as a baby.
3. When I left university, I weighed a hair under 300lbs. I’m 6′4″ or so, which makes that slightly less egregious than it might otherwise be, but under no circumstances can it be understood to be particularly healthy.  I have no idea how it happened.  I lost it through boring old “eat less, exercise more” so don’t expect any winning diet tips here.



4. Over the years my parents have furnished me with, in addition to a mother and father (implicitly): 2 step-fathers, 1 step-mother, 5 step-brothers, 2-step sisters, 2-step-brothers-in-law, 2 step-sisters-in-law, plus a brother and sister that, while nominally “half-siblings,” I grew up with and disregard any supposed half-ish-ness.  This has allowed me, in various contexts, to be the oldest child, the youngest child, the middle child and an only child.  Stick that in your pipe and smoke it, Alfred god-damned Adler.
   



5. I grew up in a little house in the suburbs which, despite being a little house in the suburbs, housed over 100 animals.  We had, at one point or another: cats, dogs, rats, mice, gerbils, hamsters, degus, chinchillas (chinchillae?), parakeets, budgies, love birds, cockatiels, grey parrots, rabbits, guinea pigs, baby Russian tree squirrels, baby feral black squirrels, sugar gliders, anoles, salamanders, snakes, plus an 85-gallon freshwater fish tank with red tailed sharks, plecostamii (plecostamuses?), tetras (neon and otherwise), guppies, angel fish, texas cichlids, kuhli loaches, goldfish and, of course, snails. This is not because our parents were over-indulgent of our whims, rather the reverse, we kids indulged mom; we also loved it. She is down to 4 cats, 1 dog now, but also up 5 chickens.



6. I read a lot, on a variety of subjects, but I have a profound inability to resist books which tell the history of the world through the lens of a single commodity or substance, e.g. Cod, Salt, Horse, Oak, The Secret Life of Lobsters, A Perfect Red, Pigeons, A History of the World in 6 Glasses, Crows, Parasite Rex. I am similarly drawn to books that document usually invisible subcultures, like Body Brokers, The Island of the Colorblind, Word Freak, Gang Leader for a Day, The Game, and Candyfreak. I’d say less than 20% of my reading is fiction, for which I sometimes feel bad, like I’m letting my literary brain die, but the mind wants what it wants.
7. I, like a surprising number of people here, married my highschool sweetheart. Amy Nightingale (née Lush, yes really) and I started dating in grade 10, less than a month after she stopped dating my best friend (saucy!).  I continue to be astonished at the amount of patience she exhibits in our day to day lives (”I just bought a new bench grinder to help make my own lockpicks!”, “I’m going to make homebrew beer in the kitchen!”, “I’m going to walk to the auto parts store, because I just drained the car of oil and now I can’t get the filter off!”, “I’m going to stop shaving for a month, and get the other guys at the office to similarly annoy their wives!”) and intimidated by her ability to show her emotions more honestly and more bravely than I am capable of doing.  I have been blessed to have a lot of really amazing women in my life (you know who you are), but I hope none of them will object to me pointing out that she is my favourite.

   

The Tagging Who’s left?  I mean, honestly.  I am basically gliding down the stale end of this adoption curve, aren’t I?  Here are some people more laggard than I.

• gen – Not only do we need more Mozilla Japan representation, but Gen is also so awesome he’s illegal in 14 states.
• thunder – Because I never get to see Dan any more.
• pav – Honestly, how has Stuart been allowed to slack this long?
• sayre – I know it violates some new york aloofness statute, but your public wants to know.
• mary – Because I like her glasses and her moxie.
• neil – What spirits haunt the mind of a man who chooses to work on focus bugs, and also writes comedies for Second City?
• davidb – David just started in the Toronto office, and while he’s giving away our secrets, it’s only fair he should share some of his own.
  …
• and Amy – who can spread this pox to whole other populations, if she ever decides to blog again.

Sorry about the Java – I would love to see someone amazingly awesome do this using SVG maybe, or JS+Canvas. I would give that person a bottle of reasonably nice wine, in fact!

In other news: NSID continues on apace, and is getting delightfully scruffy! Join the revolution!  If you’ve quit, rejoin!  Who dares oppose?