Cloud Hosting

The impact of pulling those certificates would be swift and severe. Once browsers like Chrome and Firefox found them missing, they would flash warnings to any visitors that the sites werent safe. Some browsers would block access altogether. A not insignificant chunk of the internet would effectively be taken out of commission. All because of this one small flaw in one niche corner of the Lets Encrypt operation.

Within two minutes of confirming the bug, the Lets Encrypt team stopped issuing any new certificates in a bid to stanch the bleeding. A little over two hours after that, they fixed the bug itself. And then they let everyone know what was coming.

We cant contact everybody, so we started contacting the largest subscribers, telling them about the situation, getting them as informed as possible, says Aas. And then we worked with them to get them to replace their certificates as quickly as possible.

Once a site operator renewed a certificate, Lets Encrypt could safely revoke the old one. No harm would befall the site. Which sounds like a simple enough solutionbut nothings simple at this kind of scale.

Bigger organizations had an easier time fixing the problem, because they generally have the resources to monitor any signs of trouble that surface and the tools to automate the renewal process. If youve got a dozen or two dozen servers or something, thats some poor sleepy-eyed soul in the middle of the night at a keyboard, says MongoDBs White. We reissued a little over 15,000 certificates [for clients], and we did it in a few hours. There was some work involved, but it wasnt catastrophic. We had measures in place to be able to rotate quickly.

Smaller sites got a big assist from the Electronic Frontier Foundation, which operates Certbot, a free software tool that automatically adds Lets Encrypt certificates to sites and renews them every 60 days. In the last two months alone, Certbot has generated certificates for 19.2 million unique sites. Fortunately we had anticipated the need to check revoked certificates for renewal in 2015, says EFF engineering director Max Hunter. Because Let's Encrypt communicated the issue early, and the code path for the query was already in place, our work was relatively straightforward. By Tuesday a team from EFF, along with volunteers in Paris and Finland, had updated Certbot to renew any revoked certificates.

Meanwhile, Lets Encrypt sent an email to every address it had on file. It created a searchable database of every affected domain so that hosting companies could see if they needed to act. We marked those certificates as expired in our internal system, and then our normal automated processes kicked in to generate and deploy new certificates, says Justin Samuel, CEO of Less Bits, a startup that operates hosting company ServerPilot.

On Tuesday night, 30 minutes before the deadline, Lets Encrypt made another announcement. Of the 3 million potentially impacted sites, 1.7 million had managed to renew their certificates, an astonishing number given the short window of time. No other CA comes close to making large-scale cert reissuing not only feasible but also fast, says Samuel.

That success also emboldened Aas to make a difficult call. Lets Encrypt would let the remaining certificates slide. We made the decision that instead of breaking more than a million websites, potentially, we just arent going to revoke them by the deadline, says Aas. We think its the right decision for the health of the internet.

It was the internet equivalent of a call from the governor minutes before midnight. Lets Encrypt will continue to revoke certificates if it can confirm that the sites have renewed them, but otherwise it is content to leave them be in their slightly broken form. The security risk is small, Aas says, and since Lets Encrypt certificates are only viable for 90 days to begin with, any stragglers will have washed out of the ecosystem by summertime at the latest.

If anything, this just reinforces that they are one of the most transparent, modern certificate authorities in the world, says MongoDBs White, who points to previous certificate snafus that for-profit companies like Symantec have badly mishandled. Its easy to armchair quarterback. But I think if people are overly critical thats misplaced.

The intricacies of internet infrastructure are generally ignored until something goes terrible wrong. This time, though, its useful to reflect on what went right. For once, the story is that nothing broke.

More Great WIRED Stories

See the original post:
The Internet Avoided a Minor Disaster Last Week - WIRED