The impact of pulling those certificates would be swift and severe. Once browsers like Chrome and Firefox found them missing, they would flash warnings to any visitors that the sites werent safe. Some browsers would block access altogether. A not insignificant chunk of the internet would effectively be taken out of commission. All because of this one small flaw in one niche corner of the Lets Encrypt operation.
Within two minutes of confirming the bug, the Lets Encrypt team stopped issuing any new certificates in a bid to stanch the bleeding. A little over two hours after that, they fixed the bug itself. And then they let everyone know what was coming.
We cant contact everybody, so we started contacting the largest subscribers, telling them about the situation, getting them as informed as possible, says Aas. And then we worked with them to get them to replace their certificates as quickly as possible.
Once a site operator renewed a certificate, Lets Encrypt could safely revoke the old one. No harm would befall the site. Which sounds like a simple enough solutionbut nothings simple at this kind of scale.
Bigger organizations had an easier time fixing the problem, because they generally have the resources to monitor any signs of trouble that surface and the tools to automate the renewal process. If youve got a dozen or two dozen servers or something, thats some poor sleepy-eyed soul in the middle of the night at a keyboard, says MongoDBs White. We reissued a little over 15,000 certificates [for clients], and we did it in a few hours. There was some work involved, but it wasnt catastrophic. We had measures in place to be able to rotate quickly.
Smaller sites got a big assist from the Electronic Frontier Foundation, which operates Certbot, a free software tool that automatically adds Lets Encrypt certificates to sites and renews them every 60 days. In the last two months alone, Certbot has generated certificates for 19.2 million unique sites. Fortunately we had anticipated the need to check revoked certificates for renewal in 2015, says EFF engineering director Max Hunter. Because Let's Encrypt communicated the issue early, and the code path for the query was already in place, our work was relatively straightforward. By Tuesday a team from EFF, along with volunteers in Paris and Finland, had updated Certbot to renew any revoked certificates.
Meanwhile, Lets Encrypt sent an email to every address it had on file. It created a searchable database of every affected domain so that hosting companies could see if they needed to act. We marked those certificates as expired in our internal system, and then our normal automated processes kicked in to generate and deploy new certificates, says Justin Samuel, CEO of Less Bits, a startup that operates hosting company ServerPilot.
On Tuesday night, 30 minutes before the deadline, Lets Encrypt made another announcement. Of the 3 million potentially impacted sites, 1.7 million had managed to renew their certificates, an astonishing number given the short window of time. No other CA comes close to making large-scale cert reissuing not only feasible but also fast, says Samuel.
That success also emboldened Aas to make a difficult call. Lets Encrypt would let the remaining certificates slide. We made the decision that instead of breaking more than a million websites, potentially, we just arent going to revoke them by the deadline, says Aas. We think its the right decision for the health of the internet.
It was the internet equivalent of a call from the governor minutes before midnight. Lets Encrypt will continue to revoke certificates if it can confirm that the sites have renewed them, but otherwise it is content to leave them be in their slightly broken form. The security risk is small, Aas says, and since Lets Encrypt certificates are only viable for 90 days to begin with, any stragglers will have washed out of the ecosystem by summertime at the latest.
If anything, this just reinforces that they are one of the most transparent, modern certificate authorities in the world, says MongoDBs White, who points to previous certificate snafus that for-profit companies like Symantec have badly mishandled. Its easy to armchair quarterback. But I think if people are overly critical thats misplaced.
The intricacies of internet infrastructure are generally ignored until something goes terrible wrong. This time, though, its useful to reflect on what went right. For once, the story is that nothing broke.
More Great WIRED Stories
See the original post:
The Internet Avoided a Minor Disaster Last Week - WIRED
- Is COVID-19 Making the Internet Sick? - Government Technology - May 27th, 2020
- Thanks to Physics, This Chocolate Is Iridescentand Safe to Eat - Smithsonian.com - May 27th, 2020
- $100 million in bounties paid by HackerOne to ethical hackers - BleepingComputer - May 27th, 2020
- Types of Encryption: 5 Encryption Algorithms & How to Choose the Right One - Security Boulevard - May 27th, 2020
- Asian consumers worried about securing their data - BusinessWorld Online - May 27th, 2020
- Move online to survive businesss new mantra - BizNews - May 27th, 2020
- China Demands Us Withdraw Sanctions on Tech Suppliers - Manufacturing Business Technology - May 27th, 2020
- DDoS Protection Market Overview, Regional And Restraint Analysis By 2020 2026 - 3rd Watch News - May 27th, 2020
- Galaxy S20 security is already old hat as Samsung launches new safety silicon - The Register - May 27th, 2020
- Amid the COVID-19 crisis and the looming economic recession, the Electronic Bill Presentment and Payment (EBPP) market worldwide will grow by a... - May 27th, 2020
- Global Internet of Things (IoT) Security Market Research Studies Competitive Strategies, Regional Analysis Forecast 2025 - WaterCloud News - May 27th, 2020
- When COVID-19 and Economic Fallout Put Millions of Kids in Unsafe Places, Communities in Schools Went in After Them. - The 74 - May 27th, 2020
- DNS over HTTPS: How to activate it on Windows 10 Build 19628 - WinCentral - May 27th, 2020
- The lack of women in cybersecurity leaves the online world at greater risk - The Conversation US - May 17th, 2020
- COVID-19 Impact and Recovery Analysis | Internet of Things (IoT) Security Market 2020-2024 | Increasing Incidence of Cyberattacks to Boost Growth |... - May 17th, 2020
- Break On Through To The Other Side - Seeking Alpha - May 17th, 2020
- Post-COVID 19: The Virtual World And Digital Participation And Its Challenges In Ghana - Modern Ghana - May 17th, 2020
- Embracing Remote Learning and Working after COVID-19 as our New Reality - THISDAY Newspapers - May 17th, 2020
- The lack of women in cybersecurity leaves the online world at greater risk - Kiowa County Press - May 16th, 2020
- Internet security Market Research Report 2020 By Size, Share, Trends, Analysis and Forecast to 2026 - Cole of Duty - May 16th, 2020
- The best antivirus protection of 2020 for Windows 10 - CNET UK - May 16th, 2020
- Bill Proposes to Incentivize Cybersecurity Innovations With Cash Prizes - Nextgov - May 16th, 2020
- The Confessions of Marcus Hutchins, the Hacker Who Saved the Internet - WIRED - May 16th, 2020
- Spains DGT Warns Of A New Scam Circulating Which Tricks Users Into Giving Their Data - Euro Weekly News - May 16th, 2020
- What is the Internet of Things? - Fox Business - May 16th, 2020
- Air Force aims to make secure mobile identity management the norm - FCW.com - May 16th, 2020
- Internet Security Market Analysis, Size, Regional Outlook, Competitive Strategies and Forecasts to 2027 - Cole of Duty - May 9th, 2020
- (2020-2026) Internet Security Audit Market to Witness Robust Expansion throughout the Forecast Period - Cole of Duty - May 9th, 2020
- OODAcast A Conversation With Lou Manousos, CEO of RiskIQ - OODA Loop - May 9th, 2020
- COVID-19 Crisis To Help Akamai Replicate $300 Million+ Revenue Growth From The Past 2 Years? - Forbes - May 9th, 2020
- Global Internet Security Market is Segmented by Applications, Technology, Product Type And Service and Region - TechnoVally - May 9th, 2020
- Google Releases May 2020 Android Security Patch; Fixes Bug That Allowed Remote Code Execution - Mashable India - May 9th, 2020
- Gregory Boehm | The Harvard Press | News | Obituaries - Harvard Press - May 9th, 2020
- Wifi/ Internet/ IoT Testing and Security Solutions Market 2020 by Company, Regions, Type and Application, Forecast to 2024 - Cole of Duty - May 9th, 2020
- More Salt in their wounds: DigiCert hit as hackers wriggle through (patched) holes in buggy config tool - The Register - May 9th, 2020
- Zoom's Rise, Reign and Era of Reform at the Top of the Teleconferencing Throne - BroadbandBreakfast.com - May 9th, 2020
- The Pleasures and Pitfalls of Motherhood on Instagram - ELLE.com - May 9th, 2020
- Global IT Security Spending in Government Market Expected to reach highest CAGR by 2025: Check Point Software Technologies, Cisco Systems, Fortinet,... - May 9th, 2020
- Millions of Android users need to update, or risk having attackers take over their phone - Express - May 9th, 2020
- Deal of the Month: 50% off Integos Mac Internet Security X9 bundle - 9to5Mac - April 20th, 2020
- COVID-19 impact: Internet Security Firewall Market: Promising Growth Outlook with a Steady CAGR of X% 2020-2026 Cole Reports - Cole of Duty - April 20th, 2020
- Russia And China Hijack Your Internet Traffic: Heres What You Do - Forbes - April 20th, 2020
- Internet security Market 2020 Break Down by Top Companies, Applications, Challenges, Opportunities and Forecast 2026 Cole Reports - Cole of Duty - April 20th, 2020
- Investor Paul Meeks overhauls tech strategy due to coronavirus risks, turns negative on two widely held stocks - CNBC - April 20th, 2020
- Authentic8's Front Line of Defense Tool Aims to Safeguard Government Agencies from Cyberthreats - WashingtonExec - April 20th, 2020
- Blockchain: The Most Awaited Ally For The Security Of The Internet Of Things - CoinCodex - April 20th, 2020
- The internet's battle against bots is heating up - The Hustle - April 20th, 2020
- Bot creates millions of fake eyeballs to rip off smart-TV advertisers - Naked Security - April 20th, 2020
- Where to buy Kaspersky Internet Security? - RecentlyHeard.com - April 17th, 2020
- Citing coronavirus disruptions, PhishCloud offers year of free service to prevent phishing scams - GeekWire - April 17th, 2020
- The Global Software Defined Perimeter Market size is expected to reach $10.7 billion by 2025, rising at a market growth of 23.7% CAGR during the... - April 17th, 2020
- Women are essential helpers during crises but they need access to the internet | TheHill - The Hill - April 17th, 2020
- How To Browse The Internet Privately on Your Phone (Our #1 Tips) - Know Your Mobile - April 17th, 2020
- Faster Internet and protection against cyberattacks: UPC Business offers customers free additional services during the coronavirus crisis -... - April 17th, 2020
- Teaching your kids to surf the internet safely - The Star Online - April 17th, 2020
- No, the Internet Is Not Good Again - The Atlantic - April 17th, 2020
- Cyberattacks on endpoints will rise by up to 40 per cent unless we act quickly - Techerati - April 17th, 2020
- RBR's CyberPatriots Continue their Winning Ways - The Two River Times - April 17th, 2020
- The Weaponization of Dogs on the Internet - Lawfare - April 17th, 2020
- Investment opportunities in the internet sector - Times of Malta - April 17th, 2020
- Matt Hancock has no answers for anything but he does have a six-point plan and a very small badge - The Independent - April 17th, 2020
- Best internet security suites of 2020: anti-virus and anti-malware cyber security - TechRadar - April 17th, 2020
- The security conundrum of 5G network slicing - Urgent Communications - April 13th, 2020
- How to Make Sure that Antivirus is on your Endpoints - Security Boulevard - April 13th, 2020
- Why you can't trust your vote to the internet - CyberScoop - April 13th, 2020
- IoT security, neglected infrastructure, and a crisis of trust deemed major threats for 2022 - TechRepublic - April 13th, 2020
- Aspects of cybersecurity not to overlook when working from home - Big Think - April 13th, 2020
- The Rise of the Secure Internet Gateway - Communal News - April 13th, 2020
- The University and its students must be more considerate of essential USC staff - Daily Trojan Online - April 13th, 2020
- Foreign Operatives Allegedly Using Zoom To Spy On Americans - Brinkwire - April 13th, 2020
- Top Ways to Guard Against Work-from-Home Phishing Threats - Infosecurity Magazine - April 13th, 2020
- Get 2 years of Webroot internet security and antivirus for $50 at Amazon - BGR - April 12th, 2020
- Internet Security Software Market Growth Analysis, Top Manufacturers, Shares, Growth Opportunities and Forecast to 2026 - Germany English News - April 12th, 2020
- Foreign Spies Are Targeting Americans on Zoom and Other Video Chat Platforms, U.S. Intel Officials Say - TIME - April 12th, 2020
- iOS users beware: Myth of Apple security invulnerability is just that - The Star Online - April 12th, 2020
- Experts: Internet voting isn't ready in the face of coronavirus pandemic - CyberScoop - April 12th, 2020
- Slack in the security spotlight lessons for collaboration servers - Naked Security - April 12th, 2020
- Google removes Android VPN with critical vulnerability from Play Store - Naked Security - April 12th, 2020
- Is there a way to find this file that's being used by another processor? - Windows 10 Support - BleepingComputer - April 12th, 2020
- Global Internet of Things (IoT) Security Industry 2020 Market Research With Size, Growth, Manufacturers, Segments And 2026 Forecasts Research -... - April 12th, 2020