Seattle More than 2 billion people log into Facebook every month. Every day, the social-media crowd uploads billions of photos, calls up hundreds of millions of hours of video, and fires off a prodigious flurry of likes and comments. Somebody has to store that slice of humanitys digital record.
Much of that task falls to Surendra Verma, a Seattle engineer who for more than 20 years has been building software that files away and retrieves large volumes of data.
Verma leads the storage team at Facebook, the group charged with making sure that the social network can accommodate its daily deluge without losing someones wedding photos.
Most of that unit is based in Seattle, part of a workforce that today numbers 1,600 people, up from just 400 three years ago. That makes Facebook one of the fastest- growing technology companies outside of Amazon, anyway in the city.
While Facebook employees work on a wide range of products in Seattle, the office has developed a specialty in the geeky realm of systems software.
About a quarter of the Facebook engineers in Seattle work on the companys infrastructure projects, the tools to transmit, store and analyze the growing heap of data people feed into the social network.
Thats a common trade in the region, where Amazon Web Services, Microsoft and Google are all building their own clouds giant, globe-straddling networks of data centers and the software that manages them.
Facebook could have built its products on computing power rented from those cloud giants, but it decided to build its own tools, from custom hardware designs all the way to mobile applications. Supporting Facebooks network are nine massive data centers a 10th, in Ohio, was announced earlier this month.
Facebooks cloud is different from the others in that its designed to support just one customer: Facebooks own apps.
They happen to be some of the most widely used pieces of software in the world and their use keeps expanding.
Verma is an Indian-born engineer who got his start at IBM before moving to Microsoft, where he worked most recently on Windows file systems. He joined Facebook in 2015.
By then, the Seattle office had come to pilot Facebooks storage-software efforts. We could find some very good engineers here, he said. And so a bunch of these projects started, and just got momentum from there.
His teams job, he said, is to be invisible to Facebooks product groups, letting the companys other engineers build whatever they can think up on top of a reliable foundation.
But a wave of new services, and the rapid growth in the number and quality of videos and photos that people share, is putting a huge burden on the networks infrastructure.
Facebook in January 2016 started a monthslong rollout of live video streaming, a milestone in the social networks effort to compete with popular streaming services.
Then Instagram Stories, aimed at competing with the ephemeral photo-sharing application Snapchat, launched last August, and quickly made its way to 250 million monthly users. (Facebook bought Instagram in 2012.)
Up next: live video broadcasts on Instagram, called Live Stories, a feature the product group hoped to launch before Thanksgiving.
Vermas team wasnt sure its systems could meet that deadline, and negotiated a few days of delay on the planned start date. After scrambling, the team scraped together the internal storage space needed to accommodate the new feature, which went live Nov. 21.
We looked very hard at our capacity, everywhere, Verma said. We scrounged everything we could, looked at nooks and crannies.
One of the people doing the looking was J.R. Tipton, a manager on Vermas team.
Tipton left the Chicago suburbs and came to Seattle in 2001 for the same reason as thousands of others in the 1990s and 2000s: a job offer from Microsoft.
Fifteen years later, he became part of another phenomenon reshaping the region, opting to leave maturing Microsoft for a job with a younger tech giant in the area.
I wanted an adventure, Tipton said of his move to Facebook last year.
Tipton and Verma are among the 640 people at Facebook in Seattle who, on LinkedIn, list past experience at Microsoft.
Seattle, home to Boeings legions of engineers long before Microsoft came along, has never really been a one-company town. But the range of options for technologists has ballooned in the last decade, with Amazon uncorking another Microsoft-like growth spurt, and Silicon Valley giants seeding local outposts to scoop up some of the talented software developers here.
When Facebook set up shop in Seattle in 2010, it was the companys first U.S. engineering office outside its Menlo Park, Calif., headquarters. Last year, Facebook Seattle, which then numbered about 1,000, moved into nine floors of custom-designed office space. That was followed in short order by two more leases that would give the company enough space for about 5,000 employees in Seattle.
Today, Tipton works on the Facebook system that is the last line of defense keeping peoples photos and videos from getting wiped out by a software error or power outage at a data center.
That would be cold storage, the backup system storing redundant copies of all of the photos, videos and other data Facebook stores.
The engineers who designed the system in 2013 wanted to build something more efficient than the typical rack of servers, which suck up energy both to keep their hard drives powered on and to run the fans and air-circulation systems that keep the stack of electronics from overheating.
The company landed on a design that leaves most hard drives in Facebooks rows of servers powered down until needed, hence the cold in cold storage. The software built to run that system was designed to handle plenty of growth in the amount of data people were throwing at Facebook.
Each cold-storage room in Facebooks data centers was built to house up to one exabyte of data. Thats 1,000 petabytes, or 1 billion gigabytes, or the combined hard drive space of 500,000 top-of-the-line MacBook Pro laptops. At the time, it seemed laughably huge, Tipton said of the software layer built to manage all of that.
Last year, it became clear it wouldnt be enough, the result of growth in both the number of Facebook users and the launch of data-intensive features like Instagrams Live Stories.
Tipton and his Seattle-based group spent the first half of 2016 taking a long look at the cold-storage software, which was starting to show its age in a world of 360-degree video. They could invest more time and energy in keeping the old architecture working, or rebuild.
They opted to rebuild.
It has to be much, much bigger, he said.
Tipton has metaphors at the ready to describe that continuing challenge, the fruit of years explaining his day job to nontechies.
You stand on top of a hill and see the buffalo herd coming, he said.
And if that doesnt sound harrowing enough, he tries again: Were basically laying the railroad track down as the train is coming.
Here is the original post:
Digital Deluge on the Cloud - Valley News
- Unlocking the Key to the Cloud - Security Boulevard - January 23rd, 2020
- What is the difference between SaaS and Cloud Computing - WhaTech - January 23rd, 2020
- Cloud Database and Database as a Service (DBaaS) Market Analysis on Top Key Vendors, Revenue Growth and Business Development Forecast 2017-2025 -... - January 23rd, 2020
- Helsinki-based UpCloud raises 18 million to rapidly grow its team and cloud computing services - Tech.eu - January 23rd, 2020
- Nextcloud Hub takes on Google Docs and Office 365 - ZDNet - January 23rd, 2020
- Google Cloud Run: What every IT Ops team should know - TechBeacon - January 23rd, 2020
- Microsoft's Nadella On Why The Future Is All About Cloud; Buyers Are Into TikTok's Self-Serve Ad Platform - AdExchanger - January 23rd, 2020
- Why Many Tech Companies are Turning to the Hybrid Cloud to Tackle Today's Business Challenges - Data Center Frontier - January 23rd, 2020
- Global Cloud Server Market 2020 By Analysis -Worldwide Opportunities, Revenue, Production, Demand and Geographical Forecast To 2025 Dagoretti News -... - January 23rd, 2020
- The trillionaires club: How big data, cloud and AI are powering the tech economy - SiliconANGLE - January 23rd, 2020
- Nebulon gets less nebulous, aims to replace SAN and hyperconverged storage - Blocks and Files - January 23rd, 2020
- Global Private Cloud Server Market Growth Opportunities, Challenges, Competitive Analysis And Forecast 2020-2025 - Expedition 99 - Expedition 99 - January 23rd, 2020
- Experts discuss industry response to multicloud - IT Brief New Zealand - January 23rd, 2020
- Data Governance And Smart Cities Are Helping Improve Quality Of Life In Japan - Forbes - January 23rd, 2020
- The Benefits of Identity Management for Healthcare Businesses - Solutions Review - January 23rd, 2020
- How the cloud went from 0 to 100 in ten years - Express Computer - January 23rd, 2020
- Suspect Wanted in May Shooting Arrested in St. Cloud - WJON News - January 23rd, 2020
- Wyze data leak: Key takeaways from server mistake that exposed information from 2.4M customers - GeekWire - January 1st, 2020
- Conquering the Cyber Security Challenges of the Cloud - CPO Magazine - January 1st, 2020
- City of Cottonwood now a certified sustainable business | Sedona.Biz - The Internet Voice of Sedona and The Verde Valley - Sedona.biz - January 1st, 2020
- A restaurant server got a massive tip and it is helping to change her life - KFOR Oklahoma City - January 1st, 2020
- Cloud Hopper Attacks Far More Extensive than First Thought - CloudWedge - January 1st, 2020
- 2019: The Year in Hyperscale - Data Center Frontier - January 1st, 2020
- Using the Windows Admin Center Azure services feature - TechTarget - January 1st, 2020
- Privacy scare leads Wyze to unpair all devices from Google Assistant and Alexa, you'll need to add them back (Update: Further responses) - Android... - January 1st, 2020
- 5 top tech predictions for 2020 and beyond - BusinessTech - January 1st, 2020
- Exposed databases are as bad as data breaches, and they're not going anywhere - CNET - December 28th, 2019
- From bookstore to cloud hosting: How Amazon 'took the lead' in the technology world - CBC.ca - December 28th, 2019
- Bare Metal Cloud Market Future Aspect Analysis and Current Trends by 2017 to 2025 - Market Research Sheets - December 28th, 2019
- Microsoft Stock Is Floating on a Cloud - Investorplace.com - December 28th, 2019
- How Important Is The Intelligent Cloud Segment To Microsofts Stock? - Forbes - December 28th, 2019
- Privacy scare leads Wyze to unpair all devices from Google Assistant and Alexa, you'll need to add them back - Android Police - December 28th, 2019
- To protect data and code in the age of hybrid cloud, you can always turn to Intel SGX - The Register - December 28th, 2019
- The silver lining in cloud for financial institutions - ITProPortal - December 28th, 2019
- Server Microprocessor Market Projected to Grow at an Impressive CAGR Of XX% Between 2017 2027 Bulletin Line - Bulletin Line - December 28th, 2019
- Listen to the boomers - or cloud could make you go bust - ITProPortal - December 28th, 2019
- Dr. Max Welling on Federated Learning and Bayesian Thinking - Synced - December 28th, 2019
- 7 crackpot technologies that might transform IT - CIO East Africa - December 28th, 2019
- Over 750,000 Applications for Copies of US Birth Certificates Left Exposed Online - Security Boulevard - December 14th, 2019
- For Cloud-native App Security, Few Companies Have Embraced DevSecOps - Security Boulevard - December 14th, 2019
- Amping Up The Arm Server Roadmap - The Next Platform - December 14th, 2019
- Over 750,000 Applications for US Birth Certificates Left Exposed Online - Security Boulevard - December 14th, 2019
- AWS outperforms rivals in test of cloud capabilities - TechRepublic - December 14th, 2019
- Hyperscale operators accounted for a third of all data center spending in first three quarters of 2019 - FierceTelecom - December 14th, 2019
- IoT news of the week for Dec. 13, 2019 - Stacey on IoT - December 14th, 2019
- Data Sovereignty: The imperative for action - DatacenterDynamics - December 14th, 2019
- Google Transfer Service makes moving your data to the cloud easier than ever - ITProPortal - December 14th, 2019
- Why Middle East startups are choosing the cloud - Khaleej Times - December 14th, 2019
- The 10 Hottest New Business And Enterprise Servers Of 2019 - CRN: Technology news for channel partners and solution providers - December 14th, 2019
- Pulseway Introduces the All New, Integrated Cloud Backup Solution - PR Web - December 14th, 2019
- Edge predictions for 2020: From SD-WAN and cloud interconnection to security - Small Business - December 14th, 2019
- Broadcom Launches Another Tomahawk Into The Datacenter - The Next Platform - December 14th, 2019
- Four Technologies That Are Changing The Way We Do Business - TechShout! - December 14th, 2019
- Meet the power players behind Microsoft's Azure cloud - Business Insider - December 8th, 2019
- Hyperscale Data Center Market - Global Outlook and Forecast 2019-2024: Adoption of Cloud-based Services and & Big Data Driving Hyperscale Data... - December 8th, 2019
- Inside Intel's billion-dollar transformation in the age of AI - Fast Company - December 8th, 2019
- Amazon's cloud business bombards the market with dozens of new features as it looks to preserve its lead - CNBC - December 8th, 2019
- What is Infrastructure-as-a-Service? Everything you need to know about IaaS - TechRadar - December 8th, 2019
- Report: Growing HCI Space Boosted by Cloud, AI - Virtualization Review - December 8th, 2019
- 20 VPS providers to shut down on Monday, giving customers two days to save their data - ZDNet - December 8th, 2019
- How AWS Plans to Speed Up the Cloud - Toolbox - December 8th, 2019
- Finally: AWS Gives Servers A Real Shot In The Arm - The Next Platform - December 8th, 2019
- 5 Top Features of the Cloud that can Benefit Your Small Business - Atlanta Small Business Network - December 8th, 2019
- Once-Reticent Utilities Accelerating Move to the Cloud - Transmission & Distribution World - December 8th, 2019
- Ampere preps an 80-core Arm processor for the cloud - Network World - December 8th, 2019
- Cisco Down Two Execs As One Reportedly Jumps To John Chambers-Backed Pensando - CRN: The Biggest Tech News For Partners And The IT Channel - December 8th, 2019
- Meet CIMON-2, a new and improved robotic AI astronaut - Astronomy Magazine - December 8th, 2019
- Qualcomms Snapdragon Tech Summit focuses on 5G AI chips and use cases - VentureBeat - December 8th, 2019
- Are All Russian Mobile Apps Really a Potential Counterintelligence Threat? This is What the Experts Say - Newsweek - December 8th, 2019
- Overcome these VMware Cloud on AWS migration challenges - TechTarget - December 8th, 2019
- Aptiv To Unveil Open Source Advanced Technology Architecture - Forbes - December 8th, 2019
- Bitcoin Ichimoku Cloud Analysis BTC Price Prediction When To Buy Bitcoin? - Nasdaq - December 8th, 2019
- FBI classifies FaceApp as counterintelligence threat, citing ties to Russian intelligence | TheHill - The Hill - December 2nd, 2019
- Tech co. with Glasgow office merges, rebrands - Delaware Business Times - December 2nd, 2019
- Gatsby Cloud and its Impact on the Headless CMS Space - CMSWire - December 2nd, 2019
- This Holiday Season, Give Your Data Center Manager the Gift of Clarity - Data Economy - December 2nd, 2019
- Unsecured Server with Private Data from Millions of SMS Messages Found Online - Security Boulevard - December 2nd, 2019
- Media Giant Meredith Corporation Selects Nasuni To Help Move Traditional On-Premises File Infrastructure To Amazon Web Services - PRNewswire - December 2nd, 2019
- Encapulsation means application migration is no longer the blocker to digital transformation - Business MattersBusiness Matters - December 2nd, 2019
- Cloud Server Market by 2023 Worldwide Growth Opportunities Recent Trends Forecast by Types and Application to 2023 - Space Market Research - December 2nd, 2019