Deep in the bowels of the Smithsonian National Museum of American History in Washington DC sits a large metal cabinet the size of a walk-in wardrobe. The cabinet houses a remarkable computer the front is covered in dials, switches and gauges and inside it is filled with potentiometers controlled by small electric motors. Behind one of the cabinet doors is a 20 x 20 array of light sensitive cells, a kind of artificial eye.
This is the Perceptron Mark I, a simplified electronic version of a biological neuron. It was designed by the American psychologist Frank Rosenblatt at Cornell University in the late 1950s who taught it to recognize simple shapes such as triangles.
Rosenblatts work is now widely recognized as the foundation of modern artificial intelligence but at the time it was controversial. Despite the original success, researchers were unable to build on this, not least because more complex pattern recognition required vastly more computational power than was available at the time. This insatiable appetite prevented further study of artificial neurons and the networks they create.
Todays deep learning machines also eat power, lots of it. And that raises an interesting question about how much they will need in future. Is this appetite sustainable as the goals of AI become more ambitious?
Today we get an answer thanks to the work of Neil Thompson at the Massachusetts Institute of Technology in Cambridge and several colleagues. This team has measured the improved performance of deep learning systems in recent years and show that how it depends on increases in computing power.
By extrapolating this trend, they say that future advances will soon become unfeasible. Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable, say Thompson and colleagues, echoing the problems that emerged for Rosenblatt in the 1960s.
The teams approach is relatively straightforward. They analyzed over 1000 papers on deep learning to understand how learning performance scales with computational power. The answer is that the correlation is clear and dramatic.
In 2009, for example, deep learning was too demanding for the computer processors of the time. The turning point seems to have been when deep learning was ported to GPUs, initially yielding a 5 15 speed-up, they say.
This provided the horsepower for a neural network called AlexNet, which famously triumphed in a 2012 image recognition challenge where it wiped out the opposition. The victory created huge and sustained interest in deep neural networks that continues to this day.
But while deep learning performance increased by 35x between 2012 and 2019, the computational power behind it increased by an order of magnitude each year. Indeed, Thompson and co say this and other evidence suggests the computational power for deep learning has increased 9 orders of magnitude faster than the performance.
So how much computational power will be required in future? Thompson and co say that error rate for image recognition is currently 11.5 percent using 10^14 gigaflops of computational power at a cost of millions of dollars (ie 10^6 dollars).
They say achieving an error rate of just 1 per cent will require 10^28 gigaflops. And extrapolating at the current rate, this will cost 10^20 dollars. By comparison, the total amount of money in the world right now is measured in trillions ie 10^12 dollars.
Whats more, the environmental cost of such a calculation will be enormous, an increase in the amount of carbon produced of 14 orders of magnitude. Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable, conclude Thompson and colleagues.
The future isnt entirely bleak, however. Thompson and cos extrapolations assume that future deep learning systems will use the same kinds of computers that are available today.
But various new approaches offer much more efficient computation. For example, in some tasks the human brain can outperform the best supercomputers while running on little more than a bowl of porridge. Neuromorphic computing attempts to copy this. And quantum computing promises orders of magnitude more computing power with relatively little increase in power consumption.
Another option is to abandon deep learning entirely and concentrate on other forms of machine learning that are less power hungry.
Of course, there is no guarantee that these new techniques and technologies will work. But if they dont, its hard to see how artificial intelligence will get much better than it is now.
Curiously, something like this happened after the Perceptron Mark I first appeared, a period that lasted for decades and is now known as the AI winter. The Smithsonian doesnt currently have it on display, but it is surely marks a lesson worth remembering.
Ref: arxiv.org/abs/2007.05558 : The Computational Limits of Deep Learning
Read the original here:
The Computational Limits of Deep Learning Are Closer Than You Think - Discover Magazine
- Every Thing You Need to Know About Quantum Computers - Analytics Insight - October 23rd, 2020
- Quantum computing will impact the enterprise--we just don't know how - TechRepublic - October 23rd, 2020
- Quantum Computing and the Cryptography Conundrum - CXOToday.com - October 23rd, 2020
- IBM and Mastercard among partners of 11.1m Irish quantum project - Siliconrepublic.com - October 23rd, 2020
- Quantum Computing Market Research including Growth Factors, Types and Application by regions by 2026 - Eurowire - October 23rd, 2020
- University of Rhode Island names respected professor, researcher, computational scientist to lead research computing efforts - URI Today - October 23rd, 2020
- Quantum Computing in Aerospace and Defense Market Trends and Forecast to 2028 - TechnoWeekly - October 23rd, 2020
- New York needs to be reimagined with technology and job training - Crain's New York Business - October 23rd, 2020
- Global Smart Cities Market Analysis 2020-2025: AI, IoT, and 5G (AIoT5G) will be the Most Influential Technologies - 63%, 34%, and 52% Respectively -... - October 23rd, 2020
- Quantum Computing Market 2020 | Outlook, Growth By Top Companies, Regions, Types, Applications, Drivers, Trends & Forecasts by 2025 - PRnews... - October 17th, 2020
- What is an algorithm? How computers know what to do with data - The Conversation US - October 17th, 2020
- Put Employees at the Center of Your Post-Pandemic Digital Strategy - Harvard Business Review - October 17th, 2020
- Most Read articles - LED drivers, Foundry market, Arm staffing - Electronics Weekly - October 17th, 2020
- 4 Reasons Why Now Is the Best Time to Start With Quantum Computing - Medium - October 15th, 2020
- The Future of Computing: Hype, Hope, and Reality - CIOReview - October 15th, 2020
- Menlo Micro, a startup bringing semiconductor tech to the humble switch, is ready for its closeup - TechCrunch - October 15th, 2020
- Rare magnetism found in the world's strongest material - Live Science - October 15th, 2020
- Room-temperature superconductivity has been achieved for the first time - MIT Technology Review - October 15th, 2020
- Global quantum computing market is projected to register a healthy CAGR of 29.5% in the forecast period of 2019 to 2026. - re:Jerusalem - October 15th, 2020
- Bring On The Qubits: How The Quantum Computing Arms Race Affects Legal - Technology - United States - Mondaq News Alerts - September 30th, 2020
- Under the dragons thumb: Chinese heft in VPNs and Indias vulnerability in a quantum-computing era - Economic Times - September 30th, 2020
- New Study Reveals 81% of Fortune 1000 Decision-Makers Have a Quantum Computing Use-Case In Mind For The Next Three Years Quantum computing emerges as... - September 30th, 2020
- Quantum Computing Technologies Market Potential Growth, Size, Share, Demand and Analysis of Key Players Research Forecasts to 2027 - The Daily... - September 30th, 2020
- Quantum Computing in Aerospace and Defense Market Analysis, Trends, Opportunity, Size and Segment Forecasts to 2028 - Crypto Daily - September 30th, 2020
- Pentagon Is Clinging to Aging Technologies, House Panel Warns - The New York Times - September 30th, 2020
- The global silicon photonics market accounted for $520.0 million in 2019 and is expected to reach $3.07 billion by 2025 - PRNewswire - September 30th, 2020
- Are We Close To Realising A Quantum Computer? Yes And No, Quantum Style - Swarajya - September 14th, 2020
- Spin-Based Quantum Computing Breakthrough: Physicists Achieve Tunable Spin Wave Excitation - SciTechDaily - September 14th, 2020
- NSF and DOE to Advance Industries of the Future - ARC Viewpoints - September 14th, 2020
- Global Quantum Computing Market 2025 To Expect Maximum Benefit and Growth Potential During this COVID 19 Outbreak: D-Wave Systems, Google, IBM, Intel,... - September 14th, 2020
- Why quantum computing matters - Axios - August 26th, 2020
- The future of artificial intelligence and quantum computing - Military & Aerospace Electronics - August 26th, 2020
- BBVA Uncovers The Promise Of Quantum Computing For Banking And Financial Services - Forbes - August 26th, 2020
- Has the world's most powerful computer arrived? - The National - August 26th, 2020
- Giant atoms enable quantum processing and communication in one - MIT News - August 4th, 2020
- Computer Scientist Don Towsley Named to Team Developing the Quantum Internet - UMass News and Media Relations - August 4th, 2020
- COVID-19 Impact on Quantum Computing Market Research, Growth, Industry Analysis, Size and Share 2025 | IBM Corporation, Google - My Kids Health - August 4th, 2020
- IBM and the University of Tokyo Unveil the Quantum Innovation Initiative Consortium to Accelerate Japan's Quantum Research and Development Leadership... - August 2nd, 2020
- Insights & Outcomes: a new spin on quantum research, and the biology of sex - Yale News - August 2nd, 2020
- This simple explainer tackles the complexity of quantum computing - Boing Boing - July 29th, 2020
- UK firm reaches final stages of the NIST quest for quantum-proof encryption algorithms - www.computing.co.uk - July 29th, 2020
- Looking Back on The First-Ever Photo of Quantum Entanglement - ScienceAlert - July 29th, 2020
- Quantum reckoning: The day when computers will break cryptography - ITWeb - July 29th, 2020
- Ripple CTO: Quantum computers will be a threat to Bitcoin and XRP - Crypto News Flash - July 29th, 2020
- The 6 Biggest Technology Trends In Accounting And Finance - Forbes - July 29th, 2020
- Ripple Executive Says Quantum Computing Will Threaten Bitcoin, XRP and Crypto Markets Heres When - The Daily Hodl - July 25th, 2020
- D-Waves quantum computing cloud comes to India - The Hindu - July 25th, 2020
- Hear how three startups are approaching quantum computing differently at TC Disrupt 2020 - TechCrunch - July 25th, 2020
- The Hyperion-insideHPC Interviews: Dr. Michael Resch Talks about the Leap from von Neumann: 'I Tell My PhD Candidates: Go for Quantum' - insideHPC - July 25th, 2020
- China's newest technology stock exchange is thriving despite the pandemic - The Economist - July 25th, 2020
- Almost One-Third of Life Science Companies Set to Begin Quantum Computing Evaluation This Year - Lab Manager Magazine - July 17th, 2020
- Europe Quantum Computing Market 2020 | Scope of Current and Future Industry 2025 - Owned - July 17th, 2020
- Opinion |Dance of the synchronized quantum particles - Livemint - July 17th, 2020
- Quantum Software Market 2020: Potential Growth, Challenges, and Know the Companies List Could Potentially Benefit or Loose out From the Impact of... - July 17th, 2020
- Quantum Computing Market Brief Analysis and Application, Growth by 2026 - 3rd Watch News - July 17th, 2020
- Standard Chartered and Universities Space Research Association join forces on Quantum Computing - PRNewswire - July 13th, 2020
- The crypto-agility mandate, and how to get there - Help Net Security - July 13th, 2020
- Standard Chartered teams up with Universities Space Research Association on development of quantum computing apps - FinanceFeeds - July 13th, 2020
- How American Express is tapping the benefits of hybrid cloud - The Enterprisers Project - July 13th, 2020
- MIT's New Diamond-Based Quantum Chip Is the Largest Yet - Interesting Engineering - July 11th, 2020
- Chicago Quantum Exchange Welcomes Seven New Partners in Tech, Computing and Finance - HPCwire - July 11th, 2020
- In 1st Of Its Kind Webinar On Quantum Information And Computation In India, IIIT Hyderabad Successfully Conducts Quantum Talks 2020 - IndianWeb2.com - July 11th, 2020
- Satoshi Nakamoto Inspiration Gives Advice On Bitcoins Next Move - Forbes - July 11th, 2020
- QCI Hosts Webinar Series Featuring Optimizations that Deliver Quantum-Ready Solutions at Breakthrough Speed - Stockhouse - July 11th, 2020
- Quantum Computing Technologies Market to Witness a Pronounce Growth During 2025 - News by aeresearch - July 11th, 2020
- Topological Quantum Computing Market Growth By Manufacturers, Type And Application, Forecast To 2026 - 3rd Watch News - July 6th, 2020
- Quantum Software Market (impact of COVID-19) Growth, Overview with Detailed Analysis 2020-2026| Origin Quantum Computing Technology, D Wave, IBM,... - July 6th, 2020
- Regional Analysis and Strategies of Quantum Computing Technology Market during the Forecasted Period 2020-2030 - 3rd Watch News - July 6th, 2020
- Healthcare Shopping: The new age of consumerism - The Financial Express - July 6th, 2020
- Six things you need to learn about quantum computing in finance - eFinancialCareers - July 4th, 2020
- Cybersecurity in the quantum era - ETCIO.com - July 4th, 2020
- There's a Hidden Economic Trendline That Is Shattering the Global Trade System - IDN InDepthNews | Analysis That Matters - July 4th, 2020
- How Will The World Look Like In 2025 And The Future Of Cybersecurity - Entrepreneur - July 4th, 2020
- Better encryption for wireless privacy at the dawn of quantum computing - UC Riverside - June 30th, 2020
- Menten AIs combination of buzzword bingo brings AI and quantum computing to drug discovery - TechCrunch - June 30th, 2020
- Paper Outlines the Role of ERM in Managing Risks Related to New Technologies - Business Wire - June 30th, 2020
- Airbus CTO Grazia Vittadini: Aviation needs to tap emerging technologies, diverse talent to get climate-neutral - Verdict Medical Devices - Medical... - June 30th, 2020
- Is IT regulation in the DARQ? - IT PRO - June 30th, 2020
- Sen. Warner: 5G ORAN Bill Added to Must-Pass Legislation - Multichannel News - June 30th, 2020
- Is teleportation possible? Yes, in the quantum world - University of Rochester - June 25th, 2020