Deep in the bowels of the Smithsonian National Museum of American History in Washington DC sits a large metal cabinet the size of a walk-in wardrobe. The cabinet houses a remarkable computer the front is covered in dials, switches and gauges and inside it is filled with potentiometers controlled by small electric motors. Behind one of the cabinet doors is a 20 x 20 array of light sensitive cells, a kind of artificial eye.
This is the Perceptron Mark I, a simplified electronic version of a biological neuron. It was designed by the American psychologist Frank Rosenblatt at Cornell University in the late 1950s who taught it to recognize simple shapes such as triangles.
Rosenblatts work is now widely recognized as the foundation of modern artificial intelligence but at the time it was controversial. Despite the original success, researchers were unable to build on this, not least because more complex pattern recognition required vastly more computational power than was available at the time. This insatiable appetite prevented further study of artificial neurons and the networks they create.
Todays deep learning machines also eat power, lots of it. And that raises an interesting question about how much they will need in future. Is this appetite sustainable as the goals of AI become more ambitious?
Today we get an answer thanks to the work of Neil Thompson at the Massachusetts Institute of Technology in Cambridge and several colleagues. This team has measured the improved performance of deep learning systems in recent years and show that how it depends on increases in computing power.
By extrapolating this trend, they say that future advances will soon become unfeasible. Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable, say Thompson and colleagues, echoing the problems that emerged for Rosenblatt in the 1960s.
The teams approach is relatively straightforward. They analyzed over 1000 papers on deep learning to understand how learning performance scales with computational power. The answer is that the correlation is clear and dramatic.
In 2009, for example, deep learning was too demanding for the computer processors of the time. The turning point seems to have been when deep learning was ported to GPUs, initially yielding a 5 15 speed-up, they say.
This provided the horsepower for a neural network called AlexNet, which famously triumphed in a 2012 image recognition challenge where it wiped out the opposition. The victory created huge and sustained interest in deep neural networks that continues to this day.
But while deep learning performance increased by 35x between 2012 and 2019, the computational power behind it increased by an order of magnitude each year. Indeed, Thompson and co say this and other evidence suggests the computational power for deep learning has increased 9 orders of magnitude faster than the performance.
So how much computational power will be required in future? Thompson and co say that error rate for image recognition is currently 11.5 percent using 10^14 gigaflops of computational power at a cost of millions of dollars (ie 10^6 dollars).
They say achieving an error rate of just 1 per cent will require 10^28 gigaflops. And extrapolating at the current rate, this will cost 10^20 dollars. By comparison, the total amount of money in the world right now is measured in trillions ie 10^12 dollars.
Whats more, the environmental cost of such a calculation will be enormous, an increase in the amount of carbon produced of 14 orders of magnitude. Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable, conclude Thompson and colleagues.
The future isnt entirely bleak, however. Thompson and cos extrapolations assume that future deep learning systems will use the same kinds of computers that are available today.
But various new approaches offer much more efficient computation. For example, in some tasks the human brain can outperform the best supercomputers while running on little more than a bowl of porridge. Neuromorphic computing attempts to copy this. And quantum computing promises orders of magnitude more computing power with relatively little increase in power consumption.
Another option is to abandon deep learning entirely and concentrate on other forms of machine learning that are less power hungry.
Of course, there is no guarantee that these new techniques and technologies will work. But if they dont, its hard to see how artificial intelligence will get much better than it is now.
Curiously, something like this happened after the Perceptron Mark I first appeared, a period that lasted for decades and is now known as the AI winter. The Smithsonian doesnt currently have it on display, but it is surely marks a lesson worth remembering.
Ref: arxiv.org/abs/2007.05558 : The Computational Limits of Deep Learning
Read the original here:
The Computational Limits of Deep Learning Are Closer Than You Think - Discover Magazine
- Major Quantum Computing Projects And Innovations Of 2020 - Analytics India Magazine - January 2nd, 2021
- A little better all the time in 2021 - Science Magazine - January 2nd, 2021
- Quantum Computing Entwined with AI is Driving the Impossible to Possible - Analytics Insight - January 2nd, 2021
- QubitTech shapes the future of quantum computing - IBTimes India - January 2nd, 2021
- The Biggest Science Stories of 2020 | Technically Speaking - Inside Tucson Business - January 2nd, 2021
- Tech trends to watch in 2021 - India Today - January 2nd, 2021
- Tech trends in 2021: How artificial intelligence and technology will reshape businesses - The Financial Express - January 2nd, 2021
- The silver lining of 2020 - SouthCoastToday.com - January 2nd, 2021
- Quantum Computing Technologies Market Size, Share, Application Analysis, Regional Outlook, Growth Trends, Key Players, Competitive Strategies and... - January 2nd, 2021
- Malaysia: Leveraging On Digitalisation Trends - The ASEAN Post - January 2nd, 2021
- 01 Communique to Present at the SNN Network Canada Virtual Event on Thursday January 7, 2021 - IT News Online - January 2nd, 2021
- Beam me up: long-distance quantum teleportation has happened for the first time ever - SYFY WIRE - December 27th, 2020
- Global Quantum Computing Market Predicted to Garner $667.3 Million by 2027, Growing at 30.0% CAGR from 2020 to 2027 - [193 pages] Informative Report... - December 27th, 2020
- Two Years into the Government's National Quantum Initiative - Nextgov - December 27th, 2020
- Bitcoin is quantum computing resistant regardless of rising fears among investors - FXStreet - December 27th, 2020
- Fidelity Investments leaps back to the future in an experiment to restore active management to its lofty perch, using technology that is still more... - December 27th, 2020
- World first Australian solution to protect sensitive data from Quantum threat - Industry Update Manufacturing Media - December 27th, 2020
- Rewind 2020: Business, politics, social and professional impact, and what lies ahead - YourStory - December 27th, 2020
- As China Leads Quantum Computing Race, U.S. Spies Plan for a World with Fewer Secrets - Newsweek - December 15th, 2020
- Chinese quantum computer may be the most powerful ever seen - Siliconrepublic.com - December 15th, 2020
- Anyon Systems to Deliver a Quantum Computer to the Canadian Department of National Defense - GlobeNewswire - December 15th, 2020
- Show Your Work: D-Wave Opens the Door to Performance Comparisons Between Quantum Computing Architectures - HPCwire - December 15th, 2020
- NWA funding for taking quantum technology to the public Bits&Chips - Bits&Chips - December 3rd, 2020
- 01 Communique to Present at the Benzinga Global Small Cap Conference on December 8 - IT News Online - December 3rd, 2020
- Quantum Computing Market : Analysis and In-depth Study on Size Trends, and Regional Forecast - Cheshire Media - December 3rd, 2020
- Quantum computer race intensifies as alternative technology gains steam - Nature.com - November 19th, 2020
- Quantum computing now is a bit like SQL was in the late 80s: Wild and wooly and full of promise - ZDNet - November 19th, 2020
- Construction begins for Duke University's new quantum computing center - WRAL Tech Wire - November 19th, 2020
- Is Now the Time to Start Protecting Government Data from Quantum Hacking? - Nextgov - November 19th, 2020
- CCNY & partners in quantum algorithm breakthrough | The City College of New York - The City College of New York News - November 19th, 2020
- Quantum Computing in Aerospace and Defense Market Forecast to 2028: How it is Going to Impact on Global Industry to Grow in Near Future - Eurowire - November 19th, 2020
- What's Next In AI, Chips And Masks - SemiEngineering - November 19th, 2020
- Physicists discover the 'Kings and Queens of Quantumness' - Livescience.com - November 19th, 2020
- Every Thing You Need to Know About Quantum Computers - Analytics Insight - October 23rd, 2020
- Quantum computing will impact the enterprise--we just don't know how - TechRepublic - October 23rd, 2020
- Quantum Computing and the Cryptography Conundrum - CXOToday.com - October 23rd, 2020
- IBM and Mastercard among partners of 11.1m Irish quantum project - Siliconrepublic.com - October 23rd, 2020
- Quantum Computing Market Research including Growth Factors, Types and Application by regions by 2026 - Eurowire - October 23rd, 2020
- University of Rhode Island names respected professor, researcher, computational scientist to lead research computing efforts - URI Today - October 23rd, 2020
- Quantum Computing in Aerospace and Defense Market Trends and Forecast to 2028 - TechnoWeekly - October 23rd, 2020
- New York needs to be reimagined with technology and job training - Crain's New York Business - October 23rd, 2020
- Global Smart Cities Market Analysis 2020-2025: AI, IoT, and 5G (AIoT5G) will be the Most Influential Technologies - 63%, 34%, and 52% Respectively -... - October 23rd, 2020
- Quantum Computing Market 2020 | Outlook, Growth By Top Companies, Regions, Types, Applications, Drivers, Trends & Forecasts by 2025 - PRnews... - October 17th, 2020
- What is an algorithm? How computers know what to do with data - The Conversation US - October 17th, 2020
- Put Employees at the Center of Your Post-Pandemic Digital Strategy - Harvard Business Review - October 17th, 2020
- Most Read articles - LED drivers, Foundry market, Arm staffing - Electronics Weekly - October 17th, 2020
- 4 Reasons Why Now Is the Best Time to Start With Quantum Computing - Medium - October 15th, 2020
- The Future of Computing: Hype, Hope, and Reality - CIOReview - October 15th, 2020
- Menlo Micro, a startup bringing semiconductor tech to the humble switch, is ready for its closeup - TechCrunch - October 15th, 2020
- Rare magnetism found in the world's strongest material - Live Science - October 15th, 2020
- Room-temperature superconductivity has been achieved for the first time - MIT Technology Review - October 15th, 2020
- Global quantum computing market is projected to register a healthy CAGR of 29.5% in the forecast period of 2019 to 2026. - re:Jerusalem - October 15th, 2020
- Bring On The Qubits: How The Quantum Computing Arms Race Affects Legal - Technology - United States - Mondaq News Alerts - September 30th, 2020
- Under the dragons thumb: Chinese heft in VPNs and Indias vulnerability in a quantum-computing era - Economic Times - September 30th, 2020
- New Study Reveals 81% of Fortune 1000 Decision-Makers Have a Quantum Computing Use-Case In Mind For The Next Three Years Quantum computing emerges as... - September 30th, 2020
- Quantum Computing Technologies Market Potential Growth, Size, Share, Demand and Analysis of Key Players Research Forecasts to 2027 - The Daily... - September 30th, 2020
- Quantum Computing in Aerospace and Defense Market Analysis, Trends, Opportunity, Size and Segment Forecasts to 2028 - Crypto Daily - September 30th, 2020
- Pentagon Is Clinging to Aging Technologies, House Panel Warns - The New York Times - September 30th, 2020
- The global silicon photonics market accounted for $520.0 million in 2019 and is expected to reach $3.07 billion by 2025 - PRNewswire - September 30th, 2020
- Are We Close To Realising A Quantum Computer? Yes And No, Quantum Style - Swarajya - September 14th, 2020
- Spin-Based Quantum Computing Breakthrough: Physicists Achieve Tunable Spin Wave Excitation - SciTechDaily - September 14th, 2020
- NSF and DOE to Advance Industries of the Future - ARC Viewpoints - September 14th, 2020
- Global Quantum Computing Market 2025 To Expect Maximum Benefit and Growth Potential During this COVID 19 Outbreak: D-Wave Systems, Google, IBM, Intel,... - September 14th, 2020
- Why quantum computing matters - Axios - August 26th, 2020
- The future of artificial intelligence and quantum computing - Military & Aerospace Electronics - August 26th, 2020
- BBVA Uncovers The Promise Of Quantum Computing For Banking And Financial Services - Forbes - August 26th, 2020
- Has the world's most powerful computer arrived? - The National - August 26th, 2020
- Giant atoms enable quantum processing and communication in one - MIT News - August 4th, 2020
- Computer Scientist Don Towsley Named to Team Developing the Quantum Internet - UMass News and Media Relations - August 4th, 2020
- COVID-19 Impact on Quantum Computing Market Research, Growth, Industry Analysis, Size and Share 2025 | IBM Corporation, Google - My Kids Health - August 4th, 2020
- IBM and the University of Tokyo Unveil the Quantum Innovation Initiative Consortium to Accelerate Japan's Quantum Research and Development Leadership... - August 2nd, 2020
- Insights & Outcomes: a new spin on quantum research, and the biology of sex - Yale News - August 2nd, 2020
- This simple explainer tackles the complexity of quantum computing - Boing Boing - July 29th, 2020
- UK firm reaches final stages of the NIST quest for quantum-proof encryption algorithms - www.computing.co.uk - July 29th, 2020
- Looking Back on The First-Ever Photo of Quantum Entanglement - ScienceAlert - July 29th, 2020
- Quantum reckoning: The day when computers will break cryptography - ITWeb - July 29th, 2020
- Ripple CTO: Quantum computers will be a threat to Bitcoin and XRP - Crypto News Flash - July 29th, 2020
- The 6 Biggest Technology Trends In Accounting And Finance - Forbes - July 29th, 2020
- Ripple Executive Says Quantum Computing Will Threaten Bitcoin, XRP and Crypto Markets Heres When - The Daily Hodl - July 25th, 2020
- D-Waves quantum computing cloud comes to India - The Hindu - July 25th, 2020