Machine learning had a rich history long before deep learning reached fever pitch. Researchers and vendors were using machine learning algorithms to develop a variety of models for improving statistics, recognizing speech, predicting risk and other applications.
While many of the machine learning algorithms developed over the decades are still in use today, deep learning -- a form of machine learning based on multilayered neural networks -- catalyzed a renewed interest in AI and inspired the development of better tools, processes and infrastructure for all types of machine learning.
Here, we trace the significance of deep learning in the evolution of machine learning, as interpreted by people active in the field today.
The story of machine learning starts in 1943 when neurophysiologist Warren McCulloch and mathematician Walter Pitts introduced a mathematical model of a neural network. The field gathered steam in 1956 at a summer conference on the campus of Dartmouth College. There, 10 researchers came together for six weeks to lay the ground for a new field that involved neural networks, automata theory and symbolic reasoning.
The distinguished group, many of whom would go on to make seminal contributions to this new field, gave it the name artificial intelligence to distinguish it from cybernetics, a competing area of research focused on control systems. In some ways these two fields are now starting to converge with the growth of IoT, but that is a topic for another day.
Early neural networks were not particularly useful -- nor deep. Perceptrons, the single-layered neural networks in use then, could only learn linearly separable patterns. Interest in them waned after Marvin Minsky and Seymour Papert published the book Perceptrons in 1969, highlighting the limitations of existing neural network algorithms and causing the emphasis in AI research to shift.
"There was a massive focus on symbolic systems through the '70s, perhaps because of the idea that perceptrons were limited in what they could learn," said Sanmay Das, associate professor of computer science and engineering at Washington University in St. Louis and chair of the Association for Computing Machinery's special interest group on AI.
The 1973 publication of Pattern Classification and Scene Analysis by Richard Duda and Peter Hart introduced other types of machine learning algorithms, reinforcing the shift away from neural nets. A decade later, Machine Learning: An Artificial Intelligence Approach by Ryszard S. Michalski, Jaime G. Carbonell and Tom M. Mitchell further defined machine learning as a domain driven largely by the symbolic approach.
"That catalyzed a whole field of more symbolic approaches to [machine learning] that helped frame the field. This led to many Ph.D. theses, new journals in machine learning, a new academic conference, and even helped to create new laboratories like the NASA Ames AI Research branch, where I was deputy chief in the 1990s," said Monte Zweben, CEO of Splice Machine, a scale-out SQL platform.
In the 1990s, the evolution of machine learning made a turn. Driven by the rise of the internet and increase in the availability of usable data, the field began to shift from a knowledge-driven approach to a data-driven approach, paving the way for the machine learning models that we see today.
The turn toward data-driven machine learning in the 1990s was built on research done by Geoffrey Hinton at the University of Toronto in the mid-1980s. Hinton and his team demonstrated the ability to use backpropagation to build deeper neural networks.
"This was a major breakthrough enabling new kinds of pattern recognition that were previously not feasible with neural nets," Zweben said. This added new layers to the networks and a way to strengthen or weaken connections back across many layers in the network, leading to the term deep learning.
Although possible in a lab setting, deep learning did not immediately find its way into practical applications, and progress stalled.
"Through the '90s and '00s, a joke used to be that 'neural networks are the second-best learning algorithm for any problem,'" Washington University's Das said.
Meanwhile, commercial interest in AI was starting to wane because the hype around developing an AI on par with human intelligence had gotten ahead of results, leading to an AI winter, which lasted through the 1980s. What did gain momentum was a type of machine learning using kernel methods and decision trees that enabled practical commercial applications.
Still, the field of deep learning was not completely in retreat. In addition to the ascendancy of the internet and increase in available data, another factor proved to be an accelerant for neural nets, according to Zweben: namely, distributed computing.
Machine learning requires a lot of compute. In the early days, researchers had to keep their problems small or gain access to expensive supercomputers, Zweben said. The democratization of distributed computing in the early 2000s enabled researchers to run calculations across clusters of relatively low-cost commodity computers.
"Now, it is relatively cheap and easy to experiment with hundreds of models to find the best combination of data features, parameters and algorithms," Zweben said. The industry is pushing this democratization even further with practices and associated tools for machine learning operations that bring DevOps principles to machine learning deployment, he added.
Machine learning is also only as good as the data it is trained on, and if data sets are small, it is harder for the models to infer patterns. As the data created by mobile, social media, IoT and digital customer interactions grew, it provided the training material deep learning techniques needed to mature.
By 2012, deep learning attained star status after Hinton's team won ImageNet, a popular data science challenge, for their work on classifying images using neural networks. Things really accelerated after Google subsequently demonstrated an approach to scaling up deep learning across clusters of distributed computers.
"The last decade has been the decade of neural networks, largely because of the confluence of the data and computational power necessary for good training and the adaptation of algorithms and architectures necessary to make things work," Das said.
Even when deep neural networks are not used directly, they indirectly drove -- and continue to drive -- fundamental changes in the field of machine learning, including the following:
Deep learning's predictive power has inspired data scientists to think about different ways of framing problems that come up in other types of machine learning.
"There are many problems that we didn't think of as prediction problems that people have reformulated as prediction problems -- language, vision, etc. -- and many of the gains in those tasks have been possible because of this reformulation," said Nicholas Mattei, assistant professor of computer science at Tulane University and vice chair of the Association for Computing Machinery's special interest group on AI.
In language processing, for example, a lot of the focus has moved toward predicting what comes next in the text. In computer vision as well, many problems have been reformulated so that, instead of trying to understand geometry, the algorithms are predicting labels of different parts of an image.
The power of big data and deep learning is changing how models are built. Human analysis and insights are being replaced by raw compute power.
"Now, it seems that a lot of the time we have substituted big databases, lots of GPUs, and lots and lots of machine time to replace the deep problem introspection needed to craft features for more classic machine learning methods, such as SVM [support vector machine] and Bayes," Mattei said, referring to the Bayesian networks used for modeling the probabilities between observations and outcomes.
The art of crafting a machine learning problem has been taken over by advanced algorithms and the millions of hours of CPU time baked into pretrained models so data scientists can focus on other projects or spend more time on customizing models.
Deep learning is also helping data scientists solve problems with smaller data sets and to solve problems in cases where the data has not been labeled.
"One of the most relevant developments in recent times has been the improved use of data, whether in the form of self-supervised learning, improved data augmentation, generalization of pretraining tasks or contrastive learning," said Juan Jos Lpez Murphy, AI and big data tech director lead at Globant, an IT consultancy.
These techniques reduce the need for manually tagged and processed data. This is enabling researchers to build large models that can capture complex relationships representing the nature of the data and not just the relationships representing the task at hand. Lpez Murphy is starting to see transfer learning being adopted as a baseline approach, where researchers can start with a pretrained model that only requires a small amount of customization to provide good performance on many common tasks.
There are specific fields where deep learning provides a lot of value, in image, speech and natural language processing, for example, as well as time series forecasting.
"The broader field of machine learning is enhanced by deep learning and its ability to bring context to intelligence. Deep learning also improves [machine learning's] ability to learn nonlinear relationships and manage dimensionality with systems like autoencoders," said Luke Taylor, founder and COO at TrafficGuard, an ad fraud protection service.
For example, deep learning can find more efficient ways to auto encode the raw text of characters and words into vectors representing the similarity and differences of words, which can improve the efficiency of the machine learning algorithms used to process it. Deep learning algorithms that can recognize people in pictures make it easier to use other algorithms that find associations between people.
More recently, there have been significant jumps using deep learning to improve the use of image, text and speech processing through common interfaces. People are accustomed to speaking to virtual assistants on their smartphones and using facial recognition to unlock devices and identify friends in social media.
"This broader adoption creates more data, enables more machine learning refinement and increases the utility of machine learning even further, pushing even further adoption of this tech into people's lives," Taylor said.
Early machine learning research required expensive software licenses. But deep learning pioneers began open sourcing some of the most powerful tools, which has set a precedent for all types of machine learning.
"Earlier, machine learning algorithms were bundled and sold under a licensed tool. But, nowadays, open source libraries are available for any type of AI applications, which makes the learning curve easy," said Sachin Vyas, vice president of data, AI and automation products at LTI, an IT consultancy.
Another factor in democratizing access to machine learning tools has been the rise of Python.
"The wave of open source frameworks for deep learning cemented the prevalence of Python and its data ecosystem for research, development and even production," Globant's Lpez Murphy said.
Many of the different commercial and free options got replaced, integrated or connected to a Python layer for widespread use. As a result, Python has become the de facto lingua franca for machine learning development.
Deep learning has also inspired the open source community to automate and simplify other aspects of the machine learning development lifecycle. "Thanks to things like graphical user interfaces and [automated machine learning], creating working machine learning models is no longer limited to Ph.D. data scientists," Carmen Fontana, IEEE member and cloud and emerging tech practice lead at Centric Consulting, said.
For machine learning to keep evolving, enterprises will need to find a balance between developing better applications and respecting privacy.
Data scientists will need to be more proactive in understanding where their data comes from and the biases that may inadvertently be baked into it, as well as develop algorithms that are transparent and interpretable. They also need to keep pace with new machine learning protocols and the different ways these can be woven together with various data sources to improve applications and decisions.
"Machine learning provides more innovative applications for end users, but unless we're choosing the right data sets and advancing deep learning protocols, machine learning will never make the transition from computing a few results to providing actual intelligence," said Justin Richie, director of data science at Nerdery, an IT consultancy.
"It will be interesting to see how this plays out in different industries and if this progress will continue even as data privacy becomes more stringent," Richie said.
- Machine Learning Answers: Facebook Stock Is Down 20% In A Month, What Are The Chances It'll Rebound? - Trefis - September 22nd, 2020
- Machine Learning in Education Market Incredible Possibilities, Growth Analysis and Forecast To 2025 - The Daily Chronicle - September 22nd, 2020
- Proximity matters: Using machine learning and geospatial analytics to reduce COVID-19 exposure risk - Healthcare IT News - September 22nd, 2020
- Global Machine Learning Market Tends To Show Steady Growth Post Pandemic With Regional Overview and Top Key Players - Verdant News - September 22nd, 2020
- PREDICTING THE OPTIMUM PATH - Port Strategy - September 22nd, 2020
- AI/ML Remains The Most In-Demand Tech Skill Post COVID - Analytics India Magazine - September 22nd, 2020
- Panalgo Brings the Power of Machine-Learning to the Healthcare Industry Via Its IHD Software - AiThority - September 15th, 2020
- Microchip Partners with Machine-Learning (ML) Software Leaders to Simplify AI-at-the-Edge Design Using its 32-Bit Microcontrollers (MCUs) - EE Journal - September 15th, 2020
- What is 'custom machine learning' and why is it important for programmatic optimisation? - The Drum - September 15th, 2020
- PODCAST: NVIDIA's Director of Data Science Talks Machine Learning for Airlines and Aerospace - Aviation Today - September 15th, 2020
- The Use of Machine Learning to Forecast Progression to Advanced AMD - DocWire News - September 15th, 2020
- How Can Machine Learning Help the Teaching Profession? - FE News - September 15th, 2020
- Global Machine Learning in Automobile Market: Development History, Current Analysis and Estimated Forecast to 2024 - The Market Correspondent - September 15th, 2020
- Using machine learning to organize the chemical diversity - Tech Explorist - September 15th, 2020
- Dashboard AI Announces Its Technology Vision for the Foodservice and Hospitality Industry - PRNewswire - September 15th, 2020
- Alfa Releases Second Paper on AI, Using Machine Learning in the Wild - Monitor Daily - September 10th, 2020
- Combatting COVID-19 misinformation with machine learning (VB Live) - VentureBeat - September 10th, 2020
- This artist used machine learning to create realistic portraits of Roman emperors - The World - September 10th, 2020
- Domino Data Lab Named a Leader in Notebook-Based Predictive Analytics and Machine Learning Evaluation by Global Research Firm - Business Wire - September 10th, 2020
- Demonstration Of What-If Tool For Machine Learning Model Investigation - Analytics India Magazine - September 10th, 2020
- RXA to Participate in 2nd Annual A2.AI Conference focused on Machine Learning & Applied AI - PR Web - September 10th, 2020
- 50 Data Science and Analysts Jobs That Opened Just Last Week - Analytics India Magazine - September 10th, 2020
- FSS Launches Next Gen Recon with Machine Learning and Cloud Support - TechGenyz - September 10th, 2020
- Getting to the heart of machine learning and complex humans - The Irish Times - August 28th, 2020
- Global Machine Learning Courses Market Trends, Key Driven Factors, Segmentation And Forecast To 2020-2026 - The Scarlet - August 28th, 2020
- AI and Machine Learning Network Fetch.ai Partners Open-Source Blockchain Protocol Waves to Conduct R&D on DLT - Crowdfund Insider - August 28th, 2020
- UT Austin Selected as Home of National AI Institute Focused on Machine Learning - UT News | The University of Texas at Austin - August 26th, 2020
- Participation-washing could be the next dangerous fad in machine learning - MIT Technology Review - August 26th, 2020
- The Role of Artificial Intelligence and Machine Learning in the... - Insurance CIO Outlook - August 26th, 2020
- Machine Learning Artificial intelligence Market Size and Growth By Leading Vendors, By Types and Application, By End Users and Forecast to 2020-2027 -... - August 26th, 2020
- Air Force Taps Machine Learning to Speed Up Flight Certifications - Nextgov - August 26th, 2020
- What is AutoML and Why Should Your Business Consider It - BizTech Magazine - August 26th, 2020
- Chatbots Are Machine Learning Their Way To Human Language - Forbes - August 26th, 2020
- Explainable AI: From the peak of inflated expectations to the pitfalls of interpreting machine learning models - ZDNet - August 26th, 2020
- Focusing on ethical AI in business and government - FierceElectronics - August 26th, 2020
- Amazon's Machine Learning University To Make Its Online Courses Available To The Public - Analytics India Magazine - August 14th, 2020
- Watch 3 Videos from Coursera's New "Machine Learning for Everyone" - Machine Learning Times - machine learning & data science news - The... - August 14th, 2020
- PhD Research Fellowship in Machine Learning for Cognitive Power Management job with NORWEGIAN UNIVERSITY OF SCIENCE & TECHNOLOGY - NTNU | 219138 -... - August 14th, 2020
- Machine learning is pivotal to every line of business, every organisation must have an ML strategy - BusinessLine - August 14th, 2020
- CORRECTING and REPLACING Anyscale Hosts Inaugural Ray Summit on Scalable Python and Scalable Machine Learning - Yahoo Finance - August 14th, 2020
- Why GPT-3 Heralds a Democratic Revolution in Tech - Built In - August 14th, 2020
- BMW, Red Hat, and Malong Share Insights on AI and Machine Learning During Transform 2020 - ENGINEERING.com - August 14th, 2020
- Algorithm created by deep learning finds potential therapeutic targets throughout the human genome - National Science Foundation - August 14th, 2020
- Ensighten Launches Client-Side Threat Intelligence Initiative and Invests in Machine Learning - WFMZ Allentown - August 6th, 2020
- Hey software developers, youre approaching machine learning the wrong way - The Next Web - August 6th, 2020
- Introducing The AI & Machine Learning Imperative - MIT Sloan - August 6th, 2020
- Who Does the Machine Learning and Data Science Work? - Customer Think - August 6th, 2020
- Artificial Intelligence and Machine Learning Path to Intelligent Automation - Embedded Computing Design - August 6th, 2020
- Blacklight Solutions Unveils Software to Simplify Business Analytics with AI and Machine Learning - PRNewswire - August 6th, 2020
- AI is learning when it should and shouldnt defer to a human - MIT Technology Review - August 6th, 2020
- Moderna Announced Partnership With Amazon Web Services for Their Analytics and Machine Learning Services - Science Times - August 6th, 2020
- Surprisingly Recent Galaxy Discovered Using Machine Learning May Be the Last Generation Galaxy in the Long Cosmic History - SciTechDaily - August 6th, 2020
- STMicroelectronics Releases STM32 Condition-Monitoring Function Pack Leveraging Tools from Cartesiam for Simplified Machine Learning - ELE Times - August 6th, 2020
- Machine Learning Reveals What Makes People Happy In A Relationship - Forbes - August 4th, 2020
- Benefits Of AI And Machine Learning | Expert Panel | Security News - SecurityInformed - August 4th, 2020
- Preparing new machine learning models used to take weeks Activeloop teams up with NVIDIA to reduce that time to hours - MENAFN.COM - August 4th, 2020
- IoT automation trend rides the next wave of machine learning, Big Data - Urgent Communications - August 4th, 2020
- Decoding Practical Problems and Business Implications of Machine Learning - Analytics Insight - August 4th, 2020
- Artificial Intelligence and Machine Learning Industry 2020 Market Manufacturers Analysis, Share, Size, Growth, Trends and Research Report 2026 -... - August 4th, 2020
- Could this software help users trust machine learning decisions? - C4ISRNet - July 27th, 2020
- Top Five Data Privacy Issues that Artificial Intelligence and Machine Learning Startups Need to Know - insideBIGDATA - July 27th, 2020
- COVID-19 Impacts: Machine Learning Market will Accelerate at a CAGR of about 39% through 2020-2024 | The Increasing Adoption of Cloud-based Offerings... - July 27th, 2020
- 2 books to deepen your command of python machine learning - TechTalks - July 1st, 2020
- What I Learned From Looking at 200 Machine Learning Tools - Machine Learning Times - machine learning & data science news - The Predictive... - July 1st, 2020
- Protecting inventions which use Machine Learning and Artificial Intelligence - Lexology - July 1st, 2020
- Machine learning finds use in creating sharper maps of 'ecosystem' lines in the ocean - Firstpost - July 1st, 2020
- Fake data is great data when it comes to machine learning - Stacey on IoT - July 1st, 2020
- Decisions and NLP Logix Announce Partnership to bring the Power of Machine Learning to Business Process Management - Benzinga - July 1st, 2020
- Machine Learning in Medical Imaging Market Strategies and Insight Driven Transformation 2020-2030 - Cole of Duty - July 1st, 2020
- Impact of COVID-19 Outbreak on Artificial Intelligence and Machine Learning Market to Witness AIBrain, Amazon, Anki, CloudMinds - Cole of Duty - July 1st, 2020
- Machine Learning Market Projected to Register 43.5% CAGR to 2030 Intel, H2Oai - 3rd Watch News - July 1st, 2020
- Learn the business value of AI's various techniques - TechTarget - July 1st, 2020
- Machine Learning As A Service In Manufacturing Market Augmented Expansion to Be Registered by 2018-2023 - 3rd Watch News - July 1st, 2020
- COVID 19 Impact on Machine Learning in Medicine Market Outlook 2020 Industry Size, Top Key Manufacturers, Growth Insights, Demand Analysis and... - July 1st, 2020
- Machine learning algorithm from RaySearch enhances workflow at Swedish radiation therapy clinic - DOTmed HealthCare Business News - July 1st, 2020
- What a machine learning tool that turns Obama white can (and cant) tell us about AI bias - The Verge - June 25th, 2020
- AI and Machine Learning Are Changing Everything. Here's How You Can Get In On The Fun - ExtremeTech - June 25th, 2020
- SLAM + Machine Learning Ushers in the "Age of Perception - Robotics Business Review - June 25th, 2020
- Googles new ML Kit SDK keeps all machine learning on the device - SlashGear - June 25th, 2020
- Machine Learning vs Predictive Analytics: Are they same? - Analytics Insight - June 25th, 2020