Arguably one of the premiere events that has brought AI to popular attention in recent years was the invention of the Transformer by Ashish Vaswani and colleagues at Google in 2017. The Transformer led to lots of language programs such as Google's BERT and OpenAI's GPT-3 that have been able to produce surprisingly human-seeming sentences, giving the impression machines can write like a person.
Now, scientists at DeepMind in the U.K., which is owned by Google, want to take the benefits of the Transformer beyond text, to let it revolutionize other material including images, sounds and video, and spatial data of the kind a car records with LiDAR.
The Perceiver, unveiled this week by DeepMind in a paper posted on arXiv, adapts the Transformer with some tweaks to let it consume all those types of input, and to perform on the various tasks, such as image recognition, for which separate kinds of neural networks are usually developed.
The DeepMind work appears to be a waystation on the way to an envisioned super-model of deep learning, a neural network that could perform a plethora of tasks, and would learn faster and with less data, what Google's head of AI, Jeff Dean, has described as a "grand challenge" for the discipline.
One model to rule them all? DeepMind's Perceiver has decent performance on multiple tests of proficiency even though the program is not built for any one kind of input, unlike most neural networks that specialize. Perceiver combines a now-standard Transformer neural network with a trick called "inducing points," as a summary of the data, to reduce how much raw data from pixels or audio or video needs to be computed.
The paper, Perceiver: General Perception with Iterative Attention, by authors Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira, is to be presented this month at the International Conference on Machine Learning, which kicks of July 18th and which is being held as a virtual event this year.
Perceiver continues the trend to generality that has been underway for several years now, meaning, having less and less built into an AI program that is specific to a task. Before Vaswani et al.'s Transformer, most natural language programs were constructed with a sense of the particular language function, such as question answering or language translation. Transformer erased those distinctions, producing one program that could handle a multitude of tasks by creating a sufficiently adept representation of language.
Also: AI in sixty seconds
Likewise, Perceiver challenges the idea that different kinds of data, such as sound or image, need different neural network architectures.
However, something more profound is indicated by Perceiver. Last year, at the International Solid State Circuits Conference, an annual technical symposium held in San Francisco, Google's Dean described in his keynote address one future direction of deep learning as the "goal of being able to train a model that can perform thousands or millions of tasks in a single model."
"Building a single machine learning system that can handle millions of tasks is a true grand challenge in the field of artificial intelligence and computer systems engineering," said Dean.
In a conversation with ZDNet at the conference, Dean explained how a kind of super-model would build up from work over the years on neural networks that combine "modalities," different sorts of input such as text and image, and combinations of models known as "mixture of experts":
Mixture of experts-style approaches, I think, are going to be important, and multi-task, and multi-modal approaches, where you sort-of learn representations that are useful for many different things, and sort-of jointly learn good representations that help you be able to solve new tasks more quickly, and with less data, fewer examples of your task, because you are already leveraging all the things you already know about the world.
Perceiver is in the spirit of that multi-tasking approach. It takes in three kinds of inputs: images, videos, and what are called point clouds, a collection of dots that describes what a LiDAR sensor on top of a car "sees" of the road.
Once the system is trained, it can perform with some meaningful results on benchmark tests, including the classic ImageNet test of image recognition; Audio Set, a test developed at Google that requires a neural net to pick out kinds of audio clips from a video; and ModelNet, a test developed in 2015 at Princeton whereby a neural net must use 2,000 points in space to correctly identify an object.
Also: Google experiments with AI to design its in-house computer chips
Perceiver manages to achieve the task using two tricks, or, maybe, one trick and one cheat.
The first trick is to reduce the amount of data that the Transformer needs to operate on directly. While large Transformer neural networks have been fed gigabytes and gigabytes of text data, the amount of data in images or video or audio files, or point clouds, is potentially vastly larger. Just think of every pixel in a 244 by 244 pixel image from ImageNet. In the case of a sound file, "1 second of audio at standard sampling rates corresponds to around 50,000 raw audio samples," write Jaegle and team.
So, Jaegle and team went in search of a way to reduce the so-called "dimensionality" of those data types. They borrow from the work of Juho Lee and colleagues at Oxford University, who introduced what they called the Set Transformer. The Set Transformer reduced the computing needed for a Transformer by creating a second version of each data sample, a kind of summary, which they called inducing points. Think of it as data compression.
Jaegle and team adapt this as what they call a "learned latent array," whereby the sample data is boiled down to a summary that is far less data-hungry. The Perceiver acts in an "asymmetric" fashion: Some of its abilities are spent examining the actual data, but some only look at the summary, the compressed version. This reduces the overall time spent.
The second trick, really kind of a cheat, is to give the model some clues about the structure of the data. The problem with a Transformer is that it knows nothing about the spatial elements of an image, or the time value of an audio clip. A Transformer is always what's called permutation invariant, meaning, insensitive to these details of the structure of the particular kind of data.
That is a potential problem baked into the generality of the Perceiver. Neural networks built for images, for example, have some sense of the structure of a 2-D image. A classic convolutional neural network processes pixels as groups in a section of the image, known as locality. Transformers, and derivatives such as Perceiver, aren't built that way.
The authors, surprisingly, cite the 18th-century German philosopher Immanuel Kant, who said that such structural understanding is crucial.
"Spatial relationships are essential for sensory reasoning," Jaegle and team write, citing Kant, "and this limitation is clearly unsatisfying."
So, the authors, in order to give some sense of the structure of images or sound back to the neural network, borrow a technique employed by Google's Matthew Tancik and colleagues last year, what are called Fourier features. Fourier Features explicitly tag each piece of input with some meaningful information about structure.
For example, the coordinates of a pixel in an image can be "mapped" to an array, so that locality of data is preserved. The Perceiver then takes into account that tag, that structural information, during its training phase.
As Jaegle and team describe it,
We can compensate for the lack of explicit structures in our architecture by associating position and modality-specific features with every input element (e.g. every pixel, or each audio sample) these can be learned or constructed using high-fidelity Fourier features. This is a way of tagging input units with a high-fidelity representation of position and modal- ity, similar to the labeled lined strategy used to construct topographic and cross-sensory maps in biological neural networks by associating the activity of a specific unit with a semantic or spatial location.
The results of the benchmark tests are intriguing. Perceiver is better than the industry standard ResNet-50 neural network on ImageNet, in terms of accuracy, and better than a Transformer that has been adapted to images, the Vision Transformer introduced this year by Alexey Dosovitskiy and colleagues at Google.
On the Audio Set test, the Perceiver blows away most but not all state-of-the-art models for accuracy. And on the ModelNet test of point clouds, the Perceiver also gets quite high marks.
Jaegle and team claim for their program a kind of uber-proficiency that wins by being best all around: "When comparing these models across all different modalities and combinations considered in the paper, the Perceiver does best overall."
There are a number of outstanding issues with Perceiver that make it perhaps not actually the ideal million-task super-model that Dean has described. One is that the program doesn't always do as well as programs made for a particular modality. It still fails against some specific models. For example, on Audio Set, the Perceiver fell short of a program introduced last year by Haytham M. Fayek and Anurag Kumar of Facebook that "fuses" information about audio and video.
On the point cloud, it falls far short of a 2017 neural network built just for point clouds, PointNet++, by Charles Qi and colleagues at Stanford.
And on ImageNet, clearly the Perceiver was helped by the cheat of having Fourier features that tag the structure of images. When the authors tried a version of the Perceiver with the Fourier features removed, called "learned position," the Perceiver didn't do nearly as well as ResNet-50 and ViT.
A second issue is that nothing about Perceiver appears to bring the benefits of more-efficient computing and less data that Dean alluded to. In fact, the authors note that the data they use isn't always big enough. They observe that sometimes, the Perceiver may not be successfully generalizing, quipping that "With great flexibility comes great overfitting." Overfitting is when a neural network is so much bigger than its training data set, that it is able to simply memorize the data rather than achieve important representations that generalize the data.
Hence, "In future work, we would like to pre-train our image classification model on very large scale data," they write.
That leads to a larger question about just what is going on in what the Perceiver has "learned." If Google's Jeff Dean is right, then something like Perceiver should be learning representations that are mutually reinforcing. Clearly, the fact of a general model being able to perform well in spite of its generality suggests that something of the kind is going on. But what?
All we know is that Perceiver can learn different kinds of representations. The authors show a number of what are called attention maps, visual studies that purport to represent what the Perceiver is emphasizing in each clump of training data. Those attention maps suggest the Perceiver is adapting where it places the focus of computing.
As Jaegle and team write, "it can adapt its attention to the input content."
An attention map purports to show what the Perceiver is emphasizing in its video inputs, showing it is learning new represenations specific to the "modality" of the data.
A third weakness is specifically highlighted by the authors, and that is the question of the Fourier features, the cheat. The cheat seems to help in some cases, and it's not clear how or even if that crutch can be dispensed with.
As the authors put it, "End-to-end modality-agnostic learning remains an interesting research direction."
On a philosophical note, it's interesting to wonder if Perceiver will lead to new kinds of abilities that are specifically multi-modal. Perceiver doesn't show any apparent synergy between the different modalities, so that image and sound and point clouds still exist apart from one another. That's probably mostly to do with the tasks. All the tasks used in the evaluation have been designed for single neural networks.
Clearly, Google needs a new benchmark to test multi-modality.
For all those limitations, it's important to realize that Perceiver may be merely a stage on the way to what Dean described. As Dean told ZDNet, an eventual super-model is a kind of evolutionary process:
The nice thing about that vision of being able to have a model that does a million tasks is there are good intermediate points along the way. You can say, well, we're not going to bite off multi-modal, instead let's try to just do a hundred vision tasks in the same model first. And then a different instance of it where we try to do a hundred textual tasks, and not try to mix them together. And then say, that seems to be working well, let's try to combine the hundred vision and hundred textual tasks, and, hopefully, get them to improve each other, and start to experiment with the multi-modal aspects.
Also: Ethics of AI: Benefits and risks of artificial intelligence
- The Best Social Media Reactions and Memes at the 2021 Met Gala - Yahoo Lifestyle - September 15th, 2021
- Researchers lay the groundwork for an AI hive mind - The Next Web - September 15th, 2021
- Infinite Memory Transformer: Attending to Arbitrarily Long Contexts Without Increasing Computation Burden - Synced - September 15th, 2021
- It takes a village: how Asics got a small town to lace up and raise money for Mind UK - The Drum - September 15th, 2021
- EY study deep dives into financial protection gap - Insurance Business - September 15th, 2021
- Peptone: Putting molecular physics at the heart of UK's next billion-pound pharma company - Yahoo Money - September 15th, 2021
- Vigil episode four recap Silva and the crew are in deep water - The Guardian - September 13th, 2021
- How to cultivate a 'big mind' to deal with everyday stresses | HeraldNet.com - The Daily Herald - September 13th, 2021
- THE BOOK NOOK: Deep behind the scenes of 9/11 - Niagara Gazette - September 13th, 2021
- How 7 years of dirt is deep cleaned from mattresses - Insider - September 13th, 2021
- A therapy called rhythm that creates impact on the body and the mind - Onmanorama - September 13th, 2021
- Bidens abortion clash with the Catholic Church - POLITICO - September 13th, 2021
- God's mysterious riches, wisdom, and knowledge - Leawood - Church of the Resurrection - September 13th, 2021
- Google pulls the plug on DeepMind-developed health app Streams - Business Insider - September 1st, 2021
- Where the mind is without fear - The Times of India Blog - September 1st, 2021
- Global Cognitive Computing in Retail Market, By Component, By Technology, By Deployment, By Application, By Region, Competition Forecast &... - September 1st, 2021
- U.S. households and small businesses have stockpiled a mind-blowing record cash pile of almost $17 trillion - MarketWatch - August 26th, 2021
- What would it be like to be a conscious AI? We might never know. - MIT Technology Review - August 26th, 2021
- Google Health Disbanded; Staff Sent To Other Divisions - Silicon UK - August 26th, 2021
- Zen and the art of fly-fishing in Sweden a photo essay - The Guardian - August 26th, 2021
- 7 Ways to Stop Racing Thoughts at Night and Get to Sleep - Livestrong - August 26th, 2021
- Possumhaw: Out of sight, out of mind - The Dispatch - The Commercial Dispatch - August 24th, 2021
- UK aims to boost solar by predicting cloud movements with A.I. - CNBC - August 24th, 2021
- This Special Nature Writing Ritual Will Connect You To Your Divine Feminine - mindbodygreen.com - August 24th, 2021
- Shipwrecked: A Shocking Tale of Love, Loss, and Survival in the Deep Blue Sea - Boston magazine - August 24th, 2021
- AllDay notes after deeper reading of the prospectus - Philstar.com - August 24th, 2021
- UCSD & Microsoft Improve Image Recognition With Extremely Low FLOPs - Synced - August 24th, 2021
- Are These the Hidden Deepfakes in the Anthony Bourdain Movie? - WIRED - August 24th, 2021
- Scientists report new findings on the role that fish play in balancing coral, algae on reefs - ASU Now - August 24th, 2021
- Accurate prediction of protein structures and interactions using a three-track neural network - Science Magazine - August 22nd, 2021
- The Wonders That Live at the Very Bottom of the Sea - The New York Times - August 22nd, 2021
- Tamo J is 'digging up deep-rooted issues' in therapy - Jamaica Gleaner - August 22nd, 2021
- How to use 'progressive muscle relaxation' to calms stress and ease anxiety - GLAMOUR UK - August 22nd, 2021
- Review: 'Sister Sorry' plunges deep into a shadowy abyss of the psyche. But are its characters worth caring about? - Berkshire Eagle - August 22nd, 2021
- Always on your mind - The New Indian Express - August 22nd, 2021
- This Former Pastor Is Changing Evangelicals' Minds on COVID Vaccines Mother Jones - Mother Jones - August 22nd, 2021
- How DeepMind's John Jumper unlocked the 3D secret to proteins - Fast Company - August 14th, 2021
- How to avoid falling asleep during meditation - The Indian Express - August 14th, 2021
- A New Review of Dozens of Studies Found These Are the 8 Most Effective Ways to Stop Worrying and Calm Your Mind - Inc. - August 14th, 2021
- Four Policies that Government Can Pursue to Advance Trustworthy AI - uschamber.com - August 14th, 2021
- England vs India 2nd Test, Day 3 as it happened: Roots marathon ton guides hosts to 27-run lead - Scroll.in - August 14th, 2021
- Deshbhakti Curriculum to be implemented in Delhi Govt schools. Here is what it will cover - Mint - August 14th, 2021
- Trouble at Google's DeepMind and tech workers not going back to the office - Business Insider - August 8th, 2021
- Embodied AI, superintelligence and the master algorithm - TechCrunch - August 8th, 2021
- Alserkal is serving up isolation tanks and other mind altering activities this month - Time Out Dubai - August 8th, 2021
- Tshola had a generous heart and mind - IOL - August 8th, 2021
- Keeping the faith: Remembering gratitude can help us extend love to others, transform world - The Columbus Dispatch - August 8th, 2021
- AI use cases are expanding and evolving in healthcare - Tech Wire Asia - August 1st, 2021
- DeepMind releases massive database of 3D protein structures - STAT - STAT - July 25th, 2021
- DeepMinds AI uncovers structure of 98.5 per cent of human proteins - News Nation USA - July 25th, 2021
- Good Morning, News: $600000 Settlement for 2017 PPB Killing, Deep Pockets Try Influencing MultCo DA, and Everything is GREAT at the Olympics! - The... - July 25th, 2021
- DeepMind open-sources AlphaFold 2 for protein structure predictions - VentureBeat - July 21st, 2021
- DeepMind's AlphaFold 2 reveal: Convolutions are out, attention is in - ZDNet - July 21st, 2021
- AI in Healthcare Market Growing Trade Among Emerging Economies Opening New Opportunities (2021-2031) | Nuance Communications, Inc., DeepMind... - July 21st, 2021
- Self-driving data centres: Managing the transition from human-to-AI workload management - IT Brief Australia - July 21st, 2021
- The scene is set | Royal St George's | The 149th Open - The Open - July 14th, 2021
- Tech Companies Leading the Cognitive Computing Race in 2021 - Analytics Insight - July 14th, 2021
- Science fiction: From hero's tales to deep thoughts - The Daily Advance - July 14th, 2021
- ADOT urges drivers to be prepared for monsoon weather dangers | Arizona Emergency information Network - az.gov - July 14th, 2021
- Taking the long view, taking control - Cape Cod Times - July 14th, 2021
- The ultimate reading list if you love to delve into the mind - Livemint - July 14th, 2021
- Midland Theatre executive director Anderson to retire Sept. 1 - The Newark Advocate - July 14th, 2021
- The Future of Deep Learning | - Analytics Insight - July 10th, 2021
- The movie you should watch this weekend: Eternal Sunshine of the Spotless Mind, the heart wants what it wants - The Indian Express - July 10th, 2021
- From the Extension: Keep health and safety in mind when firing up the grill - Daily Commercial - July 10th, 2021
- Follow your fever dreams with Nashville Hot Hot Dogs - The Takeout - July 10th, 2021
- Former Bureaucrats, Civil Society Express Deep Anguish On Violations and Breakdown of Governance in UP - The Citizen - July 10th, 2021
- The 36 questions that lead to love (but with your co-founder) - TechCrunch - July 10th, 2021
- At the ICA Watershed, artist Firelei Bez draws history from the deep - The Boston Globe - July 8th, 2021
- The Streaming Age Has Turned Poland Into a Deep-Pocketed Production Paradise - Hollywood Reporter - July 8th, 2021
- What Toxic Positivity Actually Is (& Why You Should Avoid It) - mindbodygreen.com - July 8th, 2021
- UCF Lake Nona Scientific Heart Is The First Sanatorium In Central Florida To Be offering Incisionless Mind Surgical treatment As An Complex Remedy For... - July 8th, 2021
- Young Cho's goals are even bigger than deep space; His team is working on astronaut nutrition that translates to earth - Westside Seattle - July 8th, 2021
- Two books to make you think differently about the ocean and the beach - Christian Science Monitor - July 8th, 2021
- Jimmy McGovern: Even the psychopaths are a deep, horrible part of me - Sydney Morning Herald - July 8th, 2021
- 'I took a deep breath and got on with it' - Jorginho on his ice-cold winning penalty and Italy's belief | Official Site | Chelsea Football Club -... - July 8th, 2021
- Tour de France: Geraint Thomas to ride on through pain with Olympics in mind - The Guardian - July 8th, 2021
- Poetry Amid Parkinsons - The Wall Street Journal - July 8th, 2021
- Quinlan-Hammond Hall of Honor Taking Shape on USM Campus - Southern Miss Now - July 8th, 2021
- Lynn Hershman Leeson: The Artist Is Prescient - The New York Times - July 8th, 2021