The State of AI Report 2020 is a comprehensive report on all things AI. Picking up from where we left off in summarizing key findings, we continue the conversation with authors Nathan Benaich and Ian Hogarth. Benaich is the founder of Air Street Capital and RAAIS, and Hogarth is an AI angel investor and a UCL IIPP visiting professor.
Key themes we covered so far were AI democratization, industrialization, and the way to artificial general intelligence. We continue with healthcare and biology's AI moment, research and application breakthroughs, AI ethics, and predictions.
A key point discussed with Benaich and Hogarth was the democratization of AI: What it means, whether it applies, and how to compete against behemoths who have the resources it takes to train huge machine learning models at scale.
One of the ideas examined in the report is to take pre-existing models and fine-tune them to specific domains. Benaich noted that taking a large model, or a pre-trained model in one field, and moving it to another field can work to bootstrap performance to a higher level:
"As far as biology and healthcare are becoming increasingly digital domains with lots of imaging, whether that relates to healthcare conditions or what cells look like when they're diseased, compiling data sets to describe that and then using transfer learning from ImageNet into those domains has yielded much better performance than starting from scratch."
This, Benaich went on to add, plays into one of the dominant themes in the report: Biology -- in which Benaich has a background -- and healthcare have their AI moment. There are examples of startups at the cutting edge of R&D moving to production tackling problems in biology. An application area Benaich highlighted was drug screening:
"If I have a software product, I can generate lots of potential drugs that could work against the disease protein that I'm interested in targeting. How do I know out of the thousands or hundreds of thousands of possible drugs, which one will work? And assuming I can figure out which one might work, how do I know if I can actually make it?"
Beyond computer vision, Benaich went on to add, there are several examples of AI language models being useful in protein engineering or in understanding DNA, "essentially treating a sequence of amino acids that encode proteins or DNA as just another form of language, a form of strings that language models can interpret just in the same way they can interpret characters that spell out words."
The FDA published a new proposal to embrace the highly iterative and adaptive nature of AI systems in what they call a "total product lifecycle" regulatory approach built on good machine learning practices.
Transformer-based language models such as GPT3 have also been applied to tasks such as completing images or converting code between different programming languages. Benaich and Hogarth note that the transformer's ability to generalize is remarkable, but at the same time offer a word of warning in the example of code: No expert knowledge required, but no guarantees that the model didn't memorize the functions either.
This discussion was triggered by the question -- posed by some researchers -- whether progress in mature areas of machine learning is stagnant. In our view, the fact that COVID19 has dominated 2020 is also reflected in the impact it has had on AI. And there are examples of how AI has been applied in biology and healthcare to tackle COVID19.
Benaich used examples from biology and healthcare to establish that beyond research, the application area is far from stagnant. The report includes work in this area ranging from startups such as InVivo and Recursionto Google Health, DeepMind, and the NHS.
What's more, the US Medicaid and Medicare system has approved a medical imaging product for stroke that's based on AI. Despite pre-existing FDA approvals for deep-learning based medical imaging, whether that's for stroke, mammography, or broken bones, this is the only so far that has actually gotten reimbursement, noted Benaich:
"Many people in the field feel that reimbursement is the critical moment. That's the economic incentive for doctors to prescribe, because they get paid back. So we think that's a major event. A lot of work to be done, of course, to scale this and to make sure that more patients are eligible for that reimbursement, but still major nonetheless."
Interestingly, the FDA has also published a new proposal to embrace the highly iterative and adaptive nature of AI systems in what they call a "total product lifecycle" regulatory approach built on good machine learning practices.
The report also includes a number of examples that Benaich stated: "Prove that the large pharma companies are actually getting value from working with their first drug discovery companies." This discussion naturally leads to the topic of progress in a specific area of machine learning: graph neural networks.
The connection was how graph neural networks (GNNs) are used to enhance chemical property prediction and guide antibiotic drug screening, leading to new drugs in vivo. Most deep learning methods focus on learning from two-dimensional input data. That is, data represented as matrices. GNNs are an emerging family of methods that are designed to process 3D data. This may sound cryptic, but it's a big deal. The reason is that it enables more information to be processed by the neural network.
"I think it comes down to one topic, which is the right representation of biological data that actually expresses all the complexity and the physics and chemistry and living nuances of a biological system into a compact, easy to describe mathematical representation that a machine learning model can do something with," said Benaich.
Sometimes it's hard to conceptualize biological systems as a matrix, so it could very well be that we're just not exploiting all of that implicit information that resides in a biological system, he went on to add. This is why graphical representations are an interesting next step -- because it feels intuitive as a tool to represent something that is connected, such as a chemical molecule.
Graph neural networks enable the representation of 3-dimensional structures for deep learning. This mean being able to capture, and use, more information, and lends itself well to the field of biology. Image: M. Bronstein
Benaich noted examples in molecule property prediction and chemical synthesis planning, but also in trying to identify novel small molecules. Small molecules are treated as Lego building blocks. By using advances in DNA sequencing, all of these chemicals are mixed in a tube with a target molecule, and researchers can see what building blocks have assembled and bound to the target of interest.
When candidate small molecules that seem to work have been identified, GNNs can be used to try and learn what commonalities these building blocks have that make them good binders for the target of interest. Adding this machine learning layer to a standard and well-understood chemical screening approach gives a several-fold improvement on the baseline.
Hogarth on his part mentioned a recent analysis, arguing that GNNs, the transformer architecture, and attention-based methods used in language models share the same underlying logic, as you can think of sentences for the connected word graphs. Hogarth noted the way that the transform architecture is creeping into lots of unusual use cases, and how scaling it up is increasing the impact:
"The meta point around the neural networks and these attention-based methods, in general, is that they seem to represent a sort of a general enough approach that there's going to be progress just by continuing to hammer very hard on that nail for the next two years. And one of the ways in which I'm challenging myself is to assume that we might see a lot more progress just by doing the same thing with more aggression for a bit.
And so I would assume that some of the gains that have been found in these GNNs cross-pollinate with the work that's happening with language models and transformers. And that approach continues to be a very fertile area for the kind of super general, high-level AGI-like research."
There's a ton of topics we could pick to dissect from Benaich and Hogarth's work, such as the use of PyTorch overtaking TensorFlow in research, the boom in federated learning, the analysis on talent and retainment per geography, progress (or lack thereof) in autonomous vehicles, AI chips, and AutoML. We encourage readers to dive into the report to learn more. But we wrap up with something different.
Hogarth mentioned that the speculation phase in AI for biology and healthcare is starting, with lots of capital flowing. There are going to be some really amazing companies that come out of it, and we will start to see a real deployment phase kick in. But it's equally certain, he went on to add, there are going to be instances that will be revealed to be total frauds.
So, what about AI ethics? Benaich and Hogarth cite work by pioneers in the field, touching upon issues such as commercial gender classification, unregulated police facial recognition, the ethics of algorithms, and regulating robots. For the most part, the report focuses on facial recognition. Facial recognition is widespread the world over and has lead to controversy, as well as wrongful arrests. More thoughtful approaches seem to gather steam, Benaich and Hogarth note.
The duo's report cites examples such as Microsoft deleting its database of 10 million faces (the largest available) collected without consent, Amazon announced a one-year pause on letting the police use its facial recognition tool Rekognition to give "congress enough time to put in place appropriate rules." And IBM announced it would sunset its general-purpose facial recognition products.
Hogarth referred to an incident in which a UK citizen claimed his human rights were breached when he was photographed while Christmas shopping. Although judges ruled against the claimant, they also established an important new duty for the police to make sure that discrimination is proactively "eliminated." This means that action on bias cannot be legally deferred until the tech has matured:
"This creates a much higher bar to deploying this software. And it creates almost a legal opportunity for anyone who experiences bias at the hands of an algorithm to have a foundation for suing the government or a private act of defiance technology," Hogarth said.
AI ethics often focuses on facial recognition, but there are more and more domains it's becoming relevant in.
Hogarth also emphasized another approach, which he termed "API driven auditability." He referred to a new law passed in Washington State with active support from Microsoft. This law restricts law enforcement's use of facial recognition technology, by demanding that the software used must be accessible to an independent third party via an API to assess for "accuracy and unfair performance differences" across characteristics like race or gender.
Of course, even narrowing our focus on AI ethics, the list is endless: From bias to the use of technology in authoritarian regimes and/or for military purposes, AI nationalism, or the US tax code incentivizing replacing humans with robots, there's no shortage of causes for concern. Benaich and Hogarth, on their part, close their report by offering a number of predictions for the coming year:
The race to build larger language models continues and we see the first 10 trillion parameter model. Attention-based neural networks move from NLP to computer vision in achieving the state of the art results. A major corporate AI lab shuts down as its parent company changes strategy. In response to US DoD activity and investment in US-based military AI startups, a wave of Chinese and European defense-focused AI startups collectively raise over $100 million in the next 12 months.
One of the leading AI-first drug discovery startups (e.g. Recursion, Exscientia) either IPOs or is acquired for over $1 billion. DeepMind makes a major breakthrough in structural biology and drug discovery beyond AlphaFold. Facebook makes a major breakthrough in augmented and virtual reality with 3D computer vision. And NVIDIA does not end up completing its acquisition of Arm.
The record for predictions offered in last year's State of AI Report was pretty good - they made 5 out of 6. Let's see how this year's set of predictions fares.
- Mind Cure Announces Build-Out of Digital Therapeutics, iSTRYM: A Technology Platform for Mental Wellness Optimization & Psychedelic Research -... - January 6th, 2021
- Global Mindfulness Meditation Apps Market 2020-2025 (Impact of Covid-19) | Deep Relax, Smiling Mind, Inner Explorer, Inc., Committee for Children,... - January 6th, 2021
- Mind, Body and Soul: Shedding and growth - Ramona Sentinel - January 6th, 2021
- These Were Our Favorite Tech Stories From Around the Web in 2020 - Singularity Hub - January 6th, 2021
- The best products and tips for getting a better night's sleep - Fast Company - January 6th, 2021
- Dumb and dumber: The future of business AI depends on HI - ZDNet - January 6th, 2021
- AI Engineers Need to Think Beyond Engineering - Harvard Business Review - October 29th, 2020
- Alertness is an evergreen state of mind for the Jewish community - Security Magazine - October 29th, 2020
- M&A Plus Insurance - Please mind the gap: managing the timing considerations of warranty and indemnity insurance - Lexology - October 29th, 2020
- A deep recession should hurt Trump's reelection bid, but this isn't a usual downturn - CNN - October 29th, 2020
- Creators Of WBUR's 'Madness' Series Talk To Host Of CBC's 'Brainwashed' - WBUR - October 29th, 2020
- Latest Update 2020: Machine Learning Artificial intelligence Market by COVID19 Impact Analysis And Top Manufacturers: AIBrain, Amazon, Anki,... - October 29th, 2020
- Global Mindfulness Meditation Apps Market Expected To Reach Highest CAGR By 2026: Deep Relax, Smiling Mind, Inner Explorer, Inc., Committee for... - October 29th, 2020
- These Texas women arent flocking to Trump. They made up their minds weeks ago. - Houston Chronicle - October 29th, 2020
- Series on Mental Illness Present in Society - The Record Newspapers - TheRecordLive.com - October 29th, 2020
- How We Got Trump Voters to Change Their Mind - The Atlantic - October 27th, 2020
- Researchers Look To Animals To Give Reinforcement Learning Systems Common Sense - Unite.AI - October 27th, 2020
- The true dangers of AI are closer than we think - MIT Technology Review - October 27th, 2020
- An Old Dog's Tale: Wild visions fill the mind at election time - Chinook Observer - October 27th, 2020
- Mind the Gap - The Indian Express - October 27th, 2020
- Breaking News - HBO's "Crazy, Not Insane," A Provocative Look at the Minds of Serial Killers, Debuts November 18 - The Futon Critic - October 27th, 2020
- Family Hardship Helps Inspire Student's Sense of Wonder and Appreciation for the Mind and Body. - Bethel University News - October 27th, 2020
- Algorithmic bias - how do we tackle the underlying problem that inhibits the full potential of AI? - Diginomica - October 27th, 2020
- Enlightening New Book 'The New Prophet' Provides Deep Meditative Truths to Awaken the Heart - GlobeNewswire - October 27th, 2020
- The Deep Dark - The Indian Express - October 27th, 2020
- Technological innovations of AI in medical diagnostics - Health Europa - October 27th, 2020
- African Mental Health Summit to emphasize the importance of cultural understanding - MinnPost - October 27th, 2020
- The All-American Mind of a Militia Member - The New Republic - October 13th, 2020
- What Psychedelic Mushrooms Are Teaching Us About Human Consciousness - Discover Magazine - October 13th, 2020
- Words of wisdom: These books put the focus on the body and the mind - The Hindu - October 13th, 2020
- Not all self-help books workbut these 8 will actually rewire the way you think, live and do your job - CNBC - October 13th, 2020
- The Boys cast reflect on "mind-blowingly fun" season 2 finale - RadioTimes - October 13th, 2020
- How to Connect With the Co-Workers Youre Missing - The New York Times - October 13th, 2020
- I'm Thinking That I'm Too Stupid to Understand "I'm Thinking of Ending Things" - The Chicago Maroon - October 13th, 2020
- How AI And Blockchain Are Driving The Energy Transition - OilPrice.com - October 8th, 2020
- Things To Keep In Mind When Buying Dietary Supplements Online - Blog - The Island Now - October 8th, 2020
- Defining the Yellow mind The Manila Times - The Manila Times - October 8th, 2020
- Farewell Convolutions ML Community Applauds Anonymous ICLR 2021 Paper That Uses Transformers for Image Recognition at Scale - Synced - October 8th, 2020
- Grid AI, From the Makers of PyTorch Lightning, Emerges From Stealth With $18.6m Series A to Close the Gap Between AI Research and Production -... - October 8th, 2020
- Ask a Therapist: How Not to Drown in the Deluge of the Negativity That Is 2020 - southseattleemerald.com - October 8th, 2020
- Local organizations earn Oregon Arts Commission grant to deliver integral arts education - The Register-Guard - October 8th, 2020
- Do Your Employees Feel Safe Reporting Abuse and Discrimination? - Harvard Business Review - October 8th, 2020
- The State of AI in 2020 and Beyond - CDOTrends - October 8th, 2020
- Good Job, Whale - The Cut - September 24th, 2020
- What the strange case of horse mutilations in France reveals about our state of mind - The Guardian - September 24th, 2020
- Review: The Flaming Lips dig deep with American Head - The Rice Thresher - September 24th, 2020
- FREE Self Development Series: "Curbing Traffic Jam in the Mind" - Patch.com - September 24th, 2020
- Happy Gut, Happy Mind: how the state of your gut affects your mental health - Evening Standard - September 24th, 2020
- How DeepMind Algorithms Helped Improve the Accuracy of Google Maps? - Analytics Insight - September 15th, 2020
- Elon Musk's brain-computer startup is getting ready to blow your mind - ZDNet - September 15th, 2020
- Far from being anti-religious, faith and spirituality run deep in Black Lives Matter - The Conversation US - September 15th, 2020
- Nvidia's Arm takeover sparks concern in the UK, co-founder says it's 'a disaster' - CNBC - September 15th, 2020
- The Guardians GPT-3-written article misleads readers about AI. Heres why. - TechTalks - September 15th, 2020
- What Is Yoga Nidra? Health Essentials from Cleveland Clinic - Health Essentials from Cleveland Clinic - September 15th, 2020
- Deep Dive: What would it take to change UNCW's mind? [Free] - Port City Daily - September 14th, 2020
- Eternal Blizzard in the Tired Mind: Kaufman delves ever deeper into the human psyche - The Stanford Daily - September 14th, 2020
- SoftBanks Arm sale hits a snag as UK opposition party warns of risk to jobs and digital sovereignty - CNBC - September 14th, 2020
- Scientific Psi? Neuralink and the smarter brain - Covalence - September 14th, 2020
- How to regulate AI, according to the 1967 Outer Space Treaty - Quartz - September 14th, 2020
- Reporter's notebook/It's time to let the games begin - The Daily Times - September 14th, 2020
- DCPS students receive online instruction from teacher in Greece - The Owensboro Times - September 14th, 2020
- Expand your mind with access to over 1,000 lectures from Tim Ferriss, Malcolm Gladwell, and more - MarketWatch - August 28th, 2020
- Global Machine Learning Artificial intelligence Market 2025 To Expect Maximum Benefit and Growth Potential During this COVID 19 Outbreak: AIBrain,... - August 28th, 2020
- It plays with the mind - Bangalore Mirror - August 28th, 2020
- We criticize because we care - Observer Online - August 28th, 2020
- Exercising toward a healthy mind - Johns Hopkins News-Letter - August 28th, 2020
- 78 percent parents don't mind if children have to skip school year due to pandemic: Survey - EdexLive - August 28th, 2020
- TSMC and Graphcore Prepare for AI Acceleration on 3nm - AnandTech - August 28th, 2020
- Frank Njenga: Over 40 years of healing the mind - Business Daily - August 28th, 2020
- Emily Dickinson is the unlikely hero of our time - The Conversation US - August 28th, 2020
- Visiongain publishes Automation in Biopharma Industry 2020-2030 report - PR Newswire UK - August 26th, 2020
- Scoop: The Trump-Navarro mind meld on the FDA - Axios - August 26th, 2020
- Emily Dickinson is the unlikely hero of our time - Huron Daily Tribune - August 26th, 2020
- 5 ways to ease pain using the mind-body connection - Harvard Health Blog - Harvard Health - August 26th, 2020
- COVID-19: Heal Your Diseases and Illness with Your Subconscious Mind - CEOWORLD magazine - August 26th, 2020
- London's 'Tech City' may never be the same again after the coronavirus - CNBC - August 26th, 2020
- Environmental Sustainability And AI - Forbes - August 26th, 2020
- BetMGM Has Crazy 100-1 Odds On The Yankees or Nats To Go Deep Tonight - Crossing Broad - July 25th, 2020
- Chest-Deep' Wildflowers Are Blooming in Reds Meadow - NBC Southern California - July 25th, 2020
- The Man-Eater part 3 - Lake County Record-Bee - July 25th, 2020