A new research paper, An Image Is Worth 1616 Words: Transformers for Image Recognition at Scale, has the machine learning community both excited and curious. With Transformer architectures now being extended to the computer vision (CV) field, the paper suggests the direct application of Transformers to image recognition can outperform even the best convolutional neural networks when scaled appropriately. Unlike prior works using self-attention in CV, the scalable design does not introduce any image-specific inductive biases into the architecture.
But just whose potential breakthrough is this? The paper is currently under double-blind review for the International Conference on Learning Representations (ICLR) 2021, and thus the authors names and institutions are masked. The paper was spotted on the ICLR 2021 research repository OpenReview, and social media ML sleuths quickly went to work.
The paper were discussing here uses a JFT- 300M dataset that is not available to the public, only to Google, noted Yannic Kilcher, host of a popular eponymous YouTube channel. (JFT-300M is an internal dataset Google built to improve computer vision algorithms, that includes 300M images labelled with 18291 categories.) Kilcher identified numerous other clues suggesting the paper comes from Google, as part of a spirited and sarcastic rant against vulnerabilities and shortcomings in the double-blind review process.
Although reviewers comments remain anonymous, that doesnt mean the double-blind peer review process is sabotage-free. Some in the community have previously voiced concerns that the positive public comments a paper attracts on social media can give a paper an advantage during the review. Others are concerned that apparent hints indicating a paper is from a renowned institution could bias reviewers decisions.
The papers premise already has many respected AI practitioners predicting it could bring revolutionary changes to the CV field, where convolutional architectures are the go-to for difficult tasks. The paper asserts this reliance on CNNs is not necessary and a pure transformer can perform very well on image classification tasks when applied directly to sequences of image patches.
Google DeepMind Research Scientist Oriol Vinyals tweeted his take on the paper as farewell convolutions : ), with OpenAI Chief Scientist Ilya Sutskever responding that the new research offers an anonymous mathematical proof for attention is all you need.
Both researchers are very familiar with Transformer architectures, which enabled DeepMinds AlphaStar bot to defeat pro StarCraft players and OpenAIs 175 billion parameter language model GPT-3 to deliver SOTA performance in NLP tasks.
Sutskevers approval of the paper is noteworthy as he was one of the first to show the potential of CNNs in CV. In 2012, as a graduate student at the University of Toronto, Sutskever worked with AI pioneer Geoffrey Hinton and first author Alex Krizhevsky on the milestone paper ImageNet Classification with Deep Convolutional Neural Networks.
Tesla Director of AI Andrej Karpathy is also excited about the new paper. His PhD at the Stanford Vision Lab focused on the intersection of convolutional/recurrent neural networks and CV and NLP applications, and his advisor at Stanford was ImageNet creator Professor Fei-Fei Li. Karpathy said the paper takes further steps towards deprecating ConvNets with Transformers. Loving the increasing convergence of Vision/ NLP and the much more efficient/ flexible class of architectures.
As Synced previously reported, the use of Transformers has already been explored in the CV field. But classic ResNet-like architectures remain dominant in large-scale tasks such as image recognition. In May, Facebook AI released Detection Transformers (DETR) for object detection and panoptic segmentation tasks. DETR can directly predict the final set of detections by combining a common CNN with a Transformer architecture. In June, OpenAI showed that large Transformer-based language models trained on pixel sequences can generate coherent images without the use of labels.
While the research community will have to wait for official confirmation of the papers source, that delay is unlikely to diminish enthusiasm surrounding the significant technical insights and potential breakthroughs for the use of Transformer architectures in the expanding CV field.
The paper An Image Is Worth 1616 Words: Transformers for Image Recognition at Scale is available on OpenReview.
Reporter: Fangyu Cai | Editor: Michael Sarazen
Synced Report |A Survey of Chinas Artificial Intelligence Solutions in Response to the COVID-19 Pandemic 87 Case Studies from 700+ AI Vendors
This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available onAmazon Kindle.Along with this report, we also introduced adatabasecovering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.
Clickhereto find more reports from us.
We know you dont want to miss any news or research breakthroughs.Subscribe to our popular newsletterSynced Global AI Weeklyto get weekly AI updates.
- Dear World Travel Groups, Stop the Mind-Boggling Confusion Over Testing and Vaccines Now - Skift - January 25th, 2021
- 2021 technology trend review, part two: AI, knowledge graphs, and the COVID-19 effect - ZDNet - January 25th, 2021
- The Minneapolis Miracle was the best moment in Vikings playoff history - SB Nation - January 25th, 2021
- Poet brings deep sense of connection to virtual Unbound date - Columbia Daily Tribune - January 25th, 2021
- Johnson wants trade deal, but Biden has mind on other things - Chinadaily.com.cn - China Daily - January 25th, 2021
- The Big Bang Theory: 10 Times The Show Tackled Deep Issues - Screen Rant - January 25th, 2021
- Movie Review: A Performer, His Story And Mind-Bending Illusions Make 'In And Of Itself' Essential Viewing - Patch.com - January 25th, 2021
- Tackling tech's big diversity problem starts with education - Wired.co.uk - January 25th, 2021
- Mind Cure Announces Build-Out of Digital Therapeutics, iSTRYM: A Technology Platform for Mental Wellness Optimization & Psychedelic Research -... - January 6th, 2021
- Global Mindfulness Meditation Apps Market 2020-2025 (Impact of Covid-19) | Deep Relax, Smiling Mind, Inner Explorer, Inc., Committee for Children,... - January 6th, 2021
- Mind, Body and Soul: Shedding and growth - Ramona Sentinel - January 6th, 2021
- These Were Our Favorite Tech Stories From Around the Web in 2020 - Singularity Hub - January 6th, 2021
- The best products and tips for getting a better night's sleep - Fast Company - January 6th, 2021
- Dumb and dumber: The future of business AI depends on HI - ZDNet - January 6th, 2021
- AI Engineers Need to Think Beyond Engineering - Harvard Business Review - October 29th, 2020
- Alertness is an evergreen state of mind for the Jewish community - Security Magazine - October 29th, 2020
- M&A Plus Insurance - Please mind the gap: managing the timing considerations of warranty and indemnity insurance - Lexology - October 29th, 2020
- A deep recession should hurt Trump's reelection bid, but this isn't a usual downturn - CNN - October 29th, 2020
- Creators Of WBUR's 'Madness' Series Talk To Host Of CBC's 'Brainwashed' - WBUR - October 29th, 2020
- Latest Update 2020: Machine Learning Artificial intelligence Market by COVID19 Impact Analysis And Top Manufacturers: AIBrain, Amazon, Anki,... - October 29th, 2020
- Global Mindfulness Meditation Apps Market Expected To Reach Highest CAGR By 2026: Deep Relax, Smiling Mind, Inner Explorer, Inc., Committee for... - October 29th, 2020
- These Texas women arent flocking to Trump. They made up their minds weeks ago. - Houston Chronicle - October 29th, 2020
- Series on Mental Illness Present in Society - The Record Newspapers - TheRecordLive.com - October 29th, 2020
- How We Got Trump Voters to Change Their Mind - The Atlantic - October 27th, 2020
- Researchers Look To Animals To Give Reinforcement Learning Systems Common Sense - Unite.AI - October 27th, 2020
- The true dangers of AI are closer than we think - MIT Technology Review - October 27th, 2020
- An Old Dog's Tale: Wild visions fill the mind at election time - Chinook Observer - October 27th, 2020
- Mind the Gap - The Indian Express - October 27th, 2020
- Breaking News - HBO's "Crazy, Not Insane," A Provocative Look at the Minds of Serial Killers, Debuts November 18 - The Futon Critic - October 27th, 2020
- Family Hardship Helps Inspire Student's Sense of Wonder and Appreciation for the Mind and Body. - Bethel University News - October 27th, 2020
- Algorithmic bias - how do we tackle the underlying problem that inhibits the full potential of AI? - Diginomica - October 27th, 2020
- Enlightening New Book 'The New Prophet' Provides Deep Meditative Truths to Awaken the Heart - GlobeNewswire - October 27th, 2020
- The Deep Dark - The Indian Express - October 27th, 2020
- Technological innovations of AI in medical diagnostics - Health Europa - October 27th, 2020
- African Mental Health Summit to emphasize the importance of cultural understanding - MinnPost - October 27th, 2020
- The All-American Mind of a Militia Member - The New Republic - October 13th, 2020
- What Psychedelic Mushrooms Are Teaching Us About Human Consciousness - Discover Magazine - October 13th, 2020
- Words of wisdom: These books put the focus on the body and the mind - The Hindu - October 13th, 2020
- The state of AI in 2020: Biology and healthcare's AI moment, ethics, predictions, and graph neural networks - ZDNet - October 13th, 2020
- Not all self-help books workbut these 8 will actually rewire the way you think, live and do your job - CNBC - October 13th, 2020
- The Boys cast reflect on "mind-blowingly fun" season 2 finale - RadioTimes - October 13th, 2020
- How to Connect With the Co-Workers Youre Missing - The New York Times - October 13th, 2020
- I'm Thinking That I'm Too Stupid to Understand "I'm Thinking of Ending Things" - The Chicago Maroon - October 13th, 2020
- How AI And Blockchain Are Driving The Energy Transition - OilPrice.com - October 8th, 2020
- Things To Keep In Mind When Buying Dietary Supplements Online - Blog - The Island Now - October 8th, 2020
- Defining the Yellow mind The Manila Times - The Manila Times - October 8th, 2020
- Grid AI, From the Makers of PyTorch Lightning, Emerges From Stealth With $18.6m Series A to Close the Gap Between AI Research and Production -... - October 8th, 2020
- Ask a Therapist: How Not to Drown in the Deluge of the Negativity That Is 2020 - southseattleemerald.com - October 8th, 2020
- Local organizations earn Oregon Arts Commission grant to deliver integral arts education - The Register-Guard - October 8th, 2020
- Do Your Employees Feel Safe Reporting Abuse and Discrimination? - Harvard Business Review - October 8th, 2020
- The State of AI in 2020 and Beyond - CDOTrends - October 8th, 2020
- Good Job, Whale - The Cut - September 24th, 2020
- What the strange case of horse mutilations in France reveals about our state of mind - The Guardian - September 24th, 2020
- Review: The Flaming Lips dig deep with American Head - The Rice Thresher - September 24th, 2020
- FREE Self Development Series: "Curbing Traffic Jam in the Mind" - Patch.com - September 24th, 2020
- Happy Gut, Happy Mind: how the state of your gut affects your mental health - Evening Standard - September 24th, 2020
- How DeepMind Algorithms Helped Improve the Accuracy of Google Maps? - Analytics Insight - September 15th, 2020
- Elon Musk's brain-computer startup is getting ready to blow your mind - ZDNet - September 15th, 2020
- Far from being anti-religious, faith and spirituality run deep in Black Lives Matter - The Conversation US - September 15th, 2020
- Nvidia's Arm takeover sparks concern in the UK, co-founder says it's 'a disaster' - CNBC - September 15th, 2020
- The Guardians GPT-3-written article misleads readers about AI. Heres why. - TechTalks - September 15th, 2020
- What Is Yoga Nidra? Health Essentials from Cleveland Clinic - Health Essentials from Cleveland Clinic - September 15th, 2020
- Deep Dive: What would it take to change UNCW's mind? [Free] - Port City Daily - September 14th, 2020
- Eternal Blizzard in the Tired Mind: Kaufman delves ever deeper into the human psyche - The Stanford Daily - September 14th, 2020
- SoftBanks Arm sale hits a snag as UK opposition party warns of risk to jobs and digital sovereignty - CNBC - September 14th, 2020
- Scientific Psi? Neuralink and the smarter brain - Covalence - September 14th, 2020
- How to regulate AI, according to the 1967 Outer Space Treaty - Quartz - September 14th, 2020
- Reporter's notebook/It's time to let the games begin - The Daily Times - September 14th, 2020
- DCPS students receive online instruction from teacher in Greece - The Owensboro Times - September 14th, 2020
- Expand your mind with access to over 1,000 lectures from Tim Ferriss, Malcolm Gladwell, and more - MarketWatch - August 28th, 2020
- Global Machine Learning Artificial intelligence Market 2025 To Expect Maximum Benefit and Growth Potential During this COVID 19 Outbreak: AIBrain,... - August 28th, 2020
- It plays with the mind - Bangalore Mirror - August 28th, 2020
- We criticize because we care - Observer Online - August 28th, 2020
- Exercising toward a healthy mind - Johns Hopkins News-Letter - August 28th, 2020
- 78 percent parents don't mind if children have to skip school year due to pandemic: Survey - EdexLive - August 28th, 2020
- TSMC and Graphcore Prepare for AI Acceleration on 3nm - AnandTech - August 28th, 2020
- Frank Njenga: Over 40 years of healing the mind - Business Daily - August 28th, 2020
- Emily Dickinson is the unlikely hero of our time - The Conversation US - August 28th, 2020
- Visiongain publishes Automation in Biopharma Industry 2020-2030 report - PR Newswire UK - August 26th, 2020
- Scoop: The Trump-Navarro mind meld on the FDA - Axios - August 26th, 2020