The performance of deep neural networks (DNNs) relies heavily on their structures, and designing a good structure (aka architecture) tends to require extensive effort from human experts. The idea of an automatic structure-learning algorithm that can achieve performance on par with the best human-designed structures is thus increasingly appealing to machine learning researchers.
In the paper Learning Structures for Deep Neural Networks, a team from OneFlow and Microsoft explores unsupervised structure learning, leveraging the efficient coding principle, information theory and computational neuroscience to design a structure learning method that does not require labelled information and demonstrates empirically that larger entropy outputs in a deep neural network lead to better performance.
The researchers start with the assumption that the optimal structure of neural networks can be derived from the input features without labels. Their study probes whether it is possible to learn good DNN network structures from scratch in a fully automatic fashion, and what would be a principled way to reach this end.
The team references a principle borrowed from the biological nervous system domain the efficient coding principle which posits that a good brain structure forms an efficient internal representation of external environments. They apply the efficient coding principle to DNN architecture, proposing that the structure of a well-designed network should match the statistical structure of its input signals.
The efficient coding principle suggests that the mutual information between a models inputs and outputs should be maximized, and the team presents a solid Bayesian optimal classification theoretical foundation to support this. Specifically, they show that the top layer of any neural network (softmax linear classifier) and the independency between the nodes in the top hidden layer constitute a sufficient condition for making the softmax linear classifier act as a Bayesian optimal classifier. This theoretical foundation not only backs up the efficient coding principle, it also provides a way to determine the depth of a DNN.
The team then investigates how to leverage the efficient coding principle in the design of a structure-learning algorithm, and shows that sparse coding can implement the principle under the assumption of zero-peaked and heavy-tailed prior distributions. This suggests that an effective structure learning algorithm can be designed based on global group sparse coding.
The proposed structure-learning with sparse coding algorithm learns a structure layer by layer in a bottom-up manner. The raw features are at layer one, and given the predefined number of nodes in layer two, the algorithm will learn the connection between these two layers, and so on.
The researchers also describe how this proposed algorithm can learn inter-layer connections, handle invariance, and determine DNN depth. Finally, they conduct intensive experiments on the popular CIFAR-10 data set to evaluate the classification accuracies of their proposed structure learning method, the role of inter-layer connections, and the role of structure masks and network depth.
The results show that a learned-structure single-layer network achieves an accuracy of 63.0 percent, outperforming the single-layer baseline of 60.4 percent. In an inter-layer connection density evaluation experiment, the structures generated by the sparse coding approach outperform random structures, and at the same density level, always outperform the sparsifying-restricted Boltzmann machines (RBM) baseline. In the teams structure mask role evaluation, the structure prior provided by sparse coding is seen to improve performance. The network depth experiment meanwhile empirically justifies the proposed approach for determining DNN depth via coding efficiency.
Overall, the research proves the efficient coding principles effectiveness for unsupervised structure learning, and that the proposed global sparse coding-based structure-learning algorithms can achieve performance comparable with the best human-designed structures.
The paper Learning Structures for Deep Neural Networks is on arXiv.
Author: Hecate He |Editor: Michael Sarazen, Chain Zhang
We know you dont want to miss any news or research breakthroughs.Subscribe to our popular newsletterSynced Global AI Weeklyto get weekly AI updates.
- AI use cases are expanding and evolving in healthcare - Tech Wire Asia - August 1st, 2021
- DeepMind releases massive database of 3D protein structures - STAT - STAT - July 25th, 2021
- DeepMinds AI uncovers structure of 98.5 per cent of human proteins - News Nation USA - July 25th, 2021
- Good Morning, News: $600000 Settlement for 2017 PPB Killing, Deep Pockets Try Influencing MultCo DA, and Everything is GREAT at the Olympics! - The... - July 25th, 2021
- DeepMind open-sources AlphaFold 2 for protein structure predictions - VentureBeat - July 21st, 2021
- DeepMind's AlphaFold 2 reveal: Convolutions are out, attention is in - ZDNet - July 21st, 2021
- AI in Healthcare Market Growing Trade Among Emerging Economies Opening New Opportunities (2021-2031) | Nuance Communications, Inc., DeepMind... - July 21st, 2021
- Self-driving data centres: Managing the transition from human-to-AI workload management - IT Brief Australia - July 21st, 2021
- The scene is set | Royal St George's | The 149th Open - The Open - July 14th, 2021
- Tech Companies Leading the Cognitive Computing Race in 2021 - Analytics Insight - July 14th, 2021
- Science fiction: From hero's tales to deep thoughts - The Daily Advance - July 14th, 2021
- ADOT urges drivers to be prepared for monsoon weather dangers | Arizona Emergency information Network - az.gov - July 14th, 2021
- Taking the long view, taking control - Cape Cod Times - July 14th, 2021
- The ultimate reading list if you love to delve into the mind - Livemint - July 14th, 2021
- Midland Theatre executive director Anderson to retire Sept. 1 - The Newark Advocate - July 14th, 2021
- The Future of Deep Learning | - Analytics Insight - July 10th, 2021
- The movie you should watch this weekend: Eternal Sunshine of the Spotless Mind, the heart wants what it wants - The Indian Express - July 10th, 2021
- From the Extension: Keep health and safety in mind when firing up the grill - Daily Commercial - July 10th, 2021
- Follow your fever dreams with Nashville Hot Hot Dogs - The Takeout - July 10th, 2021
- Former Bureaucrats, Civil Society Express Deep Anguish On Violations and Breakdown of Governance in UP - The Citizen - July 10th, 2021
- The 36 questions that lead to love (but with your co-founder) - TechCrunch - July 10th, 2021
- Googles Supermodel: DeepMind Perceiver is a step on the road to an AI machine that could process anything... - ZDNet - July 8th, 2021
- At the ICA Watershed, artist Firelei Bez draws history from the deep - The Boston Globe - July 8th, 2021
- The Streaming Age Has Turned Poland Into a Deep-Pocketed Production Paradise - Hollywood Reporter - July 8th, 2021
- What Toxic Positivity Actually Is (& Why You Should Avoid It) - mindbodygreen.com - July 8th, 2021
- UCF Lake Nona Scientific Heart Is The First Sanatorium In Central Florida To Be offering Incisionless Mind Surgical treatment As An Complex Remedy For... - July 8th, 2021
- Young Cho's goals are even bigger than deep space; His team is working on astronaut nutrition that translates to earth - Westside Seattle - July 8th, 2021
- Two books to make you think differently about the ocean and the beach - Christian Science Monitor - July 8th, 2021
- Jimmy McGovern: Even the psychopaths are a deep, horrible part of me - Sydney Morning Herald - July 8th, 2021
- 'I took a deep breath and got on with it' - Jorginho on his ice-cold winning penalty and Italy's belief | Official Site | Chelsea Football Club -... - July 8th, 2021
- Tour de France: Geraint Thomas to ride on through pain with Olympics in mind - The Guardian - July 8th, 2021
- Poetry Amid Parkinsons - The Wall Street Journal - July 8th, 2021
- Quinlan-Hammond Hall of Honor Taking Shape on USM Campus - Southern Miss Now - July 8th, 2021
- Lynn Hershman Leeson: The Artist Is Prescient - The New York Times - July 8th, 2021
- The Great Artist: A film that explores the brilliance of a suffering mind - The Indian Express - July 8th, 2021
- 10 UK retreats to re-energise mind, body and soul - The Guardian - June 27th, 2021
- Keep safety in mind when cooling off in the Willamette River - KGW.com - June 27th, 2021
- Matters of the Mind: Recruiting the body mind and soul for relief from emotional tumult - The Indian Express - June 27th, 2021
- 'You weep and then you try to determine how to help': Families, worshippers cling to hope after Florida condo building collapse - USA TODAY - June 27th, 2021
- Danielle Lao Gets Back to Wimbledon, and Makes the Most of It - The New York Times - June 27th, 2021
- Bernie Rabik: Is Age Only a Number? - The Times - June 27th, 2021
- New sales platform brand In Mind Cloud to rapidly digitalize the manufacturing sector - PRNewswire - June 25th, 2021
- Ophthalmology: A pioneer in the field of artificial intelligence - Ophthalmology Times - June 25th, 2021
- A Deep Dive into the Scoring Metrics of the NTIA's Broadband Infrastructure Program - Benton Foundation - June 25th, 2021
- Don't mind the gap: 'Innovation cohesion' is new route to bridge east-west divide - Science Business - June 25th, 2021
- A menu for the mind: Angelus reading recommendations 2021 - Angelus News - June 25th, 2021
- Finding Relaxation: Your 5-Minute Read - Healthline - June 25th, 2021
- DeepMind wants to use its AI to cure neglected diseases - Wired.co.uk - June 23rd, 2021
- Shame in Medical Training, and Owning Our Stories - Medscape - June 23rd, 2021
- Are toxins flushed out of the brain during sleep? - Harvard Health - June 23rd, 2021
- Thinking Differently: The Most Important Thing You Need to do to Advance Your Company - insideARM.com - June 23rd, 2021
- Vegan pop-up Don't Be Chick'n launches a Bay Area food truck - The Oaklandside - June 23rd, 2021
- Boosting inspection performance with AI and deep learning - Packaging Europe - June 22nd, 2021
- Time for a Casual Deep Dive on Karl From The Bachelorette - Yahoo Lifestyle - June 22nd, 2021
- AI in Healthcare Market 2021 Best Workable Strategy That Will Help to Boost your industry Growth Till 2031 | DeepMind Technologies Limited, IBM... - June 22nd, 2021
- International Yoga Day 2021: The Benefits Of Deep Breathing And Meditation For Diabetics - NDTV - June 22nd, 2021
- International Yoga Day: Does yoga really help heal your body and mind? - Gulf News - June 22nd, 2021
- Public Lands Preserve More Than What's on the Surface - Sierra Magazine - June 22nd, 2021
- Does alcohol have a place in a healthy lifestyle? - Walla Walla Union-Bulletin - June 22nd, 2021
- Dannals, Fisher, and Kyung Receive 2021 Tuck Teaching Excellence Awards - Tuck School - June 22nd, 2021
- Emergency and survival items you need to put in your pack - Part 2 - goHUNT.com - June 19th, 2021
- Studying content moderation, disinformation, and deep fakes. Astroturfing, bias, and rumor control. - The CyberWire - June 19th, 2021
- From The Deep Sea, A Chance To Learn, Preserve on the Gulf Coast - Mississippi Free Press - June 19th, 2021
- India Inc can delve deep within and construct its own reality: Deepak Chopra to ETILC members - Economic Times - June 19th, 2021
- Investors say Eindhoven poised to become Netherlands No. 2 tech hub - TechCrunch - June 19th, 2021
- What should you read in the summer of 2021? Here are the best books. - Berkeleyside - June 19th, 2021
- Inside the elite UK tech event attended by the rich and famous - CNBC - June 19th, 2021
- Inside the fight to reclaim AI from Big Techs control - MIT Technology Review - June 19th, 2021
- International Yoga Day: 6 tips for de-stressing the overthinker in you - wknd. - June 19th, 2021
- Validation and clinical applicability of a deep learning system for retinal disease (Tuesday, 10th August 2021) City, University of London - City,... - June 16th, 2021
- Physical works out the dark side of the mind in an honest way - Metro US - June 16th, 2021
- The Number One Voice Certain To Drive Your Leadership Off The Deep End - Forbes - June 16th, 2021
- Consumer Companies Bail on Non-Core Assets: "Deep is the New Wide" - Mergers & Acquisitions - June 16th, 2021
- Bengio Team Proposes Flow Network-Based Generative Models That Learn a Stochastic Policy From a Sequence of Actions - Synced - June 16th, 2021
- It's more than just skin-deep: Feel and look amazing with Bubble Skincare | Sponsored - Harvard Crimson - June 16th, 2021
- AI in Healthcare Market Drivers, Challenges, Opportunities and Competitive Strategy Over 2021-2031 | Nuance Communications, Inc., DeepMind... - June 16th, 2021
- NVIDIA and the battle for the future of AI chips - Wired.co.uk - June 16th, 2021
- Accelerating Deep Learning on the JVM with Apache Spark and NVIDIA GPUs - InfoQ.com - June 16th, 2021
- AI in Europe: Who's leading the way and where is it heading? - Siliconrepublic.com - June 16th, 2021
- Review: Bo Burnhams Inside is a successful depiction of a lonely mind - Los Angeles Times - June 16th, 2021