Cloud Hosting

While it is generally accepted that network depth is responsible for the high performance of todays deep learning (DL) models, adding depth also brings downsides such as increased latency and computational burdens, which can bottleneck progress in DL. Is it possible to achieve similarly high performance without deep networks?

In the new paper Non-deep Networks, a research team from Princeton University and Intel Labs argues that it is, proposing ParNet (Parallel Networks), a novel non-deep architecture that achieves performance competitive with its state-of-the-art deep counterparts.

The team summarizes their studys contributions as:

The main design feature of ParNet is its use of parallel subnetworks or substructures (referred to as streams in the paper) that process features at different resolutions. The features from different streams are fused at a later stage in the network used for downstream tasks. This approach enables ParNet to function effectively with a network depth of only 12 layers, orders of magnitude lower than ResNet models, for example, which in extreme cases can include up to 1000 layers.

A key ParNet component is its RepVGG-SSE, a modified Rep-VGG block with a purpose-built Skip-Squeeze-Excitation module. ParNet also contains a downsampling block that reduces resolution and increases width to enable multi-scale processing, and a fusion block that combines information from multiple resolutions.

In their empirical study, the team compared the proposed ParNet with state-of-the-art deep neural networks baselines such as ResNet110 and DenseNet on large-scale visual recognition benchmarks that included ImageNet, CIFAR and MS-COCO.

The results show that a ParNet with a depth of just 12 layers was able to achieve top-1 accuracies of over 80 percent on ImageNet, 96 percent on CIFAR10, and 81 percent on CIFAR100. The team also demonstrated a detection network with a 12 layer backbone that achieved an average precision of 48 percent on the MS-COCO large-scale object detection, segmentation and captioning dataset.

Overall, the study provides the first empirical proof that non-deep networks can perform competitively with their deep counterparts on large-scale visual recognition benchmarks. The team hopes their work can contribute to the development of neural networks that are a better fit for future multi-chip processors.

The code is available on the projects GitHub. The paper Non-deep Networks is on arXiv.

Author: Hecate He |Editor: Michael Sarazen

We know you dont want to miss any news or research breakthroughs.Subscribe to our popular newsletterSynced Global AI Weeklyto get weekly AI updates.

Like Loading...

Go here to read the rest:
Deeper Is Not Necessarily Better: Princeton U & Intel's 12-Layer Parallel Networks Achieve Performance Competitive With SOTA Deep Networks -...