Google sent a jolt of unease into the climate change debate this month when it disclosed that emissions from its data centers rose 13% in 2023, citing the AI transition in its annual environmental report. But according to Jeff Dean, Googles chief scientist, the report doesnt tell the full story and gives AI more than its fair share of blame.

Dean, who is chief scientist at both Google DeepMind and Google Research, said that Google is not backing off its commitment to be powered by 100% clean energy by the end of 2030. But, he said, that progress is not necessarily a linear thing because some of Googles work with clean energy providers will not come on line until several years from now.

Those things will provide significant jumps in the percentage of our energy that is carbon-free energy, but we also want to focus on making our systems as efficient as possible, Dean said at Fortunes Brainstorm Tech conference on Tuesday, in an onstage interview with Fortunes AI editor Jeremy Kahn.

Dean went on to make the larger point that AI is not as responsible for increasing data center usage, and thus carbon emissions, as critics make it out to be.

Theres been a lot of focus on the increasing energy usage of AI, and from a very small base that usage is definitely increasing, Dean said. But I think people often conflate that with overall data center usage of which AI is a very small portion right now but growing fast and then attribute the growth rate of AI based computing to the overall data center usage.

Dean said that its important to examine all the data and the true trends that underlie this, though he did not elaborate on what those trends were.

One of Googles earliest employees, Dean joined the company in 1999 and is credited with being one of the key people who transformed its early internet search engine into a powerful system capable of indexing the internet and reliably serving billions of users. Dean cofounded the Google Brain project in 2011, spearheading the companys efforts to become a leader in AI. Last year, Alphabet merged Google Brain with DeepMind, the AI company Google acquired in 2014, and made Dean chief scientist reporting directly to CEO Sundar Pichai.

By combining the two teams, Dean said that the company has a better set of ideas to build on, and can pool the compute so that we focus on training one large-scale effort like Gemini rather than multiple fragmented efforts.

Dean also responded to a question about the status of Googles Project Astraa research project which DeepMind leader Demis Hassabis unveiled in May at Google I/O, the companys annual developer conference. Described by Hassabis as a universal AI agent that can understand the context of a users environment, a video demonstration of Astra showed how users could point their phone camera to nearby objects and ask the AI agent relevant questions such as What neighborhood am I in? or Did you see where I left my glasses?

At the time, the company said the Astra technology will come to the Gemini app later this year. But Dean put it more conservatively: Were hoping to have something out into the hands of test users by the end of the year, he said.

The ability to combine Gemini models with models that actually have agency and can perceive the world around you in a multimodal way is going to be quite powerful, Dean said. Were obviously approaching this responsibly, so we want to make sure that the technology is ready and that it doesnt have unforeseen consequences, which is why well roll it out first to a smaller set of initial test users.

As for the continued evolution of AI models, Dean noted that additional data and computing power alone will not suffice. A couple more generations of scaling will get us considerably farther, Dean said, but eventually there will be a need for some additional algorithmic breakthroughs.

Dean said his team has long focused on ways to combine scaling with algorithmic approaches in order to improve factuality and reasoning capabilities, so that the model can imagine plausible outputs and reason its way through which one makes the most sense.

Those kind of advances Dean said, will be important to really make these models robust and more reliable than they already are.

Read more coverage from Brainstorm Tech 2024:

Wiz CEO says consolidation in the security market is truly a necessity as reports swirl of $23 billion Google acquisition

Why Grindrs CEO believes synthetic employees are about to unleash a brutal talent war for tech startups

Experts worry that a U.S.-China cold war could turn hot: Everyones waiting for the shoe to drop in Asia

Here is the original post:
Google AI heavyweight Jeff Dean talks about algorithmic breakthroughs and data center emissions - Fortune

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More

Mixture-of-Experts (MoE) has become a popular technique for scaling large language models (LLMs) without exploding computational costs. Instead of using the entire model capacity for every input, MoE architectures route the data to small but specialized expert modules. MoE enables LLMs to increase their parameter while keeping inference costs low. MoE is used in several popular LLMs, including Mixtral, DBRX, Grok and reportedly GPT-4.

However, current MoE techniques have limitations that restrict them to a relatively small number of experts. In a new paper, Google DeepMind introduces Parameter Efficient Expert Retrieval (PEER), a novel architecture that can scale MoE models to millions of experts, further improving the performance-compute tradeoff of large language models.

The past few years have shown that scaling language models by increasing their parameter count leads to improved performance and new capabilities. However, there is a limit to how much you can scale a model before running into computational and memory bottlenecks.

Every transformer block used in LLMs has attention layers and feedforward (FFW) layers. The attention layer computes the relations between the sequence of tokens fed to the transformer block. The feedforward network is responsible for storing the models knowledge. FFW layers account for two-thirds of the models parameters and are one of the bottlenecks of scaling transformers. In the classic transformer architecture, all the parameters of the FFW are used in inference, which makes their computational footprint directly proportional to their size.

MoE tries to address this challenge by replacing the FFW with sparsely activated expert modules instead of a single dense FFW layer. Each of the experts contains a fraction of the parameters of the full dense layer and specializes in certain areas. The MoE has a router that assigns each input to several experts who are likely to provide the most accurate answer.

By increasing the number of experts, MoE can increase the capacity of the LLM without increasing the computational cost of running it.

According to recent studies, the optimal number of experts for an MoE model is related to several factors, including the number of training tokens and the compute budget. When these variables are balanced, MoEs have consistently outperformed dense models for the same amount of compute resources.

Furthermore, researchers have found that increasing the granularity of an MoE model, which refers to the number of experts, can lead to performance gains, especially when accompanied by an increase in model size and training data.

High-granularity MoE can also enable models to learn new knowledge more efficiently. Some studies suggest that by adding new experts and regularizing them properly, MoE models can adapt to continuous data streams, which can help language models deal with continuously changing data in their deployment environments.

Current approaches to MoE are limited and unscalable. For example, they usually have fixed routers that are designed for a specific number of experts and need to be readjusted when new experts are added.

DeepMinds Parameter Efficient Expert Retrieval (PEER) architecture addresses the challenges of scaling MoE to millions of experts. PEER replaces the fixed router with a learned index to efficiently route input data to a vast pool of experts. For each given input, PEER first uses a fast initial computation to create a shortlist of potential candidates before choosing and activating the top experts. This mechanism enables the MoE to handle a very large number of experts without slowing down.

Unlike previous MoE architectures, where experts were often as large as the FFW layers they replaced, PEER uses tiny experts with a single neuron in the hidden layer. This design enables the model to share hidden neurons among experts, improving knowledge transfer and parameter efficiency. To compensate for the small size of the experts, PEER uses a multi-head retrieval approach, similar to the multi-head attention mechanism used in transformer models.

A PEER layer can be added to an existing transformer model or used to replace an FFW layer. PEER is also related to parameter-efficient fine-tuning (PEFT) techniques. In PEFT techniques, parameter efficiency refers to the number of parameters that are modified to fine-tune a model for a new task. In PEER, parameter efficiency reduces the number of active parameters in the MoE layer, which directly affects computation and activation memory consumption during pre-training and inference.

According to the paper, PEER could potentially be adapted to select PEFT adapters at runtime, making it possible to dynamically add new knowledge and features to LLMs.

PEER might be used in DeepMinds Gemini 1.5 models, which according to the Google blog uses a new Mixture-of-Experts (MoE) architecture.

The researchers evaluated the performance of PEER on different benchmarks, comparing it against transformer models with dense feedforward layers and other MoE architectures. Their experiments show that PEER models achieve a better performance-compute tradeoff, reaching lower perplexity scores with the same computational budget as their counterparts.

The researchers also found that increasing the number of experts in a PEER model leads to further perplexity reduction.

This design demonstrates a superior compute-performance trade-off in our experiments, positioning it as a competitive alternative to dense FFW layers for scaling foundation models, the researchers write.

The findings are interesting because they challenge the long-held belief that MoE models reach peak efficiency with a limited number of experts. PEER shows that by applying the right retrieval and routing mechanisms, it is possible to scale MoE to millions of experts. This approach can help further reduce the cost and complexity of training and serving very large language models.

VB Daily

Stay in the know! Get the latest news in your inbox daily

By subscribing, you agree to VentureBeat's Terms of Service.

Thanks for subscribing. Check out more VB newsletters here.

An error occured.

See the rest here:
DeepMinds PEER scales language models with millions of tiny experts - VentureBeat

It is simply not sustainable to keep training more advanced AI models using current energy technology. We need models to be trained faster, cheaper, and in more environmentally friendly ways. Google DeepMind has now shared new research on JEST (Joint Example Selection Training), a way of training AI models that is 13 times faster and 10 times more power-efficient than current techniques.

As the AI industry grows, so are concerns about the environmental impact of data centers required to train these sophisticated models. The JEST method arrives just in time, addressing the escalating energy demands of AI training processes. By significantly reducing the computational overhead, JEST could help mitigate the carbon footprint associated with AI advancements.

Traditional AI training methods typically focus on individual data points, which can be time-consuming and computationally expensive. JEST innovates by shifting the focus to entire batches of data. Heres a simplified breakdown of the JEST process:

By utilizing a smaller model to filter and select high-quality data, the larger model can be trained more effectively, leading to significant performance improvements.

JESTs efficiency stems from its ability to evaluate batches of data rather than individual examples. This method leverages multimodal contrastive learning, which looks at how different types of data (like text and images) interact with each other. By scoring entire batches and selecting the most learnable subsets, JEST accelerates the training process.

The method can be broken down into two main components:

DeepMinds experiments with JEST have shown remarkable results. The method achieves state-of-the-art performance with significantly fewer training iterations and lower computational costs. For instance, JEST matches the performance of existing models with up to 13 times fewer training iterations and ten times less energy consumption.

These improvements are not just incrementalthey represent a substantial leap forward in making AI training more sustainable and scalable. By reducing the energy required for training, JEST not only cuts costs but also helps address the pressing issue of AIs environmental impact. According to an analysis by the Electric Power Research Institute,data centers could consume between 4.6% and 9.1% of US electricity by 2030.

However, the researchers note some limitations of their approach. For example, JEST still relies on having access to smaller, well-curated datasets to guide the selection process. Developing methods to automatically infer optimal reference distributions remains an open challenge.

Nevertheless, the dramatic efficiency improvements demonstrated by JEST point to significant headroom for optimizing AI training. As models grow ever larger and more energy-intensive, such innovations will likely prove crucial for sustainable scaling of artificial intelligence capabilities.

Chris McKay is the founder and chief editor of Maginative. His thought leadership in AI literacy and strategic AI adoption has been recognized by top academic institutions, media, and global brands.

The rest is here:
Google DeepMind Unveils JEST, a New AI Training Method That Slashes Energy Use - Maginative

Rats are incredibly nimble creatures. They can climb up curtains, jump down tall ledges, and scurry across complex terrainsay, your basement stacked with odd-shaped stuffat mind-blowing speed.

Robots, in contrast, are anything but nimble. Despite recent advances in AI to guide their movements, robots remain stiff and clumsy, especially when navigating new environments.

To make robots more agile, why not control them with algorithms distilled from biological brains? Our movements are rooted in the physical world and based on experiencetwo components that let us easily explore different surroundings.

Theres one major obstacle. Despite decades of research, neuroscientists havent yet pinpointed how brain circuits control and coordinate movement. Most studies have correlated neural activity with measurable motor responsessay, a twitch of a hand or the speed of lifting a leg. In other words, we know brain activation patterns that can describe a movement. But which neural circuits cause those movements in the first place?

We may find the answer by trying to recreate them in digital form. As the famous physicist Richard Feynman once said, What I cannot create, I do not understand.

This month, Google DeepMind and Harvard University built a realistic virtual rat to home in on the neural circuits that control complex movement. The rats digital brain, composed of artificial neural networks, was trained on tens of hours of neural recordings from actual rats running around in an open arena.

Comparing activation patterns of the artificial brain to signals from living, breathing animals, the team found the digital brain could predict the neural activation patterns of real rats and produce the same behaviorfor example, running or rearing up on hind legs.

The collaboration was fantastic, said study author Dr. Bence Olveczky at Harvard in a press release. DeepMind had developed a pipeline to train biomechanical agents to move around complex environments. We simply didnt have the resources to run simulations like those, to train these networks.

The virtual rats brain recapitulated two regions especially important for movement. Tweaking connections in those areas changed motor responses across a variety of behaviors, suggesting these neural signals are involved in walking, running, climbing, and other movements.

Virtual animals trained to behave like their real counterparts could provide a platform for virtual neurosciencethat would otherwise be difficult or impossible to experimentally deduce, the team wrote in their article.

Artificial intelligence lives in the digital world. To power robots, it needs to understand the physical world.

One way to teach it about the world is to record neural signals from rodents and use the recordings to engineer algorithms that can control biomechanically realistic models replicating natural behaviors. The goal is to distill the brains computations into algorithms that can pilot robots and also give neuroscientists a deeper understanding of the brains workings.

So far, the strategy has been successfully used to decipher the brains computations for vision, smell, navigation, and recognizing faces, the authors explained in their paper. However, modeling movement has been a challenge. Individuals move differently, and noise from brain recordings can easily mess up the resulting AIs precision.

This study tackled the challenges head on with a cornucopia of data.

The team first placed multiple rats into a six-camera arena to capture their movementrunning around, rearing up, or spinning in circles. Rats can be lazy bums. To encourage them to move, the team dangled Cheerios across the arena.

As the rats explored the arena, the team recorded 607 hours of video and also neural activity with a 128-channel array of electrodes implanted in their brains.

They used this data to train an artificial neural networka virtual rats brainto control body movement. To do this, they first tracked how 23 joints moved in the videos and transferred them to a simulation of the rats skeletal movements. Our joints only bend in certain ways, and this step filters out whats physically impossible (say, bending legs in the opposite direction).

The core of the virtual rats brain is a type of AI algorithm called an inverse dynamics model. Basically, it knows where body positions are in space at any given time and, from there, predicts the next movements leading to a goalsay, grab that coffee cup without dropping it.

Through trial-and-error, the AI eventually came close to matching the movements of its biological counterparts. Surprisingly, the virtual rat could also easily generalize motor skills to unfamiliar places and scenariosin part by learning the forces needed to navigate the new environments.

The similarities allowed the team to compare real rats to their digital doppelgangers, when performing the same behavior.

In one test, the team analyzed activity in two brain regions known to guide motor skills. Compared to an older computational model used to decode brain networks, the AI could better simulate neural signals in the virtual rat across multiple physical tasks.

Because of this, the virtual rat offers a way to study movement digitally.

One long-standing question, for example, is how the brain and nerves command muscle movement depending on the task. Grabbing a cup of coffee in the morning, for example, requires a steady hand without any jerking action but enough strength to hold it steady.

The team tweaked the neural connections in the virtual rodent to see how changes in brain networks alter the final behaviorgetting that cup of coffee. They found one network measure that could identify a behavior at any given time and guide it through.

Compared to lab studies, these insights can only be directly accessed through simulation, wrote the team.

The virtual rat bridges AI and neuroscience. The AI models here recreate the physicality and neural signals of living creatures, making them invaluable for probing brain functions. In this study, one aspect of the virtual rats motor skills relied on two brain regionspinpointing them as potential regions key to guiding complex, adaptable movement.

A similar strategy could provide more insight into the computations underlying vision, sensation, or perhaps even higher cognitive functions such as reasoning. But the virtual rat brain isnt a complete replication of a real one. It only captures snapshots of part of the brain. But it does let neuroscientists zoom in on their favorite brain region and test hypotheses quickly and easily compared to traditional lab experiments, which often take weeks to months.

On the robotics side, the method adds a physicality to AI.

Weve learned a huge amount from the challenge of building embodied agents: AI systems that not only have to think intelligently, but also have to translate that thinking into physical action in a complex environment, said study author Dr. Matthew Botvinick at DeepMind in a press release. It seemed plausible that taking this same approach in a neuroscience context might be useful for providing insights in both behavior and brain function.

The team is next planning to test the virtual rat with more complex tasks, alongside its biological counterparts, to further peek inside the inner workings of the digital brain.

From our experiments, we have a lot of ideas about how such tasks are solved, said lveczky to The Harvard Gazette. We want to start using the virtual rats to test these ideas and help advance our understanding of how real brains generate complex behavior.

Image Credit: Google DeepMind

Continue reading here:
Google DeepMind's AI Rat Brains Could Make Robots Scurry Like the Real Thing - Singularity Hub

In 2021, AI research lab DeepMind announced the development of its first digital biology neural network, AlphaFold. The model was capable of accurately predicting the 3D structure of proteins, which determines the functions that these molecules play. Were just floating bags of water moving around, says Pushmeet Kohli, VP of research at DeepMind. What makes us special are proteins, the building blocks of life. How they interact with each other is what makes the magic of life happen.

AlphaFold was considered by the journal Science as the breakthrough of the year in 2021. In 2022, it was the most cited research paper in AI. People have been on [protein structures] for many decades and were not able to make that much progress, Kohli says. Then came AI. DeepMind also released the AlphaFold Protein Structure Databasewhich contained the protein structures of almost every organism whose genome has been sequencedmaking it freely available to scientists worldwide.

More than 1.7 million researchers in 190 countries have used it for research ranging from the design of plastic-eating enzymes to the development of more effective malaria vaccines. A quarter of the research involving AlphaFold was dedicated to the understanding of cancer, Covid-19, and neurodegenerative diseases like Parkinsons and Alzheimers. Last year, DeepMind released its next generation of AlphaFold, which extended its structure prediction algorithm to biomolecules like nucleic acids and ligands.

It has democratized scientific research, Kohli says. Scientists working in a developing country on a neglected tropical disease did not have access to the funds to get the structure of a protein computed. Now, at the click of a button, they can go to the AlphaFold database and get these predictions for free. For instance, one of DeepMinds early partners, the Drugs for Neglected Diseases Initiative, used AlphaFold to develop medicine for diseases that affect millionssuch as sleeping sickness, Chagas disease, and leishmaniasisyet receive comparatively little research.

DeepMinds latest breakthrough is called AlphaMissense. The model categorizes the so-called missense mutationsgenetic alterations that can result in different amino acids being produced at particular positions in proteins. Such mutations can alter the function of the protein itself, and AlphaMissense attributes a likelihood score for that mutation being either pathogenic or benign. Understanding and predicting those effects is crucial for the discovery of rare genetic diseases, Kohli says. The algorithm, which was released last year, has classified around 89 percent of all possible human missense. Before, only 0.1 percent of all possible variants had been clinically classified by researchers.

This is just the beginning, Kohli says. Ultimately, he believes AI could eventually lead to the creation of a virtual cell that could radically accelerate biomedical research, enabling biology to be explored in-silico rather than in real-world laboratories. With AI and machine learning we finally have the tools to comprehend this very sophisticated system that we call life.

This article appears in the July/August 2024 issue of WIRED UK magazine.

See the original post here:
With AI Tools, Scientists Can Crack the Code of Life - WIRED