Dataset

The data used for the present analysis are: the solar wind (SW) plasma parameters; the interplanetary magnetic field (IMF); the Dst index. The entire dataset has been obtained from the National Space Science Data Center of NASA, namely, from the OMNI database30. In particular, we used hourly averages of the three components ((B_x), (B_y), (B_z)) of the IMF in the GSM (Geocentric Solar Magnetospheric) reference frame (i.e. the x-axis of the GSM coordinate system is defined along the line connecting the center of the Sun to the center of the Earth; the origin is defined at the center of the Earth and is positive towards the Sun; the y-axis is defined as the cross product of the GSM x-axis and the magnetic dipole axis and is positive towards dusk; The z-axis is defined as the cross-product of the x- and y-axes; the magnetic dipole axis lies within the xz plane), the SW plasma temperature (T), density (D), total speed (V), pressure (P), and eastwest component of the electric field ((E_y) derived from (B_z) and (V_x)).

The dataset covers the period January 1990November 2019, and includes half of the 22nd solar cycle, all of the 23rd, and almost all of the 24th. To produce a robust forecasting of the Dst index, it is crucial to determine how the dataset is split and processed for the training and evaluation of the model. On the other hand, adopting a correct methodology for treating data is crucial to avoid bias especially when both a machine learning approach is used to develop predictive models and the data are time series.

If data are periodic, it is safe to train the model considering at least one complete period and test it on different periods. In fact, being the arrow of time fixed and the future unknown, the training operation that make use of points that follow the data used in the test can introduce bias. Therefore, the validation and test data-sets must be constructed by points of the time series that follow what is used for training one. In the present case, since we have only data from two solar cycles, the best option is to use one cycle for training and the other for both validation and test. Anyway, such a choice forces the validation to contain data relative to the first half of a solar cycle with a distribution of Dst values and storms different from the test set. Therefore, in our opinion, the most efficient choice for the validation and test process is to select points randomly for the two datasets.

Training a supervised fashion Deep Learning (DL) model requires both a balanced sampling of data referring to quiet and storm periods, and a proper evaluation of the metrics used to measure the performances. If not, the model will learn to predict only the most frequent case represented in the training set. Moreover, the standard performance metrics, computed on the full validation and test dataset, would produce a the prediction that would be correct most of the time but wrong in most relevant cases.

Taking care of these two aspects, we split the dataset using all the data before 1/1/2009 for training, and the remaining part for validation and test. In this way, we have at least one solar cycle for the training and one for the evaluation of the model. As previously said, for the validation and test we can choose dataset subsequent in time (i.e. ordered) or an equal number of points randomly from those available after 1/1/2009. The difference between random and ordered selection are displayed in Fig.1. In panel a the validation data includes the points in the first half of the cycle while the test is the other half. It is evident that the tail of the two distributions is different: in the validation dataset, events with very low Dst, which are particularly important being connected with storms, are missing. The situation completely changes when the points are picked randomly. In this case, the distributions are quite similar and also similar to the training dataset, representing the best starting point for the development of a data-driven predictive model. The last problem, directly connected to the data distribution, is that there are only few events associated with storms. In the framework used in this paper, where the algorithm learns by looking at the data, if the distribution is highly peaked around some value of the target variable, the algorithm will learn to predict only such values. To avoid this issue, we apply a re-weighting function for the sampling of the data that feed the algorithms training. In this way, every value of Dst is almost equally probable. The difference between the nominal distribution and the flatten (weighted) distribution is presented in Fig.1c.

Normalized distributions of Dst in the dataset used for training, validation and test. (a) Validation is the first half of the solar cycle period, test the second half. (b) Points for validation and test are randomly extracted. Train dataset includes all the available points before 1/1/2009. (c) Train dataset without and with re-weighting the low Dst events.

The points discussed above limit also the applicability of standard cross-validation methods usually recommended in machine learning applications to test the robustness of the models. While specific schemes of cross-validations have been developed for time series (e.g., the TimeSeriesSplit function available in the Scikit Python library), we prefer not to adopt this type of check because this kind of split increases the size of the training dataset, namely: in the first iterations, there are much fewer storms than in the latest. This automatically will favor the last iterations of the procedure in predicting storms, introducing an indirect bias in the interpretation of the results.

All the features are scaled linearly on a compact range as an additional pre-processing step. The scale is fitted on the training dataset, mapping these min and max values of data in 0.1 and 0.9, respectively. This choice leaves some room to accommodate smaller or larger values than those available in the training dataset that can emerge in future measurements of the variables.

The architecture of the Neural Network considered in this study is close to the one used in26 where a Long Short-Term Memory (LSTM) module is combined with a Fully Connected Network (FCNN). LSTM is a recurrent layer composed of cells designed to process long time series. The input of the proposed network is time series containing the variables described in Dataset for the 12 points in the time window ([t-11, t]). Each cell of the LSTM layer (Fig.2) receives in input one element (x_{t_i}) of this time series together with the outputs of the previous cells: the hidden state, (h_{t_{i-1}}), and the memory state, (c_{t_{i-1}}). As schematically depicted in the figure, these three sources of information are processed through fully connected layers and element-wise operations, all internal to the cells. In standard application of LSTM, the hidden state from the last cell represents the networks prediction, and the hidden states of all the other cells are not considered. In our approach, we collect and concatenate all the hidden states ([h_{t-11}, h_t]) in a multidimensional vector. This vector is then fed as input of a fully connected module. The output of this FCNN is the forecast of the Dst index for the hours ([t+1, t+12]).

Neural Network architecture used to forecast the Dst index as described in the text. In the LSTM cell, the square blocks are Fully Connected layers with activation function, while the circles are elementwise operations.

In optimizing DL networks, two types of parameters need to be fixed: the layers weights and the hyper-parameters specifying the architecture. During training, the back-propagation procedure takes care of the former, which can be millions or even billions (in our case 25,244). The others, typically limited in number (in our case 7), are usually determined manually by testing different solutions and considering only the training and validation dataset in the evaluation to avoid bias.

We found that better predictions are obtained using the following values for the hyper parameters:

LSTM, number of hidden layers: 2,

LSTM, size of the hidden layers: 8,

FCNN, number of layers: 4,

FCNN, number of output features for each layer: 96, 96, 48, 12.

Batch normalization is applied to the input vector of the FCNN, ReLU activation function, and a dropout layer with a drop factor of 0.2 follows every fully connected layer except the last one.

The loss function minimized during the training of the network is the Mean Absolute Error (MAE) function

$$begin{aligned} {text {MAE}} = frac{1}{N}sum _{i=1}^{N}left| y_{pred} - y_{true}right| _i end{aligned}$$

(1)

We use the Adam optimizer and a learning rate of (10^{-5}). During the training, back-propagation is applied after computing the loss on samples extracted from the dataset in batches. The procedure is repeated an arbitrary number of times. Statistics are collected after iterating back-propagation on as many samples as the number of elements in the training dataset: this is called an epoch. The training ends once the loss function stops decreasing on the validation dataset. We used batches of size 256 and stopped training after 10,000 epochs. Examples of the loss function behaviors are presented in Fig.3.

History of the loss function in the 10,000 epochs of the training.

The code with the implementation of the network architecture and the procedure to generate the training, validation, and test datasets are available as a Python notebook in the public GitLab repository gitlab.fbk.eu/dsip/dsip_physics/dsip_ph_space/Dstw.

A typical baseline forecast method for time series is the persistent model. The assumption at the base of this approach is that nothing changes between the last known value and all the future points:

$$begin{aligned} Dst(t + n) = Dst(t),quad nin mathbb {N}. end{aligned}$$

(2)

It is expected that the predictive power of this model will decrease with the increase of the forecast horizon; on the contrary, in the short term, assuming persistence is often a good approximation of the actual trend.

Different metrics can be considered to highlight and study models features and compare their predictive power. However, the focus of this work is the importance of how the training data are selected and used. This is appreciable even considering only the most common of these metrics, the Root Mean Squared Error (RMSE), defined as:

$$begin{aligned} text {RMSE}=sqrt{frac{sum _{i=1}^N left( y_{pred_i}-y_{true}right) ^2}{N}}. end{aligned}$$

(3)

Read the original here:
Prominence of the training data preparation in geomagnetic storm prediction using deep neural networks | Scientific Reports - Nature.com

Beneath our feet is an ecosystem so astonishing that it tests the limits of our imagination. Its as diverse as a rainforest or a coral reef. We depend on it for 99% of our food, yet we scarcely know it. Soil.

Under one square metre of undisturbed ground in the Earths mid-latitudes (which include the UK) there might live several hundred thousand small animals. Roughly 90% of the species to which they belong have yet to be named. One gram of this soil less than a teaspoonful contains around a kilometre of fungal filaments.

When I first examined a lump of soil with a powerful lens, I could scarcely believe what I was seeing. As soon as I found the focal length, it burst into life. I immediately saw springtails tiny animals similar to insects in dozens of shapes and sizes. Round, crabby mites were everywhere: in some soils there are half a million in every square metre.

Then I began to see creatures I had never encountered before. What I took to be a tiny white centipede turned out, when I looked it up, to be a different life form altogether, called a symphylid. I spotted something that might have stepped out of a Japanese anime: long and low, with two fine antennae at the front and two at the back, poised and sprung like a virile dragon or a flying horse. It was a bristletail, or dipluran.

As I worked my way through the lump, again and again I found animals whose existence, despite my degree in zoology and a lifetime immersed in natural history, had been unknown to me. After two hours examining a kilogram of soil, I realised I had seen more of the major branches of the animal kingdom than I would on a weeks safari in the Serengeti.

But even more arresting than soils diversity and abundance is the question of what it actually is. Most people see it as a dull mass of ground-up rock and dead plants. But it turns out to be a biological structure, built by living creatures to secure their survival, like a wasps nest or a beaver dam. Microbes make cements out of carbon, with which they stick mineral particles together, creating pores and passages through which water, oxygen and nutrients pass. The tiny clumps they build become the blocks the animals in the soil use to construct bigger labyrinths.

Soil is fractally scaled, which means its structure is consistent, regardless of magnification. Bacteria, fungi, plants and soil animals, working unconsciously together, build an immeasurably intricate, endlessly ramifying architecture that, like Dust in a Philip Pullman novel, organises itself spontaneously into coherent worlds. This biological structure helps to explain soils resistance to droughts and floods: if it were just a heap of matter, it would be swept away.

It also reveals why soil can break down so quickly when its farmed. Under certain conditions, when farmers apply nitrogen fertiliser, the microbes respond by burning through the carbon: in other words, the cement that holds their catacombs together. The pores cave in. The passages collapse. The soil becomes sodden, airless and compacted.

But none of the above captures the true wonder of soil. Lets start with something that flips our understanding of how we survive. Plants release into the soil between 11% and 40% of all the sugars they make through photosynthesis. They dont leak them accidentally. They deliberately pump them into the ground. Stranger still, before releasing them, they turn some of these sugars into compounds of tremendous complexity.

Making such chemicals requires energy and resources, so this looks like pouring money down the drain. Why do they do it? The answer unlocks the gate to a secret garden.

These complex chemicals are pumped into the zone immediately surrounding the plants roots, which is called the rhizosphere. They are released to create and manage its relationships.

Soil is full of bacteria. Its earthy scent is the smell of the compounds they produce. In most corners, most of the time, they wait, in suspended animation, for the messages that will wake them. These messages are the chemicals the plant releases. They are so complex because the plant seeks not to alert bacteria in general, but the particular bacteria that promote its growth. Plants use a sophisticated chemical language that only the microbes to whom they wish to speak can understand.

When a plant root pushes into a lump of soil and starts releasing its messages, it triggers an explosion of activity. The bacteria responding to its call consume the sugars the plant feeds them and proliferate to form some of the densest microbial communities on Earth. There can be a billion bacteria in a single gram of the rhizosphere; they unlock the nutrients on which the plant depends and produce growth hormones and other chemicals that help it grow. The plants vocabulary changes from place to place and time to time, depending on what it needs. If its starved of certain nutrients, or the soil is too dry or salty, it calls out to the bacteria species that can help.

Take a step back and you will see something that transforms our understanding of life on Earth. The rhizosphere lies outside the plant, but it functions as if it were part of the whole. It could be seen as the plants external gut. The similarities between the rhizosphere and the human gut, where bacteria also live in astonishing numbers, are uncanny. In both systems, microbes break down organic material into the simpler compounds the plant or person can absorb. Though there are more than 1,000 phyla (major groups) of bacteria, the same four dominate both the rhizosphere and the guts of mammals.

Just as human breast milk contains sugars called oligosaccharides, whose purpose is to feed not the baby but the bacteria in the babys gut, young plants release large quantities of sucrose into the soil, to feed and develop their new microbiomes. Just as the bacteria that live in our guts outcompete and attack invading pathogens, the friendly microbes in the rhizosphere create a defensive ring around the root. Just as bacteria in the colon educate our immune cells and send chemical messages that trigger our bodys defensive systems, the plants immune system is trained and primed by bacteria in the rhizosphere.

Soil might not be as beautiful to the eye as a rainforest or a coral reef, but once you begin to understand it, it is as beautiful to the mind. Upon this understanding our survival might hang.

We face what could be the greatest predicament humankind has ever encountered: feeding the world without devouring the planet. Already, farming is the worlds greatest cause of habitat destruction, the greatest cause of the global loss of wildlife and the greatest cause of the global extinction crisis. Its responsible for about 80% of the deforestation thats happened this century. Of 28,000 species known to be at imminent risk of extinction, 24,000 are threatened by farming. Only 29% of the weight of birds on Earth consists of wild species: the rest is poultry. Just 4% of the worlds mammals, by weight, are wild; humans account for 36%, and livestock for the remaining 60%.

Unless something changes, all this is likely to get worse much worse. In principle, there is plenty of food, even for a rising population. But roughly half the calories farmers grow are now fed to livestock, and the demand for animal products is rising fast. Without a radical change in the way we eat, by 2050 the world will need to grow around 50% more grain. How could we do it without wiping out much of the rest of life on Earth?

Just as farming is trashing crucial Earth systems, their destruction threatens our food supply. Sustaining even current levels of production might prove impossible. Climate breakdown is likely, on the whole, to make wet places wetter and dry places drier. One more degree of heating, one estimate suggests, would parch 32% of the worlds land surface. By the middle of this century, severe droughts could simultaneously affect an arc from Portugal to Pakistan. And this is before we consider the rising economic fragility of the global food system, or geopolitical pressures, such as the current war in Ukraine, that might threaten 30% of the worlds wheat exports.

Its not just the quantity of production thats at risk, but also its quality. A combination of higher temperatures and higher concentrations of CO2 reduces the level of minerals, protein and B vitamins that crops contain. Already, zinc deficiency alone afflicts more than a billion people. Though we seldom discuss it, one paper describes the falling concentrations of nutrients as existential threats.

Some crop scientists believe we can counter these trends by raising yields in places that remain productive. But their hopes rely on unrealistic assumptions. The most important of these is sufficient water. The anticipated growth in crop yields would require 146% more fresh water than is used today. Just one problem: that water doesnt exist.

Over the past 100 years, our use of water has increased six-fold. Irrigating crops consumes 70% of the water we withdraw from rivers, lakes and aquifers. Already, 4 billion people suffer from water scarcity for at least one month a year and 33 major cities, including So Paulo, Cape Town, Los Angeles and Chennai, are threatened by extreme water stress. As groundwater is depleted, farmers have begun to rely more heavily on meltwater from glaciers and snowpacks. But these, too, are shrinking.

A likely flashpoint is the valley of the Indus, whose water is used by three nuclear powers (India, Pakistan and China) and several unstable regions. Already, 95% of the rivers flow is extracted. As the economy and the population grow, by 2025 demand for water in the catchment is expected to be 44% greater than supply. But one of the reasons why farming there has been able to intensify and cities to grow is that, as a result of global heating, glaciers in the Hindu Kush and the Himalayas have been melting faster than theyve been accumulating, so more water has been flowing down the rivers. This cant last. By the end of the century, between one- and two-thirds of the ice mass is likely to have disappeared. It is hard to see this ending well.

And all this is before we come to the soil, the thin cushion between rock and air on which human life depends, which we treat like dirt. While there are international treaties on telecommunication, civil aviation, investment guarantees, intellectual property, psychotropic substances and doping in sport, there is no global treaty on soil. The notion that this complex and scarcely understood system can withstand all we throw at it and continue to support us could be the most dangerous of all our beliefs.

Soil degradation is bad enough in rich nations, where the ground is often left bare and exposed to winter rain, compacted and wrecked by overfertilisation and pesticides that rip through its foodwebs. But it tends to be even worse in poorer nations, partly because extreme rainfall, cyclones and hurricanes can tear bare earth from the land, and partly because hungry people are often driven to cultivate steep slopes. In some countries, mostly in Central America, tropical Africa and south-east Asia, more than 70% of the arable land is now suffering severe erosion, gravely threatening future production.

Climate breakdown, which will cause more intense droughts and storms, exacerbates the threat. The loss of a soils resilience can happen incrementally and subtly. We might scarcely detect it until a shock pushes the complex underground system past its tipping point. When severe drought strikes, the erosion rate of degraded soil can rise 6,000-fold. In other words, the soil collapses. Fertile lands turn to dustbowls.

Some people have responded to these threats by calling for the relocalisation and de-intensification of farming. I understand their concerns. But their vision is mathematically impossible.

Sign up to our Inside Saturday newsletter for an exclusive behind-the-scenes look at the making of the magazines biggest features, as well as a curated list of our weekly highlights.

A study in the journal Nature Food found the average minimum distance at which the worlds people can be fed is 2,200km. In other words, this is the shortest possible average journey that our food must travel if we are not to starve. For those who depend on wheat and similar cereals, its 3,800km. A quarter of the global population that consumes these crops needs food grown at least 5,200km away.

Why? Because most of the worlds people live in big cities or populous valleys, whose hinterland is too small (and often too dry, hot or cold) to feed them. Much of the worlds food has to be grown in vast, lightly habited lands the Canadian prairies, the US plains, wide tracts in Russia and Ukraine, the Brazilian interior and shipped to tight, densely populated places.

As for reducing the intensity of farming, what this means is using more land to produce the same amount of food. Land use is arguably the most important of all environmental issues. The more land farming occupies, the less is available for forests and wetlands, savannahs and wild grasslands, and the greater is the loss of wildlife and the rate of extinction. All farming, however kind and careful, involves a radical simplification of natural ecosystems.

Environmental campaigners rail against urban sprawl: the profligate use of land for housing and infrastructure. But agricultural sprawl using large amounts of land to produce small amounts of food has transformed much greater areas. While 1% of the worlds land is used for buildings and infrastructure, crops occupy 12% and grazing, the most extensive kind of farming, uses 28%. Only 15% of land, by contrast, is protected for nature. Yet the meat and milk from animals that rely solely on grazing provide just 1% of the worlds protein.

One paper looked at what would happen if everyone in the US followed the advice of celebrity chefs and switched from grain-fed to pasture-fed beef. It found that, because they grow more slowly on grass, the number of cattle would have to rise by 30%, while the land area used to feed them would rise by 270%. Even if the US felled all its forests, drained its wetlands, watered its deserts and annulled its national parks, it would still need to import most of its beef.

Already, much of the beef the US buys comes from Brazil, which in 2018 became the worlds largest exporter. This meat is often promoted as pasture-fed. Many of the pastures were created by illegally clearing the rainforest. Worldwide, meat production could destroy 3m sq km of highly biodiverse places in 35 years. Thats almost the size of India.

Only when livestock are extremely sparse is animal farming compatible with rich, functional ecosystems. For example, the Knepp Wildland project in West Sussex, where small herds of cattle and pigs roam freely across a large estate, is often cited as a way to reconcile meat and wildlife. But while its an excellent example of rewilding, its a terrible example of food production.

If this system were to be rolled out across 10% of the UKs farmland and if, as its champions propose, we obtained our meat this way, it would furnish each person here with 420 grams of meat a year, enough for around three meals. We could eat a prime steak roughly once every three years. If all the farmland in the UK were to be managed this way, it would provide us with 75kcal a day (one 30th of our requirement) in meat, and nothing else.

Of course, this is not how it would be distributed. The very rich would eat meat every week, other people not at all. Those who say we should buy only meat like this, who often use the slogan less and better, present an exclusive product as if it were available to everyone.

Campaigners, chefs and food writers rail against intensive farming and the harm it does to us and the world. But the problem is not the adjective: its the noun. The destruction of Earth systems is caused not by intensive farming or extensive farming, but a disastrous combination of the two.

So what can we do? Part of the answer is to take as much food production out of farming as we can. As luck would have it, the enabling technology has arrived just as we need it. Precision fermentation, producing protein and fat in breweries from soil bacteria, fed on water, hydrogen, CO2 and minerals, has the potential to replace all livestock farming, all soya farming and plenty of vegetable oil production, while massively reducing land use and other environmental impacts.

But this remarkable good fortune is threatened by intellectual property rights: it could easily be captured by the same corporations that now monopolise the global grain and meat trade. We should fiercely resist this: patents should be weak and anti-trust laws strong. Ideally, this farm-free food should be open source.

Then we could relocalise production: the new fermentation technologies could be used by local businesses to serve local markets. As some of the worlds poorest nations are rich in sunlight, they could make good use of a technology that relies on green hydrogen. Microbial production horrifies some of those who demand food sovereignty and food justice. But it could deliver both more effectively than farming does.

Such technologies grant us, for the first time since the Neolithic period, the opportunity to transform not only our food system but our entire relationship with the living world. Vast tracts of land can be released from both intensive and extensive farming. The age of extinction could be replaced by an age of regenesis.

Of course, we would still need to produce cereals, roots, fruit and vegetables. So how do we do it safely and productively? The answer might lie in our new understanding of the soil.

On a farm in south Oxfordshire, techniques developed by a vegetable grower called Iain Tolhurst Tolly seem to have anticipated recent discoveries by soil scientists.

Tolly is a big, tough-looking man in his late 60s, with etched and weathered skin, a broad, heavy jaw, long blond hair, one gold earring, hands grained with earth and oil. He started farming without training or instruction, without land or any means to buy it. After a string of misadventures, he managed to lease seven hectares (17.3 acres) of very poor land at a reduced rent, 34 years ago.

No conventional grower would even look at this ground, he told me. Its 40% stone. Theyd call it building rubble. It isnt even classed as arable: an agronomist would say its only good for grass or trees. But over the past 12 months, we harvested 120 tonnes of vegetables and fruit.

Astonishingly, for these 34 years Tolly has been farming this rubble without pesticides, herbicides, mineral treatments, animal manure or any other kind of fertiliser. He has pioneered a way of growing that he calls stockfree organic. This means he uses no livestock or livestock products at any point in the farming cycle, yet he also uses no artificial inputs.

Until he proved the model, this was thought to be a formula for sucking the fertility out of the land. Vegetables in particular are considered hungry crops, which require plenty of extra nutrients to grow. Yet Tolly, while adding none, has raised his yields until theyve hit the lower bound of what intensive growers achieve with artificial fertilisers on good land: a feat widely considered impossible. Remarkably, the fertility of his soil has climbed steadily.

On my first visit, one June, I was struck by the great range and health of Tollys crops. One plot was a blue haze of onion plants, another a patchwork of sea greens: young cauliflower plants, several kinds of cabbage and kale. There were rows of rainbow chard with gold, green, white and crimson stems. Broad bean pods had begun to sprout from tight pillars of flower. His potatoes were in full bloom, nightshade sinister, stamens like yellow stings. Courgettes extruded rudely behind their trumpet flowers. There were carrots, tomatoes, peppers, beans of all kinds, herbs, parsnips, celeriac, cucumbers, lettuces. He raises 100 varieties of vegetables, which he sells in his farm shop and to subscribers to his veg box.

Separating the plots were untended banks, in which scientists studying his farm have found 75 species of wildflowers. These banks are an essential component of his system, harbouring the insect predators that control crop pests. Though he uses no pesticides, none of the vegetable plants I saw showed signs of significant insect damage: the leaves were dark and wide, with scarcely a hole or a spot.

Almost single-handedly, through trial and error, Tolly has developed a new and revolutionary model of horticulture. At first it looks like magic. In reality, its the result of many years of meticulous experiments.

Two of his innovations appear to be crucial. The first, as he puts it, is to make the system watertight: preventing rain from washing through the soil, taking the nutrients with it. What this means is ensuring the land is almost never left bare. Beneath his vegetables grows an understorey of green manure, plants that cover the soil. Under the leaves of his pumpkins, I could see thousands of tiny seedlings: the weeds he had deliberately sown. When the crops are harvested, the green manure fills the gap and soon becomes a thicket of colour: blue chicory flowers, crimson clover, yellow melilot and trefoil, mauve Phacelia, pink sainfoin.

Theres green manure under the green manure, Tolly told me. As soon as we cut the bigger plants, it comes into flower, and the bees go crazy.

Some of the plants in his mix put down deep roots that draw nutrients from the subsoil. Every so often, Tolly runs a mower over them, chopping them into a coarse straw. Earthworms pull this down and incorporate it into the ground. The idea is to let the plants put back at least as much carbon and minerals as we take out.

Tolly tells me that the green manure ties up nutrients, fixes nitrogen, adds carbon and enhances the diversity of the soil. The more plant species you sow, the more bacteria and fungi you encourage. Every plant has its own associations. Roots are the glue that holds and builds the soil biology.

The other crucial innovation is to scatter over the green manure an average of one millimetre a year of chipped and composted wood, produced from his own trees or delivered by a local tree surgeon. This tiny amendment appears to make a massive difference. In the five years after he started adding woodchip, his yields roughly doubled. As Tolly explains: It isnt fertiliser; its an inoculant that stimulates microbes. The carbon in the wood encourages the bacteria and fungi that bring the soil back to life. Tolly believes hes adding enough carbon to help the microbes build the soil, but not so much that they lock up nitrogen, which is what happens if you give them more than they need.

What Tolly appears to be doing is strengthening and diversifying the relationships in the rhizosphere the plants external gut. By keeping roots in the soil, raising the number of plant species and adding just the right amount of carbon, he seems to have encouraged bacteria to build their catacombs in his stony ground, improving the soils structure and helping his plants to grow.

Tollys success forces us to consider what fertility means. Its not just about the amount of nutrients the soil contains. Its also a function of whether theyre available to plants at the right moments, and safely immobilised when plants dont need them. In a healthy soil, crops can regulate their relationships with bacteria in the rhizosphere, ensuring that nutrients are unlocked only when theyre required. In other words, fertility is a property of a functioning ecosystem. Farm science has devoted plenty of attention to soil chemistry. But the more we understand, the more important the biology appears to be.

Can Tollys system be replicated? So far the results are inconclusive. But if we can discover how to mediate and enhance the relationship between crop plants and bacteria and fungi in a wide range of soils and climates, it should be possible to raise yields while reducing inputs. Our growing understanding of soil ecology could catalyse a greener revolution.

I believe we could combine this approach with another suite of innovations, by a non-profit organisation in Salina, Kansas, called the Land Institute. Its seeking to develop perennial grain crops to replace the annual plants from which we obtain the great majority of our food. Annuals are plants that die after a single growing season. Perennials survive from one year to the next.

Large areas dominated by annuals are rare in nature. They tend to colonise ground in the wake of catastrophe: a fire, flood, landslide or volcanic eruption that exposes bare rock or soil. In cultivating annuals, we must keep the land in a catastrophic state. If we grew perennial grain crops, we would be less reliant on smashing living systems apart to produce our food.

For 40 years, the Land Institute has been scouring the world for perennial species that could replace the annuals we grow. Already, working with Fengyi Hu and his team at Yunnan University in China, it has developed a perennial rice with yields that match, and in some cases exceed, those of modern annual breeds. Farmers are queueing up for seed. While annual rice farming can cause devastating erosion, the long roots of the perennial varieties bind and protect the soil. Some perennial rice crops have now been harvested six times without replanting.

Perennials are their own green manures. The longer they grow, the stronger their relationships with microbes that fix nitrogen from the air and release other minerals. One estimate suggests that perennial systems hold five times as much of the water that falls on the ground as annual crops do.

The Land Institute is developing promising lines of perennial wheat, oil crops and other grains. The deep roots and tough structures of perennial plants could help them to withstand climate chaos. The perennial sunflowers the institute is breeding have sailed through two severe droughts, one of which entirely destroyed the annual sunflowers grown alongside them.

While no solution is a panacea, I believe that some of the components of a new global food system one that is more resilient, more distributed, more diverse and more sustainable are falling into place. If it happens, it will be built on our new knowledge of the most neglected of major ecosystems: the soil. It could resolve the greatest of all dilemmas: how to feed ourselves without destroying the living systems on which we depend. The future is underground.

George Monbiot will discuss Regenesis at a Guardian Live event in London on Monday 30 May. Book tickets to join the event in person, or via the livestream here.

Regenesis: Feeding the World Without Devouring the Planet by George Monbiot is published by Penguin Books at 20 on 26 May. To support the Guardian and Observer, order your copy at guardianbookshop.com. Delivery charges may apply.

See original here:
The secret world beneath our feet is mind-blowing and the key to our planet's future - The Guardian

Its a world of haughty actors and totally ripped actresses, decked out galas and stressed-out investors. Welcome to the world of dreams and madness. Welcome to Nyallywood, the setting of GKIDS latest cinematic effort: Pompo the Cinephile.

I do think it has a universal appeal - or, at least, I hope so, says writer and director Takayuki Hirao, previously known for his directorial work on anime series like God Eater and films like Majokko shimai no Yoyo to Nene. It's really important to be chasing your dreams. In this world, there are so many people who don't fit in or think that they're in the minority, or they're suffering because they're in the minority of dreamers. If I can, through this movie, guide them to move forward, I'd be really happy.

Hiraos hopes for the film actually reference one of the more pivotal lines spoken in the feature: Dont just look down. Look ahead, or youll lose something important.

Pompo the Cinephile - which released in U.S. theaters this past Friday, April 29 - begins with a gutsy producer named Pompo who is famous for her scandalous and trashy B-movies. Meanwhile, her timid, but movie-obsessed, assistant Gene has a dream to one day make great films that will shake the souls of viewers. But Gene believes he still has a long way to go to achieve his goals. Pompo believes otherwise and, after punching out an unexpectedly sincere script about a touching drama, announces that Gene will serve as the films director and sole editor.

As Pompo and Gene assemble their cast, with legendary Brando-esque actor Martin Braddock playing the tortured composer protagonist, and young actress Nathalie Woodward seeking her first break as the disarming heroine, Gene wonders if hes truly up to the task to take Pompos vision and find his own aria within the pages of her script working from 72 hours of filmed shots. But, in the words of Gene, I have no place to go but here. Only two choices: make a movie or die. So, Ill stake my life on this.

Its a sentiment Hirao is also familiar with.

When I was young, I was also the type that couldn't fit in with everybody else, he says. So, watching movies or reading manga or watching anime sort of saved me from loneliness. In that, I was really able to relate to Gene. So, I wanted to create a movie that has themes about the minority versus the majority and the original work by Shogo Sugitani had a lot of those same elements.

Hiraos film, produced by Your Names Kadokawa Productions and Clap Animation Studio, is based off a 2017 manga by Sugitani titled, in Japanese, Eiga Daisuki Ponpo-san. But the movie is saturated with classic Hollywood film inspiration. I was really inspired by Scorsese's Goodfellas, with how the most important scene in the movie is at the beginning and meant to really catch the attention of the viewers, notes Hirao. The inspiration I took from Whiplash came from the spirit of having to sacrifice something to chase your dreams. And then there's also a scene where one of the characters, Mr. Peterzen, is cutting old film and the film that he's cutting is an homage to Taxi Driver.

Specific to the design and sequencing, Hirao also uses film wipes for his films transitions, inspired by 127 Hours as well as the desire to drive home the never-ending, painstaking task of film editing and scene cutting.

The climax of this film is really about Gene editing, says Hirao. I wanted to express that. So, in the whole of the movie I did include a lot of wipes for transitions in various places.

Amidst the storys detailed and cutesy 2D anime, Hirao also incorporates 3D/CG and retro-colored animation during Genes scenes of editing both trailers for Pompos films and the entirety of his own movie. After beginning the computer editing process, Gene finds himself suddenly transported to a black void where film reels race past him. Equipped with a large sword, Gene begins running alongside the reels, hacking, and slashing away.

Especially in the scene where hes editing his own film, because it's a climax, I really wanted to make it dynamic, explains Hirao. If you were to show someone editing a movie in the real world, it's literally going to be the director sitting at a keyboard and just clacking away. But, since this is animation, I decided to mix reality and fantasy. I wanted to turn editing into an action scene. And, because this is animation, its not weird that Gene just steps into this world of fantasy where he has a sword instead of a keyboard.

These scenes also illustrate the abstract and anxiety-ridden feeling of what goes on in the mind of a director as they piece together their film.I do feel like Im holding a sword, but I'm not slashing away, says Hirao, laughing. It's more like Im just standing there, holding the sword wondering if I should cut or not.

While Pompo the Cinephile is a tribute to filmmaking and all the creative departments responsible for bringing a movie to the big screen, its a story, first and foremost, about the brave decision to dream, to reach for the stars no matter how unqualified or unequipped you feel.

Toward the end of the movie, when Gene asks for a reshoot, I really put a lot of emotion in there because I've actually done that before, says Hirao. I made this huge mistake and there was a time when I thought that maybe I wouldnt be able to make movies again. Of course, now, here I am, making this movie. But I really put a lot of that emotion into the scene because it happened to me before. I wanted to keep creating movies and Pompo was like an expression of my resolution to continue doing that.

According to Hirao, the feeling of failure can sometimes be the best fuel for further ambition. Or, in Pompos words from the film, Happiness kills creativity.

I do agree with that to an extent, he shares. Imagination starts with having a lot of emotions. And then those emotions become the core of what you create. So, if there's something that I'm not happy about, or if there's something that I want to express, I think that emotion within you is really what drives you to express what you want to in a movie.

Hirao says he hopes to express with Pompo the Cinephile that theres a home for everyone in the sparkly, colorful world of Nyallywood and wishes that the film will be an affirming push to those who need it.

If the viewers have a dream that they're chasing, I would like this film to be something that would nudge them forward, he says. Maybe after they see this movie, theyll leave the theater thinking, Okay, starting tomorrow, I'll try harder for my dream, or if there was a dream that theyd been giving up on, maybe this will inspire them to go back to chasing that dream again.

Tickets to the film are on sale at PompoMovie.comand participating theater box offices. Theaters and participants are subject to change.

Victoria Davis is a full-time, freelance journalist and part-time Otaku with an affinity for all things anime. She's reported on numerous stories from activist news to entertainment. Find more about her work at victoriadavisdepiction.com.

Read the original:
'Pompo the Cinephile' Goes Deep into the Messy Minds of Filmmakers - Animation World Network

The most sophisticated AI systems today are capable of impressive feats, from directing cars through city streets to writing human-like prose. But they share a common bottleneck: hardware. Developing systems on the bleeding edge often requires a huge amount of computing power. For example, creating DeepMinds protein structure-predicting AlphaFold took a cluster of hundreds of GPUs. Further underlining the challenge, one source estimates that developing AI startup OpenAIs language-generating GPT-3 system using a single GPU wouldve taken 355 years.

New techniques and chips designed to accelerate certain aspects of AI system development promise to (and, indeed, already have) cut hardware requirements. But developing with these techniques calls for expertise that can be tough for smaller companies to come by. At least, thats the assertion of Varun Mohan and Douglas Chen, the co-founders of infrastructure startup Exafunction. Emerging from stealth today, Exafunction is developing a platform to abstract away the complexity of using hardware to train AI systems.

Improvements [in AI] are often underpinned by large increases in computational complexity. As a consequence, companies are forced to make large investments in hardware to realize the benefits of deep learning. This is very difficult because the technology is improving so rapidly, and the workload size quickly increases as deep learning proves value within a company, Chen told TechCrunch in an email interview. The specialized accelerator chips necessary to run deep learning computations at scale are scarce. Efficiently using these chips also requires esoteric knowledge uncommon among deep learning practitioners.

With $28 million in venture capital, $25 million of which came from a Series A round led by Greenoaks with participation from Founders Fund, Exafunction aims to address what it sees as the symptom of the expertise shortage in AI: idle hardware. GPUs and the aforementioned specialized chips used to train AI systems i.e., feed the data that the systems can use to make predictions are frequently underutilized. Because they complete some AI workloads so quickly, they sit idle while they wait for other components of the hardware stack, like processors and memory, to catch up.

Lukas Beiwald, the founder of AI development platform Weights and Biases, reports that nearly a third of his companys customers average less than 15% GPU utilization. Meanwhile, in a 2021 survey commissioned by Run:AI, which competes with Exafunction, just 17% of companies said that they were able to achieve high utilization of their AI resources while 22% said that their infrastructure mostly sits idle.

The costs add up. According to Run:AI, 38% of companies had an annual budget for AI infrastructure including hardware, software and cloud fees exceeding $1 million as of October 2021. OpenAI is estimated to have spent $4.6 million training GPT-3.

Most companies operating in deep learning go into business so they can focus on their core technology, not to spend their time and bandwidth worrying about optimizing resources, Mohan said via email. We believe there is no meaningful competitor that addresses the problem that were focused on, namely, abstracting away the challenges of managing accelerated hardware like GPUs while delivering superior performance to customers.

Prior to co-founding Exafunction, Chen was a software engineer at Facebook, where he helped to build the tooling for devices like the Oculus Quest. Mohan was a tech lead at autonomous delivery startup Nuro responsible for managing the companys autonomy infrastructure teams.

As our deep learning workloads [at Nuro] grew in complexity and demandingness, it became apparent that there was no clear solution to scale our hardware accordingly, Mohan said. Simulation is a weird problem. Perhaps paradoxically, as your software improves, you need to simulate even more iterations in order to find corner cases. The better your product, the harder you have to search to find fallibilities. We learned how difficult this was the hard way and spent thousands of engineering hours trying to squeeze more performance out of the resources we had.

Image Credits: Exafunction

Exafunction customers connect to the companys managed service or deploy Exafunctions software in a Kubernetes cluster. The technology dynamically allocates resources, moving computation onto cost-effective hardware such as spot instances when available.

Mohan and Chen demurred when asked about the Exafunction platforms inner workings, preferring to keep those details under wraps for now. But they explained that, at a high level, Exafunction leverages virtualization to run AI workloads even with limited hardware availability, ostensibly leading to better utilization rates while lowering costs.

Exafunctions reticence to reveal information about its technology including whether it supports cloud-hosted accelerator chips like Googles tensor processing units (TPUs) is cause for some concern. But to allay doubts, Mohan, without naming names, said that Exafunction is already managing GPUs for some of the most sophisticated autonomous vehicle companies and organizations at the cutting edge of computer vision.

Exafunction provides a platform that decouples workloads from acceleration hardware like GPUs, ensuring maximally efficient utilization lowering costs, accelerating performance, and allowing companies to fully benefit from hardware [The] platform lets teams consolidate their work on a single platform, without the challenges of stitching together a disparate set of software libraries, he added. We expect that [Exafunctions product] will be profoundly market-enabling, doing for deep learning what AWS did for cloud computing.

Mohan might have grandiose plans for Exafunction, but the startup isnt the only one applying the concept of intelligent infrastructure allocation to AI workloads. Beyond Run:AI whose product also creates an abstraction layer to optimize AI workloads Grid.aioffers software that allows data scientists to train AI models across hardware in parallel. For its part, Nvidia sells AI Enterprise, a suite of tools and frameworks that lets companies virtualize AI workloads on Nvidia-certified servers.

But Mohan and Chen see a massive addressable market despite the crowdedness. In conversation, they positioned Exafunctions subscription-based platform not only as a way to bring down barriers to AI development but to enable companies facing supply chain constraints to unlock more value from hardware on hand. (In recent years, for a range of different reasons, GPUs have become hot commodities.) Theres always the cloud, but, to Mohans and Chens point, it can drive up costs. One estimate found that training an AI model using on-premises hardware is up to 6.5x cheaper than the least costly cloud-based alternative.

While deep learning has virtually endless applications, two of the ones were most excited about are autonomous vehicle simulation and video inference at scale, Mohan said. Simulation lies at the heart of all software development and validation in the autonomous vehicle industry Deep learning has also led to exceptional progress in automated video processing, with applications across a diverse range of industries. [But] though GPUs are essential to autonomous vehicle companies, their hardware is frequently underutilized, despite their price and scarcity. [Computer vision applications are] also computationally demanding, [because] each new video stream effectively represents a firehose of data with each camera outputting millions of frames per day.

Mohan and Chen say that the capital from the Series A will be put toward expanding Exafunctions team and deepening the product. The company will also invest in optimizing AI system runtimes for the most latency-sensitive applications (e.g., autonomous driving and computer vision).

While currently we are a strong and nimble team focused primarily on engineering, we expect to rapidly build the size and capabilities of our org in 2022, Mohan said. Across virtually every industry, it is clear that as workloads grow more complex (and a growing number of companies wish to leverage deep-learning insights), demand for compute is vastly exceeding [supply]. While the pandemic has highlighted these concerns, this phenomenon, and its related bottlenecks, is poised to grow more acute in the years to come, especially as cutting-edge models become exponentially more demanding.

Read the original here:
Exafunction aims to reduce AI dev costs by abstracting away hardware - TechCrunch