We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 - 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!

Algorithms have always been at home in the digital world, where they are trained and developed in perfectly simulated environments. The current wave of deep learning facilitates AIs leap from the digital to the physical world. The applications are endless, from manufacturing to agriculture, but there are still hurdles to overcome.

To traditional AI specialists, deep learning (DL) is old hat. It got its breakthrough in 2012 when Alex Krizhevsky successfully deployed convolutional neural networks, the hallmark of deep learning technology, for the first time with his AlexNet algorithm. Its neural networks that have allowed computers to see, hear and speak. DL is the reason we can talk to our phones and dictate emails to our computers. Yet DL algorithms have always played their part in the safe simulated environment of the digital world. Pioneer AI researchers are working hard to introduce deep learning to our physical, three-dimensional world. Yep, the real world.

Deep learning could do much to improve your business, whether you are a car manufacturer, a chipmaker or a farmer. Although the technology has matured, the leap from the digital to the physical world has proven to be more challenging than many expected. This is why weve been talking about smart refrigerators doing our shopping for years, but no one actually has one yet. When algorithms leave their cozy digital nests and have to fend for themselves in three very real and raw dimensions there is more than one challenge to be overcome.

The first problem is accuracy. In the digital world, algorithms can get away with accuracies of around 80%. That doesnt quite cut it in the real world. If a tomato harvesting robot sees only 80% of all tomatoes, the grower will miss 20% of his turnover, says Albert van Breemen, a Dutch AI researcher who has developed DL algorithms for agriculture and horticulture in The Netherlands. His AI solutions include a robot that cuts leaves of cucumber plants, an asparagus harvesting robot and a model that predicts strawberry harvests. His company is also active in the medical manufacturing world, where his team created a model that optimizes the production of medical isotopes. My customers are used to 99.9% accuracy and they expect AI to do the same, Van Breemen says. Every percent of accuracy loss is going to cost them money.

To achieve the desired levels, AI models have to be retrained all the time, which requires a flow of constantly updated data. Data collection is both expensive and time-consuming, as all that data has to be annotated by humans. To solve that challenge Van Breemen has outfitted each of his robots with functionality that lets it know when it is performing either well or badly. When making mistakes the robots will upload only the specific data where they need to improve. That data is collected automatically across the entire robot fleet. So instead of receiving thousands of images, Van Breemens team only gets a hundred or so, that are then labeled and tagged and sent back to the robots for retraining. A few years ago everybody said that data is gold, he says. Now we see that data is actually a huge haystack hiding a nugget of gold. So the challenge is not just collecting lots of data, but the right kind of data.

His team has developed software that automates the retraining of new experiences. Their AI models can now train for new environments on their own, effectively cutting out the human from the loop. Theyve also found a way to automate the annotation process by training an AI model to do much of the annotation work for them. Van Breemen: Its somewhat paradoxical because you could argue that a model that can annotate photos is the same model I need for my application. But we train our annotation model with a much smaller data size than our goal model. The annotation model is less accurate and can still make mistakes, but its good enough to create new data points we can use to automate the annotation process.

The Dutch AI specialist sees a huge potential for deep learning in the manufacturing industry, where AI could be used for applications like defect detection and machine optimization. The global smart manufacturing industry is currently valued at 198 billion dollars and has a predicted growth rate of 11% until 2025. The Brainport region around the city of Eindhoven where Van Breemens company is headquartered is teeming with world-class manufacturing corporates, such as Philips and ASML. (Van Breemen has worked for both companies in the past.)

A second challenge of applying AI in the real world is the fact that physical environments are much more varied and complex than digital ones. A self-driving car that is trained in the US will not automatically work in Europe with its different traffic rules and signs. Van Breemen faced this challenge when he had to apply his DL model that cuts cucumber plant leaves to a different growers greenhouse. If this took place in the digital world I would just take the same model and train it with the data from the new grower, he says. But this particular grower operated his greenhouse with LED lighting, which gave all the cucumber images a bluish-purple glow our model didnt recognize. So we had to adapt the model to correct for this real-world deviation. There are all these unexpected things that happen when you take your models out of the digital world and apply them to the real world.

Van Breemen calls this the sim-to-real gap, the disparity between a predictable and unchanging simulated environment and the unpredictable, ever-changing physical reality. Andrew Ng, the renowned AI researcher from Stanford and cofounder of Google Brain who also seeks to apply deep learning to manufacturing, speaks of the proof of concept to production gap. Its one of the reasons why 75% of all AI projects in manufacturing fail to launch. According to Ng paying more attention to cleaning up your data set is one way to solve the problem. The traditional view in AI was to focus on building a good model and let the model deal with noise in the data. However, in manufacturing a data-centric view may be more useful, since the data set size is often small. Improving data will then immediately have an effect on improving the overall accuracy of the model.

Apart from cleaner data, another way to bridge the sim-to-real gap is by using cycleGAN, an image translation technique that connects two different domains, made popular by aging apps like FaceApp. Van Breemens team researched cycleGAN for its application in manufacturing environments. The team trained a model that optimized the movements of a robotic arm in a simulated environment, where three simulated cameras observed a simulated robotic arm picking up a simulated object. They then developed a DL algorithm based on cycleGAN that translated the images from the real world (three real cameras observing a real robotic arm picking up a real object) to a simulated image, which could then be used to retrain the simulated model. Van Breemen: A robotic arm has a lot of moving parts. Normally you would have to program all those movements beforehand. But if you give it a clearly described goal, such as picking up an object, it will now optimize the movements in the simulated world first. Through cycleGAN you can then use that optimization in the real world, which saves a lot of man-hours. Each separate factory using the same AI model to operate a robotic arm would have to train its own cycleGAN to tweak the generic model to suit its own specific real-world parameters.

The field of deep learning continues to grow and develop. Its new frontier is called reinforcement learning. This is where algorithms change from mere observers to decision-makers, giving robots instructions on how to work more efficiently. Standard DL algorithms are programmed by software engineers to perform a specific task, like moving a robotic arm to fold a box. A reinforcement algorithm could find out there are more efficient ways to fold boxes outside of their preprogrammed range.

It was reinforcement learning (RL) that made an AI system beat the worlds best Go player back in 2016. Now RL is also slowly making its way into manufacturing. The technology isnt mature enough to be deployed just yet, but according to the experts, this will only be a matter of time.

With the help of RL, Albert Van Breemen envisions optimizing an entire greenhouse. This is done by letting the AI system decide how the plants can grow in the most efficient way for the grower to maximize profit. The optimization process takes place in a simulated environment, where thousands of possible growth scenarios are tried out. The simulation plays around with different growth variables like temperature, humidity, lighting and fertilizer, and then chooses the scenario where the plants grow best. The winning scenario is then translated back to the three-dimensional world of a real greenhouse. The bottleneck is the sim-to-real gap, Van Breemen explains. But I really expect those problems to be solved in the next five to ten years.

As a trained psychologist I am fascinated by the transition AI is making from the digital to the physical world. It goes to show how complex our three-dimensional world really is and how much neurological and mechanical skill is needed for simple actions like cutting leaves or folding boxes. This transition is making us more aware of our own internal, brain-operated algorithms that help us navigate the world and which have taken millennia to develop. Itll be interesting to see how AI is going to compete with that. And if AI eventually catches up, Im sure my smart refrigerator will order champagne to celebrate.

Bert-Jan Woertman is the director of Mikrocentrum.

Welcome to the VentureBeat community!

DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.

If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.

You might even considercontributing an articleof your own!

Read More From DataDecisionMakers

See the article here:
Deep learning is bridging the gap between the digital and the real world - VentureBeat

Opinion Machine learning's abiding weakness is verification. Is your AI telling the truth? How can you tell?

This problem isn't unique to ML. It plagues chip design, bathroom scales, and prime ministers. Still, with so many new business models depending on AI's promise to bring the holy grail of scale to real-world data analysis, this lack of testability has new economic consequences.

The basic mechanisms of machine learning are sound, or at least statistically reliable. Within the parameters of its training data, an ML process will deliver what the underlying mathematics promise. If you understand the limits, you can trust it.

But what if there's a backdoor, a fraudulent tweak of that training data set which will trigger misbehavior? What if there's a particular quirk in someone's loan request submitted at exactly 00:45 on the 5th and the amount requested checksums to 7 that triggers automatic acceptance, regardless of risk?

Like an innocent assassin unaware they'd had a kill word implanted under hypnosis, your AI would behave impeccably until the bad guys wanted it otherwise.

Intuitively, we know that's a possibility. Now it has been shown mathematically that not only can this happen, researchers say, it's not theoretically detectable. An AI backdoor exploit engineered through training is not only just as much a problem as a traditionally coded backdoor, it's not amenable to inspection or version-on-version comparison or, indeed, anything. As far as the AI's concerned, everything is working perfectly, Harry Palmer could never confess to wanting to shoot JFK, he had no idea he did.

The mitigations suggested by researchers aren't very practical. Complete transparency of training data and process between AI company and client is a nice idea, except that the training data is the company's crown jewels and if they're fraudulent, how does it help?

At this point, we run into another much more general tech industry weakness, the idea that you can always engineer a singular solution to a particular problem. Pay the man, Janet, and let's go home. That doesn't work here; computer says no is one thing, mathematics says no quite another. If we carry on assuming that there'll be a fix akin to a patch, some new function that makes future AIs resistant to this class of fraud, we will be defrauded.

Conversely, the industry does genuinely advance once fundamental flaws are admitted and accepted, and the ecosystem itself changes in recognition.

AI has an ongoing history of not working as well as we thought, and it's not just this or that project. For example, an entire sub-industry has evolved to prove you are not a robot. Using its own trained robots to silently watch you as you move around online. If these machine monitors deem you too robotic, they spring a Voight-Kampff test on you in the guise of a Completely Automated Public Turing test to tell Computers and Humans Apart more widely known, and loathed, as a Captcha. You then have to pass a quiz designed to filter out automata. How undignified.

Do they work? It's still economically viable for the bad guys to carry on producing untold millions of programmatic fraudsters intent on deceiving the advertising industry, so that's a no on the false positives. And it's still common to be bounced from a login because your eyes aren't good enough, or the question too ambiguous, or the feature you relied on has been taken away. Not being able to prove you are not a robot doesn't get you shot by Harrison Ford, at least for now, but you may not be able to get into eBay.

The answer here is not to build a "better" AI and feed it with more and "better" surveillance signals. It's to find a different model to identify humans online, without endangering their privacy. That's not going to be a single solution invented by a company, that's an industry-wide adoption of new standards, new methods.

Likewise, you will never be able to buy a third-party AI that is testably pure of heart. To tell the truth, you'll never be able to build one yourself, at least not if you've got a big enough team or a corporate culture where internal fraud can happen. That's a team of two or more, and any workable corporate culture yet invented.

That's OK, once you stop looking for that particular unicorn. We can't theoretically verify non-trivial computing systems of any kind. When we have to use computers where failure is not an option, like flying aircraft or exploring space, we use multiple independent systems and majority voting.

If it seems that building a grand scheme on the back of the "perfect" black box works as badly as designing a human society on the model of the perfectly rational human, congratulations. Handling the complexities of real world data at real world scale means accepting that any system is fallible in ways that can't be patched or programmed out of. We're not at the point where AI engineering is edging into AI psychology, but it's coming.

Meanwhile, there's no need to give up on your AI-powered financial fraud detection. Buy three AIs from three different companies. Use them to check each other. If one goes wonky, use the other two until you can replace the first.

Can't afford three AIs? You don't have a workable business model. At least AI is very good at proving that.

More:
Your AI can't tell you it's lying if it thinks it's telling the truth. That's a problem - The Register

FDA has released a number of documents that could help clarify its expectations for artificial intelligence, machine learning, and cybersecurity. These include Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan, published in January 2021; Good Machine Learning Practice for Medical Device Development: Guiding Principles, published in October 2021; and the just-released draft guidance, Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions.

Michelle Jump, vice president of security services for Medsec; and Yarmela Pavlovic, vice president of regulatory strategy for Medtronic, explored these documents during the IME West 2022 session on April 12, Product Development Innovation in an Age of Regulatory Uncertainty.

The AI/ML action plan provides a more tailored regulatory framework for AI/ML, explained Pavlovic. She referred to FDAs 2019 discussion paper, Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback, which laid out a total product lifecycle approach to AI/ML regulations with the understanding that AI/ML products can be iterated much more efficiently and quickly than a typical medical device implant product or something that isnt software based. This is because there is an opportunity to add additional data to training sets on which the products were originally formulated, she said.

Key to the total product lifecycle approach is a predetermined change control plan" (PCCP),which describes all the types of changes the manufacturer intends to make to the product in the foreseeable future and the protocols and success criteria and data sets they will use for evaluating the performance of these products, she explained.

In answering a question from the audience about PCCPs, Pavlovic explained that when you think about formulating a plan, you have to start with what the testing package you plan to give to FDA looks like. The level of detail of the PCCP will match the level of detail in the protocol and the test reports in your starting package. What you do in the future to validate a product flows from what you did in the first instance. And then the types of changes that you describe may warrant additional or differenttypes of testing, so be thoughtful about the key scientific questions that are posed by those changes and how can you layer on the right level of evidence to be comfortable.

The Medical Device Innovation Consortium has a digital health vertical with a workstream looking to build out PCCP examples, she added.

The AI/ML action plan also includes good machine learning practices (GMLP), such as having good hygiene around data sets and data management practices as well as processes to continue to learn about the product as it is used in commercial practice to make sure you are continuing to understand the performance of your product, explained Pavlovic.

Also,algorithmic bias and robustness is an area for further regulatory science development, she said. The goal would be to ensure the performance of the product is representative of the intended use population and that we dont have unintended consequences from the use of our products because of bias present due to the way they were trained or choices of data sets.

Real-world performance is the last element of the plan, she said, which is the idea that companies would monitor performance of the product in the wild, such as in clinical use. But it is complex to have info sharing agreements with customers, she said, so there may be the need to find other ways to gather such data.

Answering an audience question on FDAs stance toward AI, Pavlovic said that it is important for companies to make it clear to FDA that they are taking a rigorous approach to evaluation of a product and that there are processes in place to ensure continued performance once the product is on the market.

We have a responsibility to get these products right so we dont undermine forward progress. All of us in this room take that responsibility seriously. A couple poorly performing products will set us back, she added.

Jump, who has been in the digital health space for more than a decade, added that she has watched FDA become more comfortable with software. For instance, FDA wasnt initially comfortable with remote updates, but now they are saying, Why arent you doing remote software updates? she said.

But what FDA may not be comfortable with could be the reasoning behind a product, she said. If its new and innovative . . . get comfortable explaining what you are doing. When it comes to AI, for instance, FDA is concerned they see things going into a black box and coming out with an answer, she explained. They dont understand what could change that would result in an unexpected decision that clinicians are trusting.

Pavlovic and Jump also shared some initial feedback on the brand new April 2022 draft guidance on cybersecurity, which is much longer than the 2014 guidance. It may scare people . . . but it is a blessing in disguise, said Jump. Follow what this guidance says, because it is what they are going to ask for . . . they are already asking for [it].

For instance, just as medical device companies have to do risk management based on other regulation, the agency now expects threat modeling, she explained.

Get your comments in now, Jump added. I dont think theyre going to change a lot. Said Pavlovic: It is very consistent with what theyve been saying.

Jump pointed out a few new terms in the guidance, such as software product development framework (SPDF). You should have a secure design processwhich is a new name for an old concept. A good secure design is much more effective than constantly going in and patching things. FDA really wants to push better, secure design approaches.

She also clarified the scope of FDAs regulatory approach. You do not have to be a connected product to be applicable to cybersecurityif it has software or programmable logic, you are under scope of the guidance. She also added that if a company is submitting a change to hardware, FDA may still ask about cybersecurity.

Read the original post:
AI, ML, & Cybersecurity: Here's What FDA May Soon Be Asking - Design News

As the tech world inches a closer to the idea of artificial general intelligence, were seeing another interesting theme emerging in the ongoing democratization of AI: a wave of startups building tech to make AI technologies more accessible overall by a wider range of users and organizations.

Today, one of these, Baseten which is building tech to make it easier to incorporate machine learning into a business operations, production and processes without a need for specialized engineering knowledge is announcing $20 million in funding and the official launch of its tools.

These include a client API and a library of pre-trained models to deploy models built in TensorFlow, PyTorch or scikit-learn; the ability to build APIs to power your own applications; and the ability the create custom UIs for your applications based on drag-and-drop components.

The company has been operating in a closed, private beta for about a year and has amassed an interesting group of customers so far, including both Stanford and the University of Sydney, Cockroach Labs and Patreon, among others, who use it to, for example, help organizations with automated abuse detection (through content moderation) and fraud prevention.

The $20 million is being discussed publicly for the first time now to coincide with the commercial launch, and its in two tranches, with equally notable names among those backers.

The seed was co-led by Greylock and South Park Commons Fund, with participation also from the AI Fund, Caffeinated Capital and individuals including Greg Brockman, co-founder and CTO at general intelligence startup OpenAI; Dylan Field, co-founder and CEO of Figma; Mustafa Suleyman, co-founder of DeepMind; and DJ Patil, ex-chief scientist of the United States.

Greylock also led the Series A, with participation from South Park Commons, early Stripe exec Lachy Groom; Dev Ittycheria, CEO of MongoDB; Jay Simon, ex-president of Atlassian, now at Bond; Jean-Denis Greze, CTO of Plaid; and Cristina Cordova, another former Stripe exec.

Tuhin Srivastava, Basetens co-founder and CEO, said in an interview that the funding will be used in part to bring on more technical and product people, and to ramp up its marketing and business development.

The issue that Baseten has identified and is trying to solve is one that is critical in the evolution of AI: Machine learning tools are becoming ever more ubiquitous and utilized, thanks to cheaper computing power, better access to training models and a growing understanding of how and where they can be used. But one area where developers still need to make a major leap, and businesses still need to make big investments, is when it comes to actually adopting and integrating machine learning: There remains a wide body of technical knowledge that developers and data scientists need to actually integrate machine learning into their work.

We were born out of the idea that machine learning will have a massive impact on the world, but its still difficult to extract value from machine learning models, Srivastava said. Difficult, because developers and data scientists need to have specific knowledge of how to handle machine learning ops, as well as technical expertise to manage production at the back end and the front end, he said. This is one reason why machine learning programs in businesses often actually have very little success: It takes too much effort to get them into production.

This is something that Srivastava and his co-founders Amir Haghighat (CTO) and Philip Howes (chief scientist) experienced firsthand when they worked together at Gumroad. Haghighat, who was head of engineering, and Srivastava and Howes, who were data scientists, wanted to use machine learning at the payments company to help with fraud detection and content moderation and realized that they needed to pick up a lot of extra full-stack engineering skills or hire specialists to build and integrate that machine learning along with all of the tooling needed to run it (e.g., notifications and integrating that data into other tools to action).

They built the systems still in use, and screening hundreds of millions of dollars of transactions but also picked up an idea in the process: Others surely were facing the same issues they did, so why not work on a set of tools to help all of them and take away some of that work?

Today, the main customers of Baseten a reference to base ten blocks, often used to help younger students learn the basics of mathematics (It humanizes the numbers system, and we wanted to make machine learning less abstract, too, said the CEO) are developers and data scientists who are potentially adopting other machine learning models, or even building their own but lack the skills to practically incorporate them into their own production flows. There, Baseten is part of a bigger group of companies that appear to be emerging building MLops solutions full sets of tools to make machine learning more accessible and usable by those working in DevOps and product. These include Databricks, Clear, Gathr and more. The idea here is to give tools to technical people to give them more power and more time to work on other tasks.

Baseten gets the process of tool-building out of the way so we can focus on our key skills: modeling, measurement and problem-solving, said Nikhil Harithras, senior machine learning engineer at Patreon, in a statement. Patreon is using Baseten to help run an image classification system, used to find content that violates its community guidelines.

Over time, there a logical step that Baseten could make, continuing on its democratization trajectory: considering how to build tools for non-technical audiences, too an interesting idea in light of the many no-code and low-code products that are being rolled out to give them more power to build their own data science applications, too.

Non-technical audiences are not something we focus on today, but that is the evolution, Srivastava said. The highest level goal is to accelerate the impact of machine learning.

More:
Baseten nabs $20M to make it easier to build machine learning-based applications - TechCrunch