Cloud Hosting

The expectation-maximization (EM) algorithm is an elegant algorithm that maximizes the likelihood function for problems with latent or hidden variables. As from the name itself it could primarily be understood that it does two things one is the expectation and the other is maximization. This article would help to understand the math behind the EM algorithm with an implementation. Following are the topics to be covered.

Lets try to understand how the expectation and maximization combination helps to decide the number of clusters to be formed but before that we need to understand the concept of the latent variable.

A latent variable is a random variable that can be observed neither in training nor in the test phase. These variables cant be measured on a quantitative scale. There are two reasons to use latent variables:

The latent variable is the direct causation of all the parameters. Now the final model is much simpler to work with and has the same efficiency without reducing the flexibility of the model. There is one drawback of latent variables: it is harder to train these models.

Are you looking for a complete repository of Python libraries used in data science,check out here.

The general form of probability distribution arises from the observed variables for the variables that arent directly observable also known as latent variables, the expectation-maximization algorithm is used to predict their values by using the values of the other observed variable. This algorithm is the building block of many unsupervised clustering algorithms in the field of machine learning. This algorithm has two major computational steps which are expectation and maximization:

A high-level idea of EM algorithm functioning is stated below.

So, we had an understanding of the EM algorithm functionality but for implementation of this algorithm in python we need to understand the model which uses this algorithm to form clusters. Lets talk about the Gaussian Mixture model.

The Gaussian Mixture Model is an important concept in machine learning which uses the concept of expectation-maximization. A Gaussian Mixture is composed of several Gaussians, each represented by k which is the subset of the number of clusters to be formed. For each Gaussian k in the mixture the following parameters are present:

The above plot explains the Gaussian distribution for the data having a mean of 4 and a variance of 0.25. This could be concluded as the normal distribution. Using an iterative process the model concludes the final number of the cluster with the help of these parameters which determines the cluster stability.

Lets implement the concept of expectation-maximization in python.

Import necessary libraries

Reading and analyzing the data

Using the famous wine data for this implementation.

Plotting a distribution

This plot helps to understand the distribution of the dependent variable over the independent variable.

Fitting the GMM

The score function returns the log-likelihood which the lower the better. The is negative because it is the product of the density evaluated at the observations and the density takes values that are smaller than one, so its logarithm will be negative. Ignoring the negative and focusing on the magnitude which is 0.73 indicates the model is good and the number of clusters should be 6.

The expectation-Maximization Algorithm represents the idea of computing the latent variables by taking the parameters as fixed and known. The algorithm is inherently fast because it doesnt depend on computing gradients. With a hands-on implementation of this concept in this article, we could understand the expectation-maximization algorithm in machine learning.

Original post:
How is the Expectation-Maximization algorithm used in machine learning? - Analytics India Magazine

Humans may be one of the biggest roadblocks keeping fully autonomous vehicles off city streets.

If a robot is going to navigate a vehicle safely through downtown Boston, it must be able to predict what nearby drivers, cyclists, and pedestrians are going to do next.

Behavior prediction is a tough problem, however, and current artificial intelligence solutions are either too simplistic (they may assume pedestrians always walk in a straight line), too conservative (to avoid pedestrians, the robot just leaves the car in park), or can only forecast the next moves of one agent (roads typically carry many users at once.)

MIT researchers have devised a deceptively simple solution to this complicated challenge. They break a multiagent behavior prediction problem into smaller pieces and tackle each one individually, so a computer can solve this complex task in real-time.

Their behavior-prediction framework first guesses the relationships between two road users which car, cyclist, or pedestrian has the right of way, and which agent will yield and uses those relationships to predict future trajectories for multiple agents.

These estimated trajectories were more accurate than those from other machine-learning models, compared to real traffic flow in an enormous dataset compiled by autonomous driving company Waymo. The MIT technique even outperformed Waymos recently published model. And because the researchers broke the problem into simpler pieces, their technique used less memory.

This is a very intuitive idea, but no one has fully explored it before, and it works quite well. The simplicity is definitely a plus. We are comparing our model with other state-of-the-art models in the field, including the one from Waymo, the leading company in this area, and our model achieves top performance on this challenging benchmark. This has a lot of potential for the future, says co-lead author Xin Cyrus Huang, a graduate student in the Department of Aeronautics and Astronautics and a research assistant in the lab of Brian Williams, professor of aeronautics and astronautics and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL).

Joining Huang and Williams on the paper are three researchers from Tsinghua University in China: co-lead author Qiao Sun, a research assistant; Junru Gu, a graduate student; and senior author Hang Zhao PhD 19, an assistant professor. The research will be presented at the Conference on Computer Vision and Pattern Recognition.

Multiple small models

The researchers machine-learning method, called M2I, takes two inputs: past trajectories of the cars, cyclists, and pedestrians interacting in a traffic setting such as a four-way intersection, and a map with street locations, lane configurations, etc.

Using this information, a relation predictor infers which of two agents has the right of way first, classifying one as a passer and one as a yielder. Then a prediction model, known as a marginal predictor, guesses the trajectory for the passing agent, since this agent behaves independently.

A second prediction model, known as a conditional predictor, then guesses what the yielding agent will do based on the actions of the passing agent. The system predicts a number of different trajectories for the yielder and passer, computes the probability of each one individually, and then selects the six joint results with the highest likelihood of occurring.

M2I outputs a prediction of how these agents will move through traffic for the next eight seconds. In one example, their method caused a vehicle to slow down so a pedestrian could cross the street, then speed up when they cleared the intersection. In another example, the vehicle waited until several cars had passed before turning from a side street onto a busy, main road.

While this initial research focuses on interactions between two agents, M2I could infer relationships among many agents and then guess their trajectories by linking multiple marginal and conditional predictors.

Real-world driving tests

The researchers trained the models using the Waymo Open Motion Dataset, which contains millions of real traffic scenes involving vehicles, pedestrians, and cyclists recorded by lidar (light detection and ranging) sensors and cameras mounted on the companys autonomous vehicles. They focused specifically on cases with multiple agents.

To determine accuracy, they compared each methods six prediction samples, weighted by their confidence levels, to the actual trajectories followed by the cars, cyclists, and pedestrians in a scene. Their method was the most accurate. It also outperformed the baseline models on a metric known as overlap rate; if two trajectories overlap, that indicates a collision. M2I had the lowest overlap rate.

Rather than just building a more complex model to solve this problem, we took an approach that is more like how a human thinks when they reason about interactions with others. A human does not reason about all hundreds of combinations of future behaviors. We make decisions quite fast, Huang says.

Another advantage of M2I is that, because it breaks the problem down into smaller pieces, it is easier for a user to understand the models decision making. In the long run, that could help users put more trust in autonomous vehicles, says Huang.

But the framework cant account for cases where two agents are mutually influencing each other, like when two vehicles each nudge forward at a four-way stop because the drivers arent sure who should be yielding.

They plan to address this limitation in future work. They also want to use their method to simulate realistic interactions between road users, which could be used to verify planning algorithms for self-driving cars or create huge amounts of synthetic driving data to improve model performance.

Predicting future trajectories of multiple, interacting agents is under-explored and extremely challenging for enabling full autonomy in complex scenes. M2I provides a highly promising prediction method with the relation predictor to discriminate agents predicted marginally or conditionally which significantly simplifies the problem, wrote Masayoshi Tomizuka, the Cheryl and John Neerhout, Jr. Distinguished Professor of Mechanical Engineering at University of California at Berkeley and Wei Zhan, an assistant professional researcher, in an email. The prediction model can capture the inherent relation and interactions of the agents to achieve the state-of-the-art performance. The two colleagues were not involved in the research.

This research is supported, in part, by the Qualcomm Innovation Fellowship. Toyota Research Institute also provided funds to support this work.

Original post:
Anticipating others' behavior on the road | MIT News | Massachusetts Institute of Technology - MIT News

We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 - 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!

Every company today is data-driven or at least claims to be. Business decisions are no longer made based on hunches or anecdotal trends as they were in the past. Concrete data and analytics now power businesses most critical decisions.

As more companies leverage the power of machine learning and artificial intelligence to make critical choices, there must be a conversation around the qualitythe completeness, consistency, validity, timeliness and uniquenessof the data used by these tools. The insights companies expect to be delivered by machine learning (ML) or AI-based technologies are only as good as the data used to power them. The old adage garbage in, garbage out, comes to mind when it comes to data-based decisions.

Statistically, poor data quality leads to increased complexity of data ecosystems and poor decision-making over the long term. In fact, roughly $12.9 million is lost every year due to poor data quality. As data volumes continue to increase, so will the challenges that businesses face with validating and their data. To overcome issues related to data quality and accuracy, its critical to first know the context in which the data elements will be used, as well as best practices to guide the initiatives along.

Data initiatives are not specific to a single business driver. In other words, determining data quality will always depend on what a business is trying to achieve with that data. The same data can impact more than one business unit, function or project in very different ways. Furthermore, the list of data elements that require strict governance may vary according to different data users. For example, marketing teams are going to need a highly accurate and validated email list while R&D would be invested in quality user feedback data.

The best team to discern a data elements quality, then, would be the one closest to the data. Only they will be able to recognize data as it supports business processes and ultimately assess accuracy based on what the data is used for and how.

Data is an enterprise asset. However, actions speak louder than words. Not everyone within an enterprise is doing all they can to make sure data is accurate. If users do not recognize the importance of data quality and governanceor simply dont prioritize them as they shouldthey are not going to make an effort to both anticipate data issues from mediocre data entry or raise their hand when they find a data issue that needs to be remediated.

This might be addressed practically by tracking data quality metrics as a performance goal to foster more accountability for those directly involved with data. In addition, business leaders must champion the importance of their data quality program. They should align with key team members about the practical impact of poor data quality. For instance, misleading insights that are shared in inaccurate reports for stakeholders, which can potentially lead to fines or penalties. Investing in better data literacy can help organizations create a culture of data quality to avoid making careless or ill-informed mistakes that damage the bottom line.

It is not practical to fix a large laundry list of data quality problems. Its not an efficient use of resources either. The number of data elements active within any given organization is huge and is growing exponentially. Its best to start by defining an organizations Critical Data Elements (CDEs), which are the data elements integral to the main function of a specific business. CDEs are unique to each business. Net Revenue is a common CDE for most businesses as its important for reporting to investors and other shareholders, etc.

Since every company has different business goals, operating models and organizational structures, every companys CDEs will be different. In retail, for example, CDEs might relate to design or sales. On the other hand, healthcare companies will be more interested in ensuring the quality of regulatory compliance data. Although this is not an exhaustive list, business leaders might consider asking the following questions to help define their unique CDEs: What are your critical business processes? What data is used within those processes? Are these data elements involved in regulatory reporting? Will these reports be audited? Will these data elements guide initiatives in other departments within the organization?

Validating and remediating only the most key elements will help organizations scale their data quality efforts in a sustainable and resourceful way. Eventually, an organizations data quality program will reach a level of maturity where there are frameworks (often with some level of automation) that will categorize data assets based on predefined elements to remove disparity across the enterprise.

Businesses drive value by knowing where their CDEs are, who is accessing them and how theyre being used. In essence, there is no way for a company to identify their CDEs if they dont have proper data governance in place at the start. However, many companies struggle with unclear or non-existent ownership into their data stores. Defining ownership before onboarding more data stores or sources promotes commitment to quality and usefulness. Its also wise for organizations to set up a data governance program where data ownership is clearly defined and people can be held accountable. This can be as simple as a shared spreadsheet dictating ownership of the set of data elements or can be managed by a sophisticated data governance platform, for example.

Just as organizations should model their business processes to improve accountability, they must also model their data, in terms of data structure, data pipelines and how data is transformed. Data architecture attempts to model the structure of an organizations logical and physical data assets and data management resources. Creating this type of visibility gets at the heart of the data quality issue, that is, without visibility into the *lifecycle* of datawhen its created, how its used/transformed and how its outputtedits impossible to ensure true data quality.

Even when data and analytics teams have established frameworks to categorize and prioritize CDEs, they are still left with thousands of data elements that need to either be validated or remediated. Each of these data elements can require one or more business rules that are specific to the context in which it will be used. However, those rules can only be assigned by the business users working with those unique data sets. Therefore, data quality teams will need to work closely with subject matter experts to identify rules for each and every unique data element, which can be extremely dense, even when they are prioritized. This often leads to burnout and overload within data quality teams because they are responsible for manually writing a large sum of rules for a variety of data elements. When it comes to the workload of their data quality team members, organizations must set realistic expectations. They may consider expanding their data quality team and/or investing in tools that leverage ML to reduce the amount of manual work in data quality tasks.

Data isnt just the new oil of the world: its the new water of the world. Organizations can have the most intricate infrastructure, but if the water (or data) running through those pipelines isnt drinkable, its useless. People that need this water must have easy access to it, they must know that its usable and not tainted, they must know when supply is low and, lastly, the suppliers/gatekeepers must know who is accessing it. Just as access to clean drinking water helps communities in a variety of ways, improved access to data, mature data quality frameworks and deeper data quality culture can protect data-reliant programs & insights, helping spur innovation and efficiency within organizations around the world.

JP Romero is Technical Manager at Kalypso

Welcome to the VentureBeat community!

DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.

If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.

You might even considercontributing an articleof your own!

Read More From DataDecisionMakers

See the original post here:
Top 5 data quality & accuracy challenges and how to overcome them - VentureBeat

Anthony Iasso, Xator Corp.

The finalists for WashingtonExecs Chief Officer Awards were announced March 25, and well be highlighting some of them until the event takes place live, in-person May 11 at the The Ritz-Carlton in McLean, Virginia.

Next is Chief Technology Officer (Private & Public) finalist Anthony Iasso, whos CTO at Xator Corp. Here, he talks primary focus areas going forward, taking professional risks, proud career moments and more.

What has made you successful in your current role?

The incredibly talented people who work at Xator, our partner companies and our customer organizations make me successful in my current role. My focus is developing and leading the Xator technology strategy and vision. We need to be leading edge, though not always bleeding edge, because our customers need proven solutions that balance innovation with risk.

Securing embassies or equipping Marines cant be a science experiment. I keep us focused on key performance measures for technical systems to be sure what we deliver works as intended, to meet the customers requirements. I do that by marshalling the tremendous talent we have in a whole-of-Xator approach, by bringing together people from across the entire organization to focus on immediate and future challenges through solutioneering.

What energizes me is to learn and understand our customer challenges, and then bring to bear our technologists, Xator core technologies and partner technologies and talent to deliver solutions better, faster and more cost effectively than any of our competitors.

Im successful when the customers mission is properly supported, and Xator, our partners and customers are proud of the work weve done.

What are your primary focus areas going forward, and why are those so important to the future of the nation?

One of my primary focuses is on the balance between security technology and privacy. We are bringing amazing technologies together in the areas of biometrics, identity understanding, machine learning, low-cost ubiquitous sensors and cameras, data collection and data analytics that are changing the way we secure our country.

But balancing what we can do, with what we should do with this technology, will be the defining question for our nations future. Technologists like me must support the transparent application of these technologies in a way that accomplishes our security objectives while at the same time safeguards privacy and protections of a free society.

How do you help shape the next generation of government leaders/industry leaders?

Leading by example is always a great start. When I graduated from West Point, I remember thinking, Wow, Im in the same spot that Eisenhower, Grant, MacArthur and countless other great leaders once stood.

I frequently look back since then and think about the process that transformed those who came before from young eager kids into great national leaders. It is a process of pulling up the next generation of leaders, while being pulled up by the previous generation of leaders.

I am still learning from my mentors and developing new mentors that are worthy of emulation, and I try to fill that role for those who have worked for and with me over the years. In that, I feel the responsibility of being a link in this multigenerational process. The military has an amazing ability to transform second lieutenants into four-star generals, by a process of gradually increasing the scope of responsibilities and letting leaders lead at each step of the way. I think that same approach applies to success in civil service and the civilian world. Never stop learning, and never stop teaching.

Which rules do you think you should break more as a government/industry leader?

This is an interesting question and I stared at it for a while before selecting it to answer for this interview, but I should be bold and go for it. I am not a rule breaker by nature, and one of my core tenets is to never burn bridges. In this business, politics and bureaucracy are intertwined with the ability to break through, win business and deliver solutions. An unwritten rule is dont rock the boat. You never know who may be making decisions in the future that can affect your core business, and a bad relationship can one day block you out.

We cant go right to our end users in government and get them to buy our solutions, even if we have the best things since sliced bread. I am becoming more inclined to call out situations where biases and obstructions, especially if they are political or bureaucratic, prevent progress and innovation, because Ive seen good businesses suffer and Ive seen end users suffer.

Maybe I cant break through, but maybe I can. Maybe I make an anti-sponsor, but maybe I make a sponsor from an anti-sponsor. Over the years, Ive become more inclined to try and to use the credibility I have built in my experience and career to that purpose.

Whats the biggest professional risk youve ever taken?

Starting and growing my own company was certainly the biggest professional risk, but it was well worth the reward. Prior to my time at Xator, I left my job working for a series of solid defense contractors and joined with two partners to build and grow InCadence. For 10 years, we built InCadence, and as president of the company, I saw first-hand the highly competitive environment of launching and growing a startup.

A big key to our success was our focus on technology-differentiated solutions, especially in the field of biometrics and identity, which is one of my major technical competencies. To be able to build a successful company, and to see it continue to thrive as a part of Xator Corp., has been a great reward for all the risks of being responsible, for all aspects of maintaining and growing a business and keeping key technical talent constantly innovating and delivering for our customers.

Looking back at your career, what are you most proud of?

I am most proud of having designed and coded, from the first line of code, the Biometrics Automated Toolset system, which I started writing when I was just out of the Army and just 29 years old. I had transitioned as an Army captain to a contractor working at the Battle Lab at the Army Intelligence Center, and I had a fantastic boss, Lt. Col. Kathy De Bolt, who asked me to build a biometrics system from the ground up.

That work took on a life of its own, being used in Kosovo, Iraq and Afghanistan. It is still an Army program of record system today and is the first digital biometrics system ever deployed on the battlefield.

From that, I built an exciting career and team of colleagues that led to where I am today, to include the success with the newest generation of biometric technologies at Xator. I know that the BAT system was indispensable to operations in support of our national security, and I still regularly have soldiers and Marines come up to me today and tell me stories of how they used BAT overseas.

More:
Chief Officer Awards Finalist Anthony Iasso: 'Never Stop Learning, and Never Stop Teaching' - WashingtonExec