In 2013, the University of Texas at Austins computer science department began usinga machine-learning system called GRADE to help make decisions about who gets into its Ph.D. program -- and who doesnt. This year, the department abandoned it.

Before the announcement, which the department released in the form of a tweet reply, few had even heard of the program. Now, its critics -- concerned about diversity, equity and fairness in admissions -- say it should never have been used in the first place.

Humans code these systems. Humans are encoding their own biases into these algorithms, said Yasmeen Musthafa, a Ph.D. student in plasma physics at the University of California, Irvine, who rang alarm bells about the system on Twitter. What would UT Austin CS department have looked like without GRADE? Well never know.

GRADE (which stands for GRaduate ADmissions Evaluator) was created by a UT faculty member and UT graduate student in computer science, originally to help the graduate admissions committee in the department save time. GRADE predicts how likely the admissions committee is to approve an applicant and expresses that prediction as a numerical score out of five. The system also explains what factors most impacted its decision.

The UT researchers who made GRADE trained it on a database of past admissions decisions. The system uses patterns from those decisions to calculate its scores for candidates.

For example, letters of recommendation containing the words best, award, research or Ph.D. are predictive of admission -- and can lead to a higher score -- while letters containing the words good, class, programming or technology are predictive of rejection. A higher grade point average means an applicant is more likely to be accepted, as does the name of an elite college or university on the rsum. Within the system, institutions were encoded into the categories elite, good and other, based on a survey of UT computer science faculty.

Every application GRADE scored during the seven years it was in use was still reviewed by at least one human committee member, UT Austin has said, but sometimes only one. Before GRADE, faculty members made multiple review passes over the pool. The system saved the committee time, according to its developers, by allowing faculty to focus on applicants on the cusp of admission or rejection and review applicants in descending order of quality.

For what its worth, GRADE did appear to successfully save the committee time. In the 2012 and 2013 application seasons, developers said in a paper about their work, it reduced the number of full reviews per candidate by 71percent and cut the total time reviewing files by 74percent. (One full review typically takes 10 to 30 minutes.) Between the years 2000 and 2012, applications to the computer science Ph.D. program grew from about 250 to nearly 650, though the number of faculty able to review those applications remained mostly constant. In the years since 2012, the number of applications has reached over 1,200.

The universitys use of the technology escaped attention for a number of years, until this month, when the physics department at the University of Maryland at College Park held a colloquium talk with the two creators of GRADE.

The talk gained attention on Twitter as graduate students accused GRADEs creators of further disadvantaging underrepresented groups in the computer science admissions process.

We put letters of recommendation in to try to lift people up who have maybe not great GPAs. We put a personal statement in the graduate application process to try to give marginalized folks a chance to have their voice heard, said Musthafa, who is also a member of the Physics and Astronomy Anti-Racism Coalition. The worst part about GRADE is that it throws that out completely.

Advocates have long been concerned about the potential for human biases to be baked into or exacerbated by machine-learning algorithms. Algorithms are trained on data. When it comes to people, what those data look like is a result of historical inequity. Preferences for one type of person over another are often the result of conscious or unconscious bias.

That hasnt stopped institutions from using machine-learning systems in hiring, policing and prison sentencing for a number of years now, often to great controversy.

Every process is going to make some mistakes. The question is, where are those mistakes likely to be made and who is likely to suffer as a result of them? said Manish Raghavan, a computer science Ph.D. candidate at Cornell University who has researched and written about bias in algorithms. Likely those from underrepresented groups or people who dont have the resources to be attending elite institutions.

Though many women and people who are Black and Latinx have had successful careers in computer science, those groups are underrepresented in the field at large. In 2017, whites, Asians and nonresident aliens received 84percent of degrees awarded for computer science in the United States.

At UT, nearly 80percent of undergraduates in computer science in 2017 were men.

Raghavan said he was surprised that there appeared to be no effort to audit the impacts of GRADE, such as how scores differ across demographic groups.

GRADEs creators have said that the system is only programmed to replicate what the admissions committee was doing prior to 2013, not to make better decisions than humans could. The system isnt programmed to use race or gender to make its predictions, theyve said. In fact, when given those features as options to help make its predictions, it chooses to give them zero weight. GRADEs creators have said this is evidence that the committees decisions are gender and race neutral.

Detractors have countered this, arguing that race and gender can be encoded into other features of the application that the system uses. Womens colleges and historically Black universities may be undervalued by the algorithm, theyve said. Letters of recommendation are known to reflect gender bias, as recommenders are more likely to describe female students as caring rather than assertive or trailblazing.

In the Maryland talk, faculty raised their own concerns. What a committee is looking for might change each year. Letters of recommendation and personal statements should be thoughtfully considered, not turned into a bag of words, they said.

Im kind of shocked you did this experiment on your students, Steve Rolston, chair of the physics department at Maryland, said during the talk. You seem to have built a model that builds in whatever bias your committee had in 2013 and youve been using it ever since.

In an interview, Rolston said graduate admissions can certainly be a challenge. His department receives over 800 graduate applications per year, which takes a good deal of time for faculty to evaluate. But, he said, his department would never use a tool like this.

If I ask you to do a classifier of images and youre looking for dogs, I can check afterwards that, yes, it did correctly identify dogs, he said. But when Im asking for decisions about people, whether it's graduate admissions, or hiring or prison sentencing, theres no obvious correct answer. You train it, but you dont know what the result is really telling you.

Rolston said having at least one faculty member review each application was not a convincing safeguard.

If I give you a file and say, Well, the algorithm said this person shouldnt be accepted, that will inevitably bias the way you look at it, he said.

UT Austin has said GRADE was used to organize admissions decisions, rather than make them.

"It was never used to make decisions to admit or reject prospective students, asat least one faculty member directly evaluates applicants at each stage of the review process," a spokesperson for the Graduate School said via email.

Despite the criticism around diversity and equity, UT Austin has said GRADE is being phased out because it is too difficult to maintain.

Changes in the data and software environment made the system increasingly difficult to maintain, and its use was discontinued, the spokesperson said via email. The Graduate School works with graduate programs and faculty members across campus to promote holistic application review and reduce bias in admissions decisions.

For Musthafa, the fact that GRADE may be gone for good does not impact the existing inequity in graduate admissions.

The entire system is steeped in racism, sexism and ableism, they said. How many years of POC computer science students got denied [because of this]?

Addressing that inequity -- as well as the competitiveness that led to the creation of GRADE -- may mean expanding committees, paying people for their time and giving Black and Latinx graduate students a voice in those decisions, they said. But automating cannot be part of that decision making.

If we automate this to any extent, its just going to lock people out of academia, Musthafa said. The racism of today is being immortalized in the algorithms of tomorrow.

Original post:

U of Texas will stop using controversial algorithm to evaluate Ph.D. applicants - Inside Higher Ed

On Dec. 14, the Sanford and Susan Greenberg Prize to End Blindness honored 13 scientistswho have made extraordinary headway in the worldwide battle against blindness. Among them was James G. Fujimoto, the Elihu Thomson Professor of Electrical Engineering within MITs Department of Electrical Engineering and Computer Science (EECS).

Recipients of the Greenberg Prize are honored in two categories: the Outstanding Achievement Prize, highlighting strides toward treating and curing blindness, and the Visionary Prize, providing funding for scientists whose research exhibits significant potential in ending this debilitating condition. Fujimoto, a principal investigator in the Research Laboratory of Electronics (RLE), was awarded the Visionary Prize for his research, which focuses upon the areas of biomedical imaging, optical coherence tomography, and advanced laser technologies and applications.

As noted recently in National Geographic, the Greenberg Prize originates in the personal experience of Sanford Greenberg, who lost his vision as a young man and subsequently vowed to spend the rest of his life working to ensure that no one else would have to share the devastation of his experience. Inspired by milestone scientific efforts such as the moon landing and the development of the polio vaccine, Greenberg has set an ambitious goal of total worldwide eradication of blindness, regardless of underlying cause. To that end, he and the other members of the governing prize committee have set out to identify and connect scientists worldwide who are making critical headway in the battle against the condition.

The award ceremony which can be viewed online featured celebrity appearances, musical performances, the unveiling of a sculpture created by Frank Stella in honor of the prize, and a tribute to the late U.S. Supreme Court Justice Ruth Bader Ginsburg, a longtime supporter of Greenbergs philanthropic work.

A principal investigator in RLE and adjunct professor of ophthalmology at Tufts University School of Medicine, James Fujimoto earned his SB, SM, and PhD in EECS from MIT in 1979, 1981, and 1984 respectively. He joined the MIT faculty in 1985, and has been conducting research ever since.

Continue reading here:

James Fujimoto wins the Visionary Prize from the Greenberg Prize to End Blindness - MIT News

New London Ah, the 16-year-old about to get her license. The cognitive wheels begin to grind, eventually arriving here: What good is a license without a car, right? Thus, the scheming begins.

Hit up mom or dad? Maybe start taking the trash out without even being asked? Broker a deal with the grandparents instead?

Or you could be Nadya Murphy and make your own bombs.

Meet Nadya: an honor student, lacrosse player and junior at the New London Science and Technology Magnet High School. Her idea The Hot Chocolate Bomb has become a bit of a cottage industry of deliciousness, much to the delight of her customers. Not only is she making money toward her first car, but she's even donating some of it to the New London Community Meal Center.

"I found the whole Hot Chocolate Bomb thing on Tik Tok with my aunt," Murphy said. "We thought it would be fun to experiment with because we saw they were pretty trendy. It's an easy way to earn money. I really want that car."

The Hot Chocolate Bomb: a chocolate shell in a mug with hot chocolate mix, marshmallow, hot milk and various toppings. Result: a ball of hot chocolate that kids (and adults) love.

"I bought some to take to Thanksgiving," New London Board of Education member Elaine Maynard-Adams said. "The kids loved them. Then at night, we made the adult version with homemade Kahlua. Let me just tell you how good they are."

Murphy's business originates on her "Nadya's Hot Chocolate Bombs" Facebook page. At specific times, Murphy posts a link as to dates and times they'll be on sale. Everything is sold out of their house contactless now in the age of COVID. They even deliver to New London, Groton and Waterford.

"We sold a little over 350 last week," Murphy said. "We sold out in an hour. Our last sale (before that) was at the New London Holiday Market pop up sale and we gave 10 percent ($200) to community meal center. My family and I were thinking about the holidays and how fortunate we are. We thought giving back would be a nice thing to do."

There's even an educational component to this that Murphy didn't necessarily see coming. She's in the computer science pathway at school.

"I actually take an accounting class," Murphy said. "A couple of concepts I've learned in accounting have become applicable to real life. I'm learning the basic steps of having a business, how sales work and having different accounts. Transferring money, transactions, things like that I'm seeing it in real life and it's helping me in class."

Murphy has to pay for chocolate milk used to make the shells, hot chocolate mix, marshmallows, sprinkles, candy cane bits, bags to package them and business cards. Not cheap.

"This wouldn't be possible without my aunt, Brenda De Los Santos," Murphy said. "She helps me manage everything from my Facebook page to my business cards. Once I get my car I will be chauffeuring her around for the next year probably."

This story hits virtually every happy note on the staff. Hard not to root for this kid. She's working for everything she wants and has, even if unwittingly, discovered a way to tie school to real life. And her product hot chocolate is something we can all pretty much agree is yummy.

"We're thinking of doing a couple more sales," Murphy said. "The demand will go down as it gets warmer. People will have to check the page to know. It's been fun. And a lot of work. But I know cars aren't cheap. That would be a pretty large burden to have my family try to pay for. So why not try to make it myself?"

This is the opinion of Day sports columnist Mike DiMauro

Follow this link:

Nadya's Hot Chocolate Bombs: yummy for the tummy - theday.com

DURHAM, N.C. -- The artificial intelligence behind self-driving cars, medical image analysis and other computer vision applications relies on whats called deep neural networks.

Loosely modeled on the brain, these consist of layers of interconnected neurons -- mathematical functions that send and receive information -- that fire in response to features of the input data. The first layer processes a raw data input -- such as pixels in an image -- and passes that information to the next layer above, triggering some of those neurons, which then pass a signal to even higher layers until eventually it arrives at a determination of what is in the input image.

But heres the problem, says Duke computer science professor Cynthia Rudin. We can input, say, a medical image, and observe what comes out the other end (this is a picture of a malignant lesion, but its hard to know what happened in between.

Its whats known as the black box problem. What happens in the mind of the machine -- the networks hidden layers -- is often inscrutable, even to the people who built it.

The problem with deep learning models is theyre so complex that we don't actually know what theyre learning, said Zhi Chen, a Ph.D. student in Rudins lab at Duke. They can often leverage information we dont want them to. Their reasoning processes can be completely wrong.

Rudin, Chen and Duke undergraduateYijie Bei have come up with a way to address this issue. By modifying the reasoning process behind the predictions, it is possible that researchers can better troubleshoot the networks or understand whether they are trustworthy.

Most approaches attempt to uncover what led a computer vision system to the right answer after the fact, by pointing to the key features or pixels that identified an image: The growth in this chest X-ray was classified as malignant because, to the model, these areas are critical in the classification of lung cancer. Such approaches dont reveal the networks reasoning, just where it was looking.

The Duke team tried a different tack. Instead of attempting to account for a networks decision-making on a post hoc basis, their method trains the network to show its work by expressing its understanding about concepts along the way. Their method works by revealing how much the network calls to mind different concepts to help decipher what it sees. It disentangles how different concepts are represented within the layers of the network, Rudin said.

Given an image of a library, for example, the approach makes it possible to determine whether and how much the different layers of the neural network rely on their mental representation of books to identify the scene.

The researchers found that, with a small adjustment to a neural network, it is possible to identify objects and scenes in images just as accurately as the original network, and yet gain substantial interpretability in the networks reasoning process. The technique is very simple to apply, Rudin said.

The method controls the way information flows through the network. It involves replacing one standard part of a neural network with a new part. The new part constrains only a single neuron in the network to fire in response to a particular concept that humans understand. The concepts could be categories of everyday objects, such as book or bike. But they could also be general characteristics, such as such as metal, wood, cold or warm. By having only one neuron control the information about one concept at a time, it is much easier to understand how the network thinks.

The researchers tried their approach on a neural network trained by millions of labeled images to recognize various kinds of indoor and outdoor scenes, from classrooms and food courts to playgrounds and patios. Then they turned it on images it hadnt seen before. They also looked to see which concepts the network layers drew on the most as they processed the data.

Chen pulls up a plot showing what happened when they fed a picture of an orange sunset into the network. Their trained neural network says that warm colors in the sunset image, like orange, tend to be associated with the concept bed in earlier layers of the network. In short, the network activates the bed neuron highly in early layers. As the image travels through successive layers, the network gradually relies on a more sophisticated mental representation of each concept, and the airplane concept becomes more activated than the notion of beds, perhaps because airplanes are more often associated with skies and clouds.

Its only a small part of whats going on, to be sure. But from this trajectory the researchers are able to capture important aspects of the networks train of thought.

The researchers say their module can be wired into any neural network that recognizes images. In one experiment, they connected it to a neural network trained to detect skin cancer in photos.

Before an AI can learn to spot melanoma, it must learn what makes melanomas look different from normal moles and other benign spots on your skin, by sifting through thousands of training images labeled and marked up by skin cancer experts.

But the network appeared to be summoning up a concept of irregular border that it formed on its own, without help from the training labels. The people annotating the images for use in artificial intelligence applications hadnt made note of that feature, but the machine did.

Our method revealed a shortcoming in the dataset, Rudin said. Perhaps if they had included this information in the data, it would have made it clearer whether the model was reasoning correctly. This example just illustrates why we shouldnt put blind faith in black box models with no clue of what goes on inside them, especially for tricky medical diagnoses, Rudin said.

The teams work appeared Dec. 7 in the journal Nature Machine Intelligence.

This research was supported by funding from MIT-Lincoln Laboratory and the National Science Foundation (OAC-1835782).

CITATION: "Concept Whitening for Interpretable Image Recognition,"Zhi Chen, Yijie Bei and Cynthia Rudin. Nature Machine Intelligence, Dec. 7, 2020. DOI: 10.1038/s42256-020-00265-z

Continue reading here:

Accurate Neural Network Computer Vision Without The 'Black Box' - Duke Today