Weapons of Math Destruction – Book Review

Imagine a world where hidden data systems are making decisions that affect your life and you can’t see the math behind them. The marriage of big data and algorithms is real and it has serious consequences. They’re called weapons of math destruction and we’ll explore how to protect your family today!

Ai Parenting Live is a judgement free community moving from screen time to quality time and our motto is don’t sedate, relate to create. 

Today we’re going to lift the curtain on big data by exposing the disturbing reality behind its use through the book Weapons of Math Destruction by Cathy O’Neil. This book takes a look at the algorithms and equations that shape our lives-from education, criminal justice, and hiring decisions to how people get priced for their insurance, and what kind of deals we get recommended to us on Amazon. 

The three main topics that we’ll explore today are:

1. Algorithms in Hiding
2. Critical Thinking
3. Gaming the System

Read to dive into it?

I’m going to share three ways that algorithms hide in plain sight. The first is to stop the creepiness but not the creeping, the second is targeting specific groups of people, and the third is the mathematical magic show. 

In my series called who created our Artificial Unconsciousness I spoke about how people like Target dad found the recommendations of Ai creepy and this resulted in the need for Ai recommendations to be hidden among other ads so it didn’t raise suspicion. 

The first method is to stop the creepiness but not the creeping. 

History repeats itself, even before Target dad there was already a history of hiding data and algorithms that automatically make decisions that have a profound impact on our lives. This is what Cathy O’Neil refers to as Weapons of Math Destruction (or WMDs) in her book which you can find on Amazon. Cathy was a data scientist for some of the largest hedge funds during the sub-prime mortgage crisis who saw first-hand the impact of mathematical models on people and felt compelled to write a book to help us understand what is really going on. 

Many workers are subject to Weapons of Math Desctruction or WMDs without even knowing it. Even if a worker doesn’t use social media, or freelancing Ai like Fiverr or Upwork they may have taken an IQ test or a big 5 personality test. IQ tests were often used by employers to determine whether or not people would be suitable for work and it was only during the arrival of the the Americans with Disabilities Act (ADA) that this became this practice of using IQ for employment became illegal.

This should be no surprise, as we’ve seen with cookie tracking today, when the public becomes aware of Weapons of Math Destruction (WMDs) what typically happens is that lawmakers change the rules and regulations. This does not mean that they stop using WMDs, instead they are often replaced with inferior pseudo-anonymous systems that are even more harmful. I had previously talked about cookies being replaced by session IDs, and ultimately the Federated Learning of Co-Horst (FLoC) where it just records an anonymous Ai model of your unconscious desires which you have little knowledge of and even less ability to argue against. Things get worse over time rather than better.

IQ tests were replaced with job applicant screening systems like Kronos which required potential candidates to do a big five personality test before hiring. The big five personality test looks for extraversion, neuroticism, openness to experience, agreeableness, and conscientiousness. 

The problem with these personality tests is that they heavily bias against the people who are low-income or neuro a-typical. For example, a recent Vanderbilt University Grad was unable to get a cashier job even though he had an outstanding GPA at a private research university. The reasons is that this grad was bipolar and found his resume red flagged by employers and even when he applied to other jobs he found that other employers had also red flagged his application. This grad likely score worse for neuroticism even though his performance in University was exemplary. Studies have shown that these tests are worse predictors of performance than IQ tests. Since IQ tests are outlawed, employers continue to use the next best WMD that produces good enough results. 

The key with any model is with the ability to improve it. For example, models are used in baseball to determine which stats are really important for the next draft pick. If a player is passed for a spot with the New York Mets and becomes the MVP for the Oakland A’s the people who are making that model would have to go back and look at why the model made this mistake and they would have to improve their model. 

The problem with many weapons of math destruction is that they make good enough decision to be useful for companies and there isn’t a financial incentive to improve them. A baseball contract could be worth millions, but what’s the cost of excluding some people from the hiring process? Almost none.

Public outrage seems to be one of the few ways that result in changes to WMDs. Any system that makes automatic decisions should be subject to some level of transparency of where they get their data and some level of scrutiny of wether the algorithms are representative of what we want to see in our society. The good news is that on April 21st we saw proposed European Union legislation that required big tech to more transparency and human oversight into high-risk Ai systems. [2]

The second way algorithms hide is by targeting specific groups of people. We saw this in the 2018 election that Fake News posts on social media were sufficient to stop specific black communities from voting.  

Usually our desires are revealed through our searches and what we spend our time watching. This allows companies to target specific groups of people that are most likely to buy. Customer pain points can be searched for and the quick fix is near by. 

Cathy says that “desperation is gold.” Consider Educational loans: The US loans up to 90% of the tuition for colleges. Companies spent billions in recruitment of the most vulnerable and lowest self esteem to take up university and collect this crippling debt. They would tout the prestige of attending a private university compared to a local safe school even though the stats showed that those from private universities made less. Marketers found that these ads didn’t work as well on the wealthy, so they continued to target the poorest and lowest self esteem.

Sub-prime mortgages were marketed to church pastors as a way to afford a home for those with the lowest income, but at the highest lending rates. A caretaker making <$30K/yr could take out a $700K loan to buy a home on speculation that the price would increase and that they could sell the home in a few years. When the housing crash came these people had no choice but to default on their loan.

The third way they hide is the mathematical magic show.

Mathematics and technology is often used to impress clients rather than to protect them against risk. This was certainly the case for mortgage-backed security‘s during the housing crisis. The complexity of Credit Default Swaps and Mortgage back securities was too complex for most people to understand at the time. They also created the perception that these loans were safe under the label of AAA rated loans.  Rating agencies knew that they had to play ball otherwise companies would bring their business to competitors for a better rating but consumers were not aware that these ratings were more a rubber stamp.

There was very little consequence for the companies whose WMD hurt large numbers of people. Many of the companies that sold sub-prime mortgages had already passed the risk on to other investors through mortgage backed securities and credit default swaps. Years after the crisis very little has changed in practice. If anything, most financial institutions felt emboldened that they were too big to fail, they now knew that with the right amount of lobbying governments would bail them out. 

Almost 40 years ago President Ronald Reagan delivered a report known as a nation at risk That stated from 1963 to 1980 SAT scores had been declining in the United States. The report blamed teachers for the decline and called for a commission to assess the “quality of teaching and learning” at the primary, secondary, and post-secondary levels. One of the authors James J Harvey wrote the now famous quote “the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people… If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war.” [3]

However the report itself which fundamentally changed education in the United States was flawed to begin with. While it was true that SAT scores were declining during that time much of the decline was due to lower income students also taking up the SAT exams. Sandia Laboratories Systems Scientists who looked at the average SAT scores among different income brackets found that SAT scores had increased in every income bracket between the periods of 1963 to 1980. 

Numbers are often used to demonstrate that decisions are made based on facts. They are objective should not be questioned. But without the ability to critically think about the numbers most of the population will just believe what numbers are given to them and they won’t think to question those numbers. This is why the poorest and most vulnerable are most at risk of being left behind by systems like AI. We need data transparency and the right to asking how these mathematical decisions are made. 

Teachers in Texas were evaluated using a WMD know as the value added model. It looked at student’s past test scores to predict their student’s future test scores. Any difference between the prediction and the results was assumed to be due to the teacher and the school, rather than the student’s ability or their family’s socio-economic status. 

A Texas primary teacher was to embarrassed about his 2/100 score to share with his colleagues until he learned that others had also received the same score. His tenure and relationships built over time had saved him but this WMD has real implications when decisions were made to fire the bottom 2% of all teachers each year. A good teacher in a poor neighbour would become collateral damage of this weapon of math destruction. 

Ironically, the following year that same teacher got a 94 out of 100 score by teaching a class of regular students. He concluded that the poor score in the first year who is due to his work with special educational need (SEN) students and gifted and talented education (GATE) students. 

The SEN students had slower progress during the year and the GATE students were already scoring quite high so would make less progress than the average student. Handling exceptions remains something that we need to rely on people to do. This is what makes Weapons of Math Destruction so dangerous for our society. It’s not just stripping away reality from the model but it’s also stripping away our humanity from the model.

So how do we fight back? 

Let’s look at how people game the system. 

I mentioned that the problem with models is that they are only an approximation of the real world. As a result, if you know the model you can game the system. 

How many of you have every looked at University rankings? The annual US News college and university rankings are often a top priority for many colleges and universities.  

WMDs create unfair systems that don’t always benefit the broader society. If we revisit US News, the original evaluation was tweaked until the Ivy League Universities bubbled up to top. This meant that the University ranking WMD would focus heavily on research publications, technology innovation, and the success of their graduates. What it did not consider however was the cost of tuition, or their income return on their education (RoE).

The result is a ranking system that is useful for the top 1% but hurts the other 99% who cannot afford to attend universities at any cost.  

Speaking of the top 1%, millions are spent every year on gaming the US news ranking. This gaming does little more than improve the ranking and tuition of the University rather than deliver returns for their actual students. 

For example in 2014, King Abdullah University in Dubai managed to land in seventh place worldwide in the US News ranking. Just behind Harvard and ahead of Cambridge and MIT.  

It understood that the US News ranking model used both publications and researcher’s authority or H-index in order to determine a universities ranking. So the solution was simple, pay top researchers $70K a year to spend 3 weeks in a 5 star hotel in Dubai and change their affiliation to King Abdullah. [4] To be clear, this hack has been patched and they currently rank 121 worldwide [5].

Let’s do a fun exercise of coming up with our own model for an insurance system. If we were to come up with a new model for insurance should we include race? I think most people would agree that that’s not a fair thing to include. 

Should we include the ZIP Code? Well it might seem fair first many will come to the conclusion overtime that this is going to result in lower income people paying more for insurance. 

The problem is that it’s very hard to come up with a system free of bias. While a first and a last name might seem obvious, even the time that someone applies for insurance might be enough to introduce bias into a system. 

When we build models we teach algorithms to discriminate, the only way to avoid this is to have others independently evaluate our systems and for there to be constant iteration. For example, Joy Boulamwini’s exploration into facial recognition bias against the highly-melanted resulted in some big tech companies abandoning the use of facial recognition at least for a time. 

So my main takeaways from this book was that Weapons of Math Destruction take in large amounts of data and make decisions that directly impact people’s lives with minimal critical thinking, and even less opportunity to argue for humane reasons for exceptions. This lack of critical thinking, and feedback makes these systems dangerous for our society. It preys on our poorest and most vulnerable, it fuels financial bubbles, stops people from getting jobs, results in longer prison sentences, and hurts the very people who are educating our children. While WMDs have existed for a long time, they are becoming more common place so it’s time to move beyond knowing if news is fake and move towards thinking critically about the math that holds us back. To see past the mathematical magic show, and game the systems for the good of us all.

References

1. Weapons of Math Destruction available on Amazon
2. New EU Regulation on Ai view blogpost
3. A nation at risk from Wikipedia
4. King Abdullah University hacks their US News ranking see article
5. King Abdullah current rating US News