What AI Can and Can’t Do? In-Depth Review of “AI Snake Oil” by Arvind Narayanan & Sayash Kapoor

What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference

Discover a comprehensive summary and review of “AI Snake Oil” by Arvind Narayanan and Sayash Kapoor. Learn how this essential 2025 book exposes AI industry myths, explains real-world risks, and offers actionable insights for navigating artificial intelligence at work and home.

Ready to cut through the hype and make smarter decisions about AI? Continue reading for expert insights from “AI Snake Oil”-and learn how to spot the difference between genuine innovation and overblown promises.

Recommendation

It’s time to differentiate between AI’s real capacities and the overhyped promises of Big Tech, say Princeton University computer science professor Arvind Narayanan and PhD candidate Sayash Kapoor. Just as salesmen once sold snake oil as a “miracle cure” when, in reality, it had no medical benefits, people today are exaggerating AI’s potential. For instance, AI tools cannot give humanity perfect insight into future events — but they can lead to poor decision-making. Narayanan and Kapoor call on you to fight for the AI future you want, rather than accepting the future tech companies are trying to sell you.

Take-Aways

• Predictive AI’s abilities are vastly exaggerated.
• Predictive AI often fails for five key reasons, leading to harmful real-world consequences.
• It’s time to accept the limits of predictive AI while improving predictive accuracy when feasible.
• Generative AI has serious risks, requiring oversight, fair labor practices, and public education.
• AI used in content moderation struggles with language, context, and disinformation.
• Mitigate the risks of AI by strengthening democratic institutions.
• Myths about AI’s potential abound because industry insiders overhype its performance.
• Coming to grips with the failures of AI to perform as promised requires mindset shifts.

Summary

Predictive AI’s abilities are vastly exaggerated.

Decision-makers, governments and companies are pushing to replace human decision-making with predictive AI analytics. There’s just one catch: AI tools don’t always work as advertised. For example, Medicare providers in the United States have started turning to predictive AI to estimate how long patients will spend in the hospital. However, these estimates are not always accurate and can undercut the quality of care patients receive. In one alarming case, an AI tool predicted that an 85-year-old would only need to spend 17 days in the hospital. The patient’s health insurance company, taking the prediction as fact, stopped funding treatment after that many days, even though the patient still couldn’t walk without assistance and remained in extreme pain.

“Predictive AI is currently used by both companies and governments, but that doesn’t mean it works.”

False promises about the potential of AI to make accurate predictions abound. For example, both Scientific American and Axios published stories touting a 2023 paper that contained the false assertion that AI could predict future hit songs with nearly 100% accuracy. However, the study’s results were spurious due to data leakage: This means that the researchers tested the tool using the same or similar data used in training it, thus greatly boosting the accuracy of their results. Systemic reviews of machine-learning research have concluded that the majority of this research is similarly flawed. Yet researchers continue to tout machine learning as a silver bullet for high-stakes situations, from predicting disease to assessing the likelihood of war.

Predictive AI often fails for five key reasons, leading to harmful real-world consequences.

Predictive AI often fails for the following five reasons:

1. A good prediction can lead to a half-baked conclusion — AI may detect patterns in the data that are accurate but which, barring closer scrutiny, could result in poor decisions. For example, in one study an AI tool found that patients with asthma experienced fewer complications from pneumonia, indicating — at first glance — that these patients should receive a lower level of care. However, asthmatic patients are actually more vulnerable to pneumonia. They only experience fewer complications due to the greater level of care they receive.
2. People may try to “game” AI systems — Job candidates may try to cram their applications with keywords when applying via automated hiring systems. Likewise, candidates may try to put bookshelves in the back of videos in applications, as people are learning that AI models often rely on arbitrary factors, such as the presence of books in a video background, when screening candidates.
3. Over-reliance on AI occurs without oversight — People have a tendency to trust AI, even when it’s wrong. For example, in 2013, the Netherlands replaced human decision-making with an AI system designed to detect welfare fraud. Their algorithm wrongly flagged 30,000 parents, using nationality as a predictor of fraud, disproportionately targeting individuals of Turkish, Moroccan, or Eastern European nationality.
4. Training data may not be representative enough — AI decisions rely on training data. A given AI tool’s accuracy thus depends on how well its training data represents the target population. For example, Allegheny County, Pennsylvania designed an AI system to predict child maltreatment risk, using public welfare data as its training data. Given that wealthier families weren’t represented in the training data, the county disproportionately targeted lower-income families for investigation and child removal.
5. Predictive AI may exacerbate inequities — People often use predictive AI in ways that disproportionately harm marginalized communities. After the implementation of the 2010 Affordable Care Act, insurers pressured hospitals to lower prices, leading providers to use AI to identify high-risk patients and allocate preventative care. Optum’s Impact Pro wrongly classified white patients as higher risk than Black patients because it equated higher health care spending with health needs, resulting in poorer care for Black patients.

It’s time to accept the limits of predictive AI while improving predictive accuracy when feasible.

As data availability has grown, social scientists have begun turning to machine learning for predictive purposes. For example, in 2015, Princeton University sociologists designed a large-scale study in the hope of predicting children’s life outcomes in the Fragile Families Challenge. The researchers used data taken over a 15-year period of 4,000 children across the United States, attempting to predict outcomes, such as grade point average, based on factors such as their mother’s educational level and previous academic performance. Unfortunately, their model’s results were only slightly better than random guessing. Unlike predicting the movement of planetary bodies, which scientists can do with astounding accuracy, there are no clearly defined laws governing human behavior and life outcomes, which are shaped by complex and often unpredictable factors.

“Change starts by challenging the deployment of harmful AI tools in your workplace, neighborhood, and community.”

Despite the fact that nobody’s clearly proven the efficacy of predictive AI in the social sciences, companies continue to sell AI models that guide people in making decisions that impact lives. In the criminal justice system in the United States, some states have used the predictive AI model COMPAS, a risk assessment tool, to determine whether an offender would re-offend. Investigative journalist Julia Angwin and her team at ProPublica found that COMPAS demonstrated significant racial bias, disproportionately classifying Black people as more likely to commit future crimes than white offenders. With human lives at stake, it’s vital that people start respecting the limits of prediction and making efforts to improve predictions where possible by improving the quality and quantity of training data.

Generative AI has serious risks, requiring oversight, fair labor practices, and public education.

There are some things AI does well, such as generating and classifying images. When you’re out for a walk, for example, you can take a photo of a tree and use an AI tool that leverages deep neural networks to find out the species — which is incredibly useful. Early research shows that knowledge workers can also benefit from generative AI, using it for tasks such as assisting with writing and programming. But generative AI comes with substantial risks. For example, people frequently use generative AI to create or manipulate images of people online. The majority of these “deepfakes” are pornographic in nature, created without the subject’s consent. Moving forward, launching public education initiatives that help people differentiate between deepfakes, which can spread disinformation, and trustworthy news sources is also advisable.

“The most serious harm from generative AI, in our view, is the labor exploitation that is at the core of the way it is built and deployed today.”

Companies contract human data annotators from poorer countries to classify toxic content in generative AI models. These workers are typically overworked, underpaid, and experience considerable trauma due to the toxic nature of the content to which they’re exposed. It’s difficult to estimate how many data annotators there are around the world, as companies operate without transparency, pushing workers to sign confidentiality agreements. It’s time for a new labor movement to improve conditions for those working in AI annotation work. AI researcher Adrienne Williams and her co-authors published a paper calling for unionization, the formation of transnational alliances, and the fostering of solidarity between wealthier tech workers and their lower-wage colleagues.

AI used in content moderation struggles with language, context, and disinformation.

While social media companies hire content moderators who are familiar with local contexts in wealthier nations, such as the United States and European countries, they don’t do the same for many countries in the Global South. For these less-affluent locales, social media companies often use AI for content moderation. However, AI tools often fail to protect users from exposure to hateful or misleading posts — or incorrectly flag content as problematic. As part of the moderation process, Facebook and other social media platforms also use AI to translate non-English content to English, but these translations tend to lack nuance because they are devoid of understanding of local contexts.

“When content moderation AI is portrayed as a solution to the moral and political dilemmas of social media rather than merely a way for companies to save money, it becomes a form of snake oil.”

There are two main types of AI-based content moderation: “fingerprint matching,” in which AI detects previously identified prohibited content, and machine learning, in which it identifies new types of harmful content via pattern detection. Fingerprint matching is limited in that it can only flag previously flagged content. Machine learning can go further, flagging new content based on its training data, but this, too, can be problematic in that it can lead to the suppression of dissenting ideas — those that contradict the training data or otherwise deviate from the status quo. It might, for example, flag political dissent or a novel scientific discovery as misinformation.

Moreover, machine learning models can’t determine if a statement is true or false; they only identify how previous statements were labeled, making them ineffective at assessing the accuracy of topics associated with disinformation, such as conspiracy theories about the COVID-19 vaccine. The prevalence of harmful content on social media isn’t inevitable — it’s by design: Platforms use algorithms that reward engagement and attention, as opposed to amplifying content that’s actually valuable or useful to people.

Mitigate the risks of AI by strengthening democratic institutions.

While many express existential fears about AI — worrying, for example, that a superintelligent AI may somehow go rogue — these concerns are overblown. They also grant AI too much power and, as a result, lead people to accept, rather than challenge, the often problematic ways in which individuals, companies, and governments deploy AI systems. Rogue AI systems are unlikely to become a real existential threat because there’s no reason to believe that people acting with the help of other AI systems will be less powerful than the systems that go rogue. The real existential threat will come from human actors and how they choose to use AI systems. For example, bad actors could use AI to seize control of nuclear plants.

“In portraying the technology as all-powerful, critics overstate its capabilities and underemphasize its limitations, playing into the hands of companies who would prefer less scrutiny.”

Keeping AI from falling into the hands of bad actors or preventing the design of AI models that might help those with malicious aims aren’t realistic goals. Instead, humanity must defend itself from specific threats. For example, strengthening democratic institutions will help reduce the harms caused by AI-generated disinformation and deepfakes. While this may seem like a “gargantuan” task, it’s a vital one, as it’s impossible “to put AI back in a bottle.”

Myths about AI’s potential abound because industry insiders overhype its performance.

In 2016, computer scientist Geoffrey Hinton asserted: “People should stop training radiologists now. It’s just completely obvious that within five years, deep learning is going to do better than radiologists.” These predictions didn’t come to fruition, and there’s a global radiologist shortage today. OpenAI also prompted many to assume that AI will replace lawyers by announcing that their GPT-4 had tested in the 90th percentile on the bar exam. However, lawyers need more than subject knowledge — they need real-world skills, which an AI lacks. There’s a general trend within the AI industry of being dismissive of the domain expertise that industry professionals hold.

“The trouble with AI hype stems from the mismatch between claims and reality.”

Part of the reason that myths about AI’s potential abound is that companies may manipulate accuracy metrics to fool the public into thinking AI models are more accurate and capable than they are. For example, when measuring accuracy in image classification, companies use a metric called “top-N” accuracy: “N” refers to the number of guesses you allocate to an AI model to accurately identify an image. If you define N as three, then the model just needs to get it right in three guesses. Companies could make N as high as 10 though if they wanted to have better-sounding results, as most people wouldn’t think to check to see whether companies are using a high N and or a low N.

Coming to grips with the failures of AI to perform as promised requires mindset shifts.

Organizations, such as universities, have attempted to use AI to “optimize” processes — using a tool that scans social media accounts to identify students who could use mental health support, for example. But far too often, these experiments merely underscore AI’s limited ability to address problems it’s tasked with fixing. Organizations would be best advised to eschew optimization for simplicity and “modest efficiency gains.” Humans ought to be able to understand AI systems, which currently function like black boxes. Organizations may also want to consider “embracing randomness,” using “partial lottery” models to guide decision-making. For example, if a school is determining who gets a scholarship, it should create a pool of all the potential students who meet the requirements and then choose the winners randomly.

“We are not OK with leaving the future of AI up to the people currently in charge.”

Tech companies often make the assertion that self-regulation is the only viable option because governments move much more slowly than the pace of technological innovation and lack a deep understanding of their industry. But government workers succeed in regulating civil engineering with construction codes, despite lacking engineering backgrounds. Rather than fight regulation, it’s better to strengthen regulatory bodies and give them more resources and funding to prevent regulators from being controlled or overly influenced by the companies they’re supposed to oversee. While it’s important to avoid overzealous regulation, as it can stifle innovation, leaving the future of AI up to profit-driven tech companies isn’t in the public interest. Whether via artistic expression or community action, everyone has a role to play in shaping AI’s future.

About the Authors

Arvind Narayanan is the director of the Center for Information Technology Policy and a professor of computer science at Princeton. Sayash Kapoor is a PhD candidate with the Center for Information Technology Policy at Princeton University.