What do the Hulk, Batman, and Spider-Man do when they need to get to work to solve a big problem? They transform into scientists. Calling Bullsh*t: The Art of Skepticism in a Data-Driven World, by Carl T. Bergstrom and Jevin D. West, shows which tools belong in your mental utility belt, and how to exercise that power with greater responsibility.
Central to the book’s thesis is Brandolini’s Principle, which states, “The amount of energy needed to refute bullsh*t is an order of magnitude bigger than [that needed] to produce it.” The authors show you how to avoid stepping in bullsh*t and how to clean it up.
Calling bullsh*t, then, can be defined as:
A performative utterance in which one repudiates something objectionable. The scope of targets is broader than the bullsh*t alone. You can call bullsh*t on bullsh*t, but you can also call bullsh*t on lies, treachery, trickery, or injustice.
With backgrounds in evolutionary biology and information research, the authors have put together a practical guide written for the casual reader and professional scientist alike. The writing is funny, insightful, and frequently illustrated. I originally got Calling Bullsh*t as an audiobook, but after listening for a few hours, I bought a hardcover copy, too! I wanted to see the graphs they analyze.
The book has interesting case studies from a wide variety of fields, like science, medicine, marketing, and politics. Calling Bullsh*t tackles systemic problems in scientific journal publishing, like when the results are irreproducible. Also, p-hacking, which is when a researcher retrofits a hypothesis to the data collected, forcing a conclusion which the initial data collection wasn’t designed to find. Also included is a fresh guide for understanding AI black boxes using basic critical thinking techniques, without the need for advanced software knowledge.
Here are some of the actions, characteristics, and considerations masterfully detailed in the book.
Use reductio ad absurdum: This is the natural logical extension of the speaker’s argument, particularly when it leads to an absurd situation, which the speaker would reject. It shows to the speaker and the audience that if you grant that their proposition is true, it leads to stupid results they hadn’t thought through. Simply extend their story a step further then they themselves imagined.
Find counterexamples: This is the most concise way to refute an argument. But to work, the counterexample must be a fitting instance of what the speaker is proposing. It must land perfectly, or the focus will shift off the speaker’s argument to how your counterexample doesn’t really capture their point. A well-constructed argument, capable of validity, is subject to this critique. Otherwise, the argument is unreasonable by design.
Provide analogies: Analogies work by relating new information to previously understood information. Analogies map the new information onto a familiar pattern or example of a relationship. Analogies are a tool for learning how a concept fits into things the audience already knows.
But it takes work to find a suitable analogy. An analogy which captures little of the point is prone to confuse and misdirect the audience. You also don’t want an analogy which is more complicated than the point you’re trying to illustrate. If the analogy is too simple, it may not reveal the inner workings of the argument. Good analogies are harder to make than people think.
Redraw figures: Check that a disputed diagram actually shows what its author claims. Frequently the graphics and charts we encounter online are misleading. They can be sloppily relaying information, or worse — designed to force an invalid conclusion. Graphics are especially effective for transmitting bullsh*t. Two basic habits to develop when checking a diagram are:
1) confirm that the axes are correctly labeled and scaled, and
2) make sure it’s the right type of graph.
Bar charts should almost always start the independent x-axis at zero. Line charts don’t need to. Why? “Bar graphs emphasize the absolute magnitude of values associated with each category, whereas a line graph emphasizes the change in the dependent variable (usually the y value) as the independent variable (usually the x value) changes.”
Employ additional scrutiny when reading a fancy chart. A stylized graph can foster totally unsupported conclusions about the substance of the data. Such a stylized graph is called a duck.
Check for a null hypothesis: When a hypothesized effect is wrong because reality shows the proposed variable has no effect, that scenario is called the null hypothesis. This is different from the scenario where the opposite of the hypothesis is true. We can make two different kinds of errors when evaluating a null hypothesis. A Type I Error is when the null is true but you rejected it, like a false negative — you reject something you should have accepted. A Type II Error is when the null is false but you accept it, like a false positive — you accept what you should have rejected.
Admit fault: Encourage trust from other people, disarm their ego, and most importantly check your own ego. If you hold yourself accountable when your hypothesis doesn’t pan out, you can better calibrate for the next time you make a claim. This practice will improve your ability to draw nuanced distinctions, ask the right questions, and be humble in the face of the complexity of the real world. If you skip this, you jeopardize your ability to learn from your mistakes, and over time you’ll lose touch with reality.
Be memorable: A clear idea will catch on more than a convoluted one.
Be correct: If you’re going to take a shot at someone’s claim, don’t miss. Put in the work to be worth the bullsh*tter’s and their audience’s attention. So at a minimum, get your facts right.
Be charitable: Avoid “strawman arguments” you can knock down easily by putting their argument in the best light. The measure of this is simple: the original speaker would say of your characterization of their point, “I couldn’t have put it better myself.” Mischaracterizing their points doesn’t refute them. So put yourself in their shoes and don’t give in to the temptation to call someone you disagree with evil or stupid. Their claim may work in their mind.
Be pertinent: Focus on what matters to the points being made.
Relevance: Being relevant will stop you from adding unnecessary complexity or trivia to the forum.
Speaker’s intention: Try to understand the bullsh*tter’s feelings and motivations. If you don’t consider their intentions, you can’t stop yourself from missing their point.
Audience: The bullsh*tter might not acknowledge your points. It may be unreasonable to expect to bring the adversary around to your point of view, so you may instead aim to win over the audience. Be aware of the power they may have over their audience.
When calling bullsh*t, remember that you’re calling attention to yourself, explicitly moving the direction of the discourse. The point is addressing the claim with skepticism and rigorous thinking. Saying something true and timely and helpful is hard enough. Calling Bullsh*t: The Art of Skepticism in a Data-Driven World is already being taught in schools and, in my opinion, is the best of its genre.
Every February, to help celebrate Darwin Day, the Science section of AIPT cranks up the critical thinking for SKEPTICISM MONTH! Skepticism is an approach to evaluating claims that emphasizes evidence and applies the tools of science. Every day this month we’ll be highlighting skepticism in pop culture and skepticism of pop culture.
Become a patron today to get exclusive perks, like access to our exclusive Discord community and our monthly comic book club, ad-free browsing on aiptcomics.com, a physical trade paperback sent to your house every month, and more!