We’re all looking for ways to do work and life better.
And there's no shortage of experts willing to help us out. Experts that claim to have real research-based proof behind their ideas or systems.
But, unfortunately, most of them are full of crap. Or at least their data is full of crap.
Back in post #19 - Half Truths I warned you that many well-known business books were based on wobbly statistics. And in post #57 - Fake News In Your Presentation I shared that in the field of psychology experts have been unable to replicate 60% of the studies they’ve tested.
And now I'm circling back with an update because the crap is starting to hit the fan. Well-known, respected researchers are getting called to the carpet for playing games with their data.
POWER POSING FALLS TO P-HACKING
Here's a great article on the topic from New York Times columnist Susan Dominus. The article includes this important quote…
“Typically, when researchers analyzed data, they were free to make various decisions, based on their judgment, about what data to maintain: whether it was wise, for example, to include experimental subjects whose results were really unusual or whether to exclude them; to add subjects to the sample or exclude additional subjects because of some experimental glitch. More often than not, those decisions — always seemingly justified as a way of eliminating noise — conveniently strengthened the findings’ results. The field (hardly unique in this regard) had approved those kinds of tinkering for years, under-appreciating just how powerfully they skewed the results in favor of false positives”
Did you get that? In non-science speak, they cheated. They freaking threw away data points they didn’t like.
They call this little game P-Hacking (read the article if you want to better understand what’s meant by a P-value). And a Princeton-minted PHD in psychology by the name of Joseph Simmons co-authored a paper that calls bullcrap on the practice. In further writings Simmons has said…
“Everyone knew it was wrong, but they thought it was wrong the way it’s wrong to jaywalk… Simulations have revealed that it’s wrong the way it’s wrong to rob a bank.”
And one of the people to get called out for playing this game is psychologist Amy Cuddy. She’s somewhat famous for a TED talk where she shares her amazing power posing theory. The idea that if you assume a powerful posture before or during your big meeting - you’ll rule the room.
Or maybe not.
Turns out she did some serious P-hacking to get her data to support her idea. And now researchers, playing by the rules, can't seem to replicate her findings.
Gee, what a surprise.
KEEP YOUR SKEPTICAL EYES WIDE OPEN
People are waking up to this problem, but there’s no quick or easy fix. In fact, it’s almost as if the system is set up to incentivize this kind of garbage…
- Researchers, scientists, psychologists have to publish breakthrough-sounding stuff to keep their jobs.
- Journalists have to write about cool-sounding breakthroughs to keep their jobs.
- Consumers, with little time to look behind the claims, are hungry for easy answers.
So these questionable ideas get out in the wild and take on a life of their own. To the point that we get overwhelmed by this fake crap and have no idea what to really believe. Or, even worse, we get sucked in by the junk science and end up wasting our time and money on dead ends.
So the crap is going to keep flowing. We’re going to keep seeing amazing claims that don’t pan out. And our only defense is to stay skeptical and get educated.
So arm yourself. Read the excellent P-hacking article. And if you're feeling more adventurous you can give these two books a shot.
I learned a lot from these books and I hope you do too...