#99 Sketchy Claims: What The Heck Is P-Hacking

We’re all looking for ways to do work and life better.

And there's no shortage of experts willing to help us out. Experts that claim to have real research-based proof behind their ideas or systems.

But, unfortunately, most of them are full of crap. Or at least their data is full of crap.

Back in post #19 - Half Truths I warned you that many well-known business books were based on wobbly statistics. And in post #57 - Fake News In Your Presentation I shared that in the field of psychology experts have been unable to replicate 60% of the studies they’ve tested.

And now I'm circling back with an update because the crap is starting to hit the fan. Well-known, respected researchers are getting called to the carpet for playing games with their data.


Here's a great article on the topic from New York Times columnist Susan Dominus. The article includes this important quote…

“Typically, when researchers analyzed data, they were free to make various decisions, based on their judgment, about what data to maintain: whether it was wise, for example, to include experimental subjects whose results were really unusual or whether to exclude them; to add subjects to the sample or exclude additional subjects because of some experimental glitch. More often than not, those decisions — always seemingly justified as a way of eliminating noise — conveniently strengthened the findings’ results. The field (hardly unique in this regard) had approved those kinds of tinkering for years, under-appreciating just how powerfully they skewed the results in favor of false positives

Did you get that? In non-science speak, they cheated. They freaking threw away data points they didn’t like.

 Power posing is Powerful BS

Power posing is Powerful BS

They call this little game P-Hacking (read the article if you want to better understand what’s meant by a P-value). And a Princeton-minted PHD in psychology by the name of Joseph Simmons co-authored a paper that calls bullcrap on the practice. In further writings Simmons has said…

“Everyone knew it was wrong, but they thought it was wrong the way it’s wrong to jaywalk… Simulations have revealed that it’s wrong the way it’s wrong to rob a bank.”

And one of the people to get called out for playing this game is psychologist Amy Cuddy. She’s somewhat famous for a TED talk where she shares her amazing power posing theory. The idea that if you assume a powerful posture before or during your big meeting - you’ll rule the room.

Or maybe not.

Turns out she did some serious P-hacking to get her data to support her idea. And now researchers, playing by the rules, can't seem to replicate her findings.

Gee, what a surprise.


People are waking up to this problem, but there’s no quick or easy fix. In fact, it’s almost as if the system is set up to incentivize this kind of garbage…

  • Researchers, scientists, psychologists have to publish breakthrough-sounding stuff to keep their jobs.
  • Journalists have to write about cool-sounding breakthroughs to keep their jobs.
  • Consumers, with little time to look behind the claims, are hungry for easy answers.

So these questionable ideas get out in the wild and take on a life of their own. To the point that we get overwhelmed by this fake crap and have no idea what to really believe. Or, even worse, we get sucked in by the junk science and end up wasting our time and money on dead ends.

So the crap is going to keep flowing. We’re going to keep seeing amazing claims that don’t pan out. And our only defense is to stay skeptical and get educated.

So arm yourself. Read the excellent P-hacking article. And if you're feeling more adventurous you can give these two books a shot.

The Halo Effect…and the 8 Other Business Delusions That Deceive Managers
Naked Statistics: Stripping the Dread from the Data.

I learned a lot from these books and I hope you do too...

#84 Dying To Take Control: Freedom Extends Lives and Helps Your Business

A few weeks back I read a blog post from a billionaire venture capitalist. In it he shared his feeling that starting and running companies led to mega stress. And then added that it’s just as stressful being a venture capitalist who cares about the companies they invest in.

This last sentence set off my bullshit meter. So I checked in with a couple of my favorite psychologists to learn a little more about stress.


This is by no means a comprehensive discussion of sources of stress. It's just a couple lists that I recall from some past reading.

The first list comes from psychologist Jonathan Haidt and his book The Happiness Hypothesis.  The context is factors that can bring happiness, or in it’s absence, become stressors.  (location 1798)

Variable or intermittent noise
Commuting in traffic
Relationship conflicts
Shame (he focuses on body shame)
Lack of control

And here’s another list. This one from psychologist Daniel Kahneman and his book Thinking Fast and Slow (page 394). The context here is a discussion of factors that influence mood at work.

Opportunity to socialize with coworkers
Exposure to loud noise
Time pressure
The immediate presence of a boss


Let’s take a look at these lists as they relate to the plight of the stressed out venture capitalist.


Noise - I guess that would depend on how loud his private jet is. Actually, that’s not a fair comment because I have no idea if he has a private jet. But he definitely has the means to make sure his home(s) and office(s) are nice and quiet.

Commuting In Traffic - He can afford to live anywhere in the world he chooses. So if he’s commuting long distances in traffic that’s on him.

Opportunity to Socialize with Coworkers - He bounces from board meeting to board meeting so I’m assuming he can socialize as much or as little as he likes.

The Immediate Presence of a Boss - He would probably argue that the investors in his fund are his bosses, but that’s pretty weak. He could flip them all the bird anytime he wants and ride off into the sunset.

Time Pressure - Definitely has this as he’s trying to raise funds and choose investments. But again, he’s not living paycheck to paycheck, so no single transaction is more than a blip on his screen.

Shame & Relationship Conflicts - I have no visibility into either of these so I’ll make no assumptions.

Lack of Control - I find it hard to imagine a job and a financial situation that offers a person more control over their time and path. If he doesn’t have this, he’s not much of a personal manager.

So, in summary, if this guy is living a stressful life it’s because he chooses to do so. Or because it sounds cool to describe his life that way.


call centerArtboard 1@300x.png

A person that lives with real stress, it would seem to me from this list, would be someone like Ken. He's the poor SOB that's chained to a machine in a noisy manufacturing environment. Can’t even go to the bathroom until a replacement is available. Or maybe Sue. She's the nice lady crammed into the noisy call center. Monitored for productivity and efficiency every single second by that fancy new phone switch you installed. 

Ken and Sue punch a timecard. Their every motion has been studied and leaned out. They either produce X units of work a day or they’re out. And even worse, they’re probably going to die prematurely.

Per a recent study out of the Indiana University Kelley School of Business…

“those in high-stress jobs with little control over their workflow die younger or are less healthy than those who have more flexibility and discretion in their jobs and are able to set their own goals as part of their employment.”

More interestingly, the study also concluded that…

"These findings suggest that stressful jobs have clear negative consequences for employee health when paired with low freedom in decision-making, while stressful jobs can actually be beneficial to employee health if also paired with freedom in decision-making."

So stress isn’t necessarily a bad thing. A high stress job that offers lots of control is actually good for you. It’s the high stress jobs with low control, like those Ken and Sue endure, that’ll kill you.

And work isn’t the only place that a lack of control can be lethal.


Here’s more research from psychologist Johnathan Haidt’s book - The Happiness Hypothesis. Nursing home residents were provided plants and movie nights. One floor of residents got to pick their plants and care for them and also got to vote on their movies. Another floor had plants provided for them and taken care of for them. And they also got to watch a movie but they didn’t get to vote on which one. 

Per the nurses observations, the residents that were given control showed signs of being happier, more active, more alert. But you and I know how iffy conclusions can be when they’re based on folks opinions and observations of squishy stuff like happiness (see post #81 - The Survey Says - to refresh your memory). BUT, there’s more. After 18 months, the residents that were given control had demonstrably better health and half as many deaths. I repeat, half as many deaths. It’s not a squishy opinion when you’re counting body bags.


You own the place, so arrange your days to maximize your control. And what's good for the goose is good for the gander - make sure you're doing the same for your employees. And, given that I spent a brief part of my career as president of a lean consulting firm, I get that this is a challenge. It’s so tempting to lock everything down into a nice tight process - remove variability. But, process has it’s limits. People ain’t robots. And, per the research above, if you treat them that way you’re killing them. 

Now that’s a bit harsh, so maybe look at it in this more positive way. Giving folks some control, some freedom, some room to experiment and grow - is long term great for your business. This is where creativity is born.

Any company can jump on the treadmill and get more efficient. But a company that aims to be amazing in the long term needs a healthy mixture of efficiency AND creativity. And that creativity comes from giving folks a little room to breathe and explore.


(This site is all about building a Map that will help me do work and life better. So at the end of each post I check in to see if any changes / insights come to mind.)

Last time I did a long overdue overhaul of my Map, but I left out something important. While I kept my strong focus on Growth (refer to the IF and CAN sections), I accidentally dropped some of the detail behind the concept. I left off what we learned from Fredrick Herzberg in terms of how to find that Growth.  Growth requires Freedom - giving folks control like we talked about today. And Feedback - as we’ve talked about many times in the past. These points are so important that I want to explicitly call them out on my Map. Which is why you'll notice a new bullet point on my Work IF that reads "helping them prioritize freedom and feedback". 

I wanted to make sure and get this back on the page as it truly is a matter of life and death. See the Map below for details.


That's all for today.

***Note: This site works best when you read the posts in order. So please head to the ARCHIVE to get started.