#57a Fake News Supplement: Too Many Studies Can't Be Replicated


I left too much information out of my last post on questionable data, so here’s a little more background.

I already introduced you to billionaire John Arnold and his wife, Laura. They’re the couple investing huge dollars to replicate scientific studies. But I didn’t tell you how they came to that decision. It all started while they were searching for charities to support.

Here’s an excerpt from WIRED magazine February 2017 that explains their plight.

Begin excerpt:

Arnold explains that his and Laura’s initial plan had been to simply locate the most effective organizations and write them checks. But figuring out which organizations were most effective turned out to be vexing. Nonprofits are very good at reporting their success rates and citing the science behind their interventions, but dig into their claims - as the Arnold’s would try to do - and you find that they often omit relevant context or confuse correlation with causation. “The more you read the research, the less you know,” Arnold says. “It became extraordinarily frustrating.”

This is what led the Arnold’s to focus on the replication of existing studies. They wanted to know what really worked and what was sloppy garbage science.

This approach makes sense when you consider the work John Arnold did as a trader. Where he took in huge amounts of data. Found a pattern. Traded on it. Made or lost money.

Notice that there’s an unyielding scorekeeper in the trading game. You go broke if you don’t root out the difference between correlation and causation. That’s an incentive or discipline that’s missing in academic research studies.

Here’s a quote, via the WIRED article I referenced above, from James Owen Weatherall, author of The Physics of Wall Street.

“In the sciences, one is mostly incentivized to publish journal articles, and especially to publish the sorts of attention-grabbing and controversial articles that get widely cited and picked up by the popular media. The articles have to appear methodologically sound, but this is generally a lower standard than being completely convincing. In finance, meanwhile, at least when one is trading with one’s own money, there are strong incentives to work to that stronger standard. One is literally betting on one’s research.”

So, in the sciences, being super rigid about data and conclusions can work against you. And the news just keeps getting worse. Here’s another excerpt from the Wired article:

“In 2012 the former head of cancer research at the biotech firm Amgen revealed the results of the company’s effort to replicate 53 “landmark” papers in hematology and oncology; only six studies’ findings could be confirmed.”

Now, as I said last time, that doesn’t mean the studies are garbage. Maybe the replicators aren’t good at replicating. But I sure hope the alarm bells are going off. This kind of finding calls into question the legitimacy of all studies. Including solid ones that should be getting great press and funding. And that could be saving lives.

And even if you’re not dealing with cancer, this sloppy data problem still impacts you. Think about your favorite research on employee morale, or motivation, or rewards. What are the odds that these studies have cleaner data than the well-funded, highly-scrutinized medical and psychological studies that are struggling with replication?

I don't know the answer to that question. But I get more skeptical everyday. Especially when you layer on the incentives that numerically-challenged journalists have to print headline-grabbing claims.

Bottom line, there's no time to scour every research finding you like. But at least go deep on any you rely on to reinforce your business and personal beliefs.

***Note: This site works best when you read the posts in order. So please head to the ARCHIVE to get started.