Episode 23: Statistical significance

It’s mentioned on the podcast pretty much every week. But what does “statistical significance” actually mean?

In this episode of The Studies Show, Tom and Stuart start 2024 off with the most exciting subject possible: p-values. THRILL as they discuss statistical misconceptions! MARVEL as they talk about how “effect size” differs from “statistical significance”! CHUCKLE as they resort to endless coin-flipping analogies! And GASP as they discuss ways to stop scientists from “hacking” their p-values and ending up with misleading research!

The Studies Show is brought to you by Works in Progress magazine - an online magazine full of essays about science, technology, and human progress. Works in Progress is the kind of magazine that makes its readers massively better-informed about every subject it covers, with deeply-researched articles by experts in the relevant fields - and it’s all free. Check it out at their site right here.

Show notes

* 89% of psychology textbooks get p-values wrong

* Letter on how the research on power-posing went wrong

* The classic “false-positive psychology” paper on how p-hacking can get you any result you want

* The FiveThirtyEight online p-hacking tool

* The “p-curve” method for detecting p-hacking

* How p-hacking is just “overfitting” by another name

* List of weasel terms like “approaching significance”

* Reading on a screen before bed “might be killing you”!

* The (much less scary) relevant study

* Tom’s BuzzFeed News article on the idea to lower the p-value threshold to 0.005

* The original paper, plus the response arguing that scientists should “justify their alpha”

* Registered Reports, and how they can deter p-hacking

Credits

The Studies Show is produced by Julian Mayers at Yada Yada Productions.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sciencefictionspod.substack.com/subscribe