Episode 31: The trouble with meta-analysis

We all love to cite meta-analyses. They’re the review studies where scientists take every single piece of research ever published on a particular question, and then calculate the overall “true” effect across all of them. Putting together all those studies is a much better way to get to the truth… isn’t it?

In this episode of The Studies Show, Tom and Stuart give a intro to meta-analysis, and then talk about several major problems with the whole idea. Is meta-analysis—relied upon for making so many important scientific decisions, and cited in so many of our previous episodes—in serious need of a rethink?

We’re proud to be sponsored by Works in Progress magazine. If you’re intrested in in-depth, data-rich articles on often-surprising topics relating to human progress, history of technology, and scientific discovery, there’s no better place than WiP. Their most recent February 2024 issue is replete with articles on organ markets, vaccine challenge trials, the underappreciated power of silk, and much more. Check it out at this link.

Show notes

* Slide show from the Cochrane Collaboration on the basics of meta-analysis

* Description of the GRADE guidelines for assessing study quality

* Below is a funnel plot, a method of testing for publication bias in meta-analysis. Source: we asked an AI to randomly generate some data and display it in a funnel plot, just for illustration. This funnel plot is relatively symmetrical and probably wouldn’t indicate much publication bias:

* Criticism of funnel plots; Nature news reporting on the criticism

* Stuart’s Substack article on the homeopathy meta-analysis (and the retraction note for that meta-analysis)

* The PET-PEESE technique for meta-analysis; and a criticism of it

* Useful paper that compares between different bias-correction methods for meta-analysis

* The p-curve website, which has the paper explaining the technique and a useful app where you can do your own p-curve

* Stuart’s Substack article on the meta-analysis on “nudges”

* Further criticism of the nudge meta-analysis, with important points about “meaningless means” (and yet more criticism)

Credits

The Studies Show is produced by Julian Mayers at Yada Yada Productions.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sciencefictionspod.substack.com/subscribe