A recent second-order meta-analysis synthesizing 14 meta-analyses, 1,600+ primary studies, and ~30 million participants finds that nudges show a small positive average effect (see the corresponding forest plot below).
However, after correcting for publication bias (see the corresponding funnel plot below), the pooled effect is statistically indistinguishable from zero.
A common critique is that “nudging” is too broad - like meta-analyzing all medicines at once - and that we should focus on specific interventions that work.
This study challenges that optimism. The authors found no robust moderators to identify successful contexts. This is likely because the foundation itself is shaky: the majority of existing meta-analyses were rated as low or critically low in methodological quality. The signal is simply lost in the noise of low-quality methods.
The implication is stark: We cannot currently distinguish the “winners” from the statistical artifacts.
For policy and practice, we must stop assuming “what works” and demand fundamental rigor. We need valid measurement before we can even begin to identify effective interventions.
Related notes
- What actually makes us happier?
- Effectiveness of interventions for encreasing employee engagement
- Evidence-Based Management: Boring basics and better questions
- How variations in analytic choices affect results?
- Does agile pay off?
📄 Read the original post with full outputs on my blog.