Corpus, Gilbert, Koenecke & Naaman report this pattern in a Nature Human Behaviour paper on Change.org’s in-platform “write with AI” tool, which is, IMO, a useful reminder that, when evaluating AI’s impact at a company, we should always check whether better outputs translate into the results we actually need.
Change.org rolled the tool out by geography: the US, Great Britain and Canada got full access 11 weeks before Australia. The authors use difference-in-differences to compare changes in those countries with changes in Australia around the launch. The causal interpretation depends on parallel trends. They studied ~1.5M petitions overall, with the main causal analysis focused on English petitions in those countries. The treatment is AI access, not confirmed use by each writer.
What changed (outputs)
With AI access, petitions got longer, more lexically diverse, and harder to read (higher Flesch–Kincaid grade level). In the static DiD, median word count rose by 53.6 words (~49%), median grade level by 1.49 grades, and MATTR by 0.049. Petitions also became more similar to each other: mean pairwise embedding similarity rose by 0.056 (~23% vs a pre-AI mean of 0.240). Independent raters judged the writing better and more persuasive too. That comparison was descriptive, outside the causal DiD.
What didn’t (outcomes)
Early engagement didn’t improve. The share reaching 10 signatures within 30 days fell by 5.33 percentage points (statistically significant). The share receiving ≥1 comment within 30 days also fell, but the change wasn’t statistically significant in the aggregate DiD. These measures track whether petitions get off the ground. They don’t tell us whether petitions achieved their goals or changed policy.
Among returning writers, second petitions written with AI access got longer and were associated with worse outcomes. Second petitions generally do worse, partly reflecting regression to the mean; the AI-access result here is associational. There was no clear benefit for writers whose first petition had already failed. Participation volume showed no detectable increase either.
What I find also interesting is that lexical features that used to predict early engagement became weaker predictors after AI access, which reminds me of the labour-market signalling study I wrote about earlier: on Freelancer.com, writing quality improved after AI adoption but became less informative about candidates’ job performance.
Related notes
- What happens to our own performance when the AI is suddenly gone?
- Beyond prediction: Exploiting organizational events for causal inference in people analytics
- Making talk cheap: How GenAI may disrupt labor-market signalling
- ChatGPT as a new email writing coach?
- Can AI help us fight the bullshit asymmetry?
📄 Read the original post with full outputs on my blog.