Ever since ChatGPT came out, everyone says "it saves time". But how much? And what about quality? Shakked Noy and Whitney Zhang, two researchers at MIT, did the right thing: a randomized controlled experiment — the gold standard of empirical research. Published in Science in July 2023, it became the most-cited paper on the topic.

1. The experiment

444 US college-educated professionals — managers, marketers, analysts, consultants, HR. Each one gets assigned two writing tasks typical of their job: internal memos, press releases, short strategic plans, analyses. Nothing artificial: things they actually do at work.

Between the first and second task, half the participants (randomly selected) receive access to ChatGPT. The other half doesn't. All results are then evaluated by other professionals from the same sector, blinded — they don't know who used AI and who didn't.

2. The three results

The numbers have become famous, but it's worth seeing them together:

Three boxes side by side: '−40%' time per task, '+18%' result quality, 'flattens' skill gap. Below: '444 professionals, writing tasks, controlled RCT'.
// fig. 01 The three numbers from Noy & Zhang (Science, 2023). Statistically significant, but with the experiment's limits (short tasks, US population, 2023 ChatGPT).
  • Time per task: −40%. ChatGPT users finish significantly faster. Not 10%, but 40%.
  • Quality: +18%. And despite the speed, the result is judged better, on average, by external reviewers. This is the counter-intuitive part: usually fast means lower quality.
  • Inequality: flattens. This is the most interesting finding.

3. The surprise: the skill gap flattens

Before the experiment, writers were rated on a baseline task (no AI). So the authors could measure the distribution of quality: who was very good, who mediocre, who below average. When ChatGPT enters, something interesting happens: the curve flattens. The weaker writers improve a lot, the strong ones improve a bit.

Chart with two lines: 'without AI' a tilted gray line (weak writers write poorly, strong ones well); 'with AI' a nearly flat black line at the top (everyone writes well). Annotation: 'the weak gain a lot', 'the strong gain a little'.
// fig. 02 The compressor effect: ChatGPT raises the floor more than the ceiling. The gap between "mediocre writer" and "good writer" narrows considerably.

The economic reading is significant: if AI levels the skills in writable tasks, the relative value of "being good at writing" drops. The skill rent compresses. It's good news for those who started disadvantaged, and less good for those who lived off that specific competence.

4. What I take away

Three things:

  • The numbers are real, but with an asterisk. The experiment measures short tasks (15–30 min). For long, collaborative, ambiguous projects, the effect is likely different. Generalizing beyond writing tasks is a leap, not a direct inference.
  • AI changes the distribution, not just the average. It's more important to look at who gains than how much on average. Skill compression is one of the most relevant economic dynamics of the next 10 years.
  • The winner is whoever has good judgment, not whoever writes well. Consistent with what Agrawal/Gans/Goldfarb say in Power and Prediction: AI does the prediction (or the drafting, in our case), judgment stays human. And judgment doesn't level out as easily.

The paper is short (3 pages + appendix), open as a preprint on Noy's site, and it's the first solid empirical brick in an otherwise speculative debate. Worth reading in full at least once.