AI Coding Research Is Obsolete by the Time You Read It
GPT-4 is the Model T of coding models, yet the studies people cite still test it. Check which model a study used before you trust it.
I was about to write a post called “Still read the diff.” I planned to argue that AI writes code faster than anyone can review it, reviewers have become the bottleneck, and AI reviewers miss bugs that humans catch. I had the sources lined up. Then I checked which models those sources tested.
They tested GPT-4, GPT-4o and Claude 3.5 Sonnet. The researchers published their studies this year, but they used models from 2023 and 2024.
GPT-4 is the Model T of coding models. It came out in March 2023, the year developers started using AI in large numbers. In Stack Overflow’s survey that spring, 44% of developers reported using AI tools, and 63% reported using them a year later. GitHub adopted GPT-4 for the next version of Copilot. In April 2025 OpenAI retired it from ChatGPT. Like the Model T, it got developers on the road, and now it belongs in a museum.
I’ve been working with Opus 5.5. It works toward a goal on its own for long stretches and almost never gets stuck. None of the sources I’d lined up for that post said anything about it.
What changed since GPT-4
METR researchers measure the length of tasks a model can finish at 50% reliability, using the time a human expert would need. I’ve listed the record-setting models from Epoch AI’s copy of the data:
| Released | Model | Task length it finishes half the time |
|---|---|---|
| Mar 2023 | GPT-4 | 5 minutes |
| Feb 2025 | Claude 3.7 Sonnet | 1 hour |
| Aug 2025 | GPT-5 | 3.4 hours |
| Feb 2026 | Claude Opus 4.6 | 12 hours |
| Apr 2026 | Claude Mythos (preview) | 17 hours |
GPT-4 could finish, half the time, a task an expert does in five minutes. The best model in April could finish one that takes an expert two working days. GPT-4 couldn’t do that kind of work in one run at all. You’d have had to cut it into small pieces, prompt each one and stitch the results together yourself.
Epoch’s copy of METR’s numbers stops in April, before Opus 5.5 or GPT-6 Astra came out. On FrontierSWE, a benchmark released in September, Opus 5.5 solves 62% of the tasks, and its runs average a little over 14 hours. I see those long runs with Opus 5.5 too: I can leave it working for hours without stepping in.
General Context Labs published LongHorizonSWE on September 28. An expert would need a day or more for each of its engineering problems. The researchers give the agent 4 to 12 hours, no visible tests and no human help. GPT-6 Astra passes 57% and Opus 5.5 passes 51%. GPT-5.6 Sol, released in July, passes 5%.
Epoch also folds dozens of benchmarks into one score, the Epoch Capabilities Index. The best model a year ago, GPT-5 Pro, scored 150. Opus 5.5 scores 167. The top score rose 5 points in the last three months, more than in any earlier quarter this year.
Studies take longer than model releases
METR’s own 2025 study shows the problem. Its researchers found that experienced developers using AI took 19% longer to finish their tasks. People keep quoting that result. The developers used Cursor with Claude 3.5 and 3.7 Sonnet. In February METR changed its experiment design and wrote that developers probably gain more speed from AI now than they did in early 2025. Its newer data is too noisy to say by how much.
Benchmarks wear out too, through contamination, flawed tests and saturation. In February OpenAI stopped reporting SWE-bench Verified: frontier models had seen its solutions in training, and many of its tests reject correct fixes. In September Google raised the bar on Android Bench with long-horizon tasks, after the top models passed about 91% of the original ones. The best model passes about 28% of the new tasks. Epoch can only track what labs and evaluators run: its newest SWE-bench Verified result is a model from June, and its newest Terminal-Bench result is from April.
What I measured
I built a research tool to check the claims behind my posts every week. It searches for evidence, reads each source and records whether it supports or contradicts the claim, plus which models it tested. Across the claims about AI coding itself, it has 121 distinct sources. My tool collected them for my own posts, so I don’t have a random sample of AI coding research. Measured against today’s best model:
- 72% get zero weight under my rule. Seventy never name the model they tested, and 17 tested a model too far behind.
- 84% of the hard data gets zero weight. That’s 36 of 43 studies and benchmarks.
- Of the studies that name their models, two thirds tested a model that’s now more than six months old, and 39% one more than a year old.
- Most sources never say which model they used. Only 51 of the 121 name one. “AI coding tools” in a survey could mean Copilot in 2023 or Opus 5.5 last week.
For three of the four claims behind “Still read the diff,” I have no source that counts against today’s models.
Weigh evidence by the model it tested
My first fix was to throw out anything older than six months, and I got that wrong. A paper published last month can still test GPT-4. Its publication date tells you when someone wrote it up, and you need the model to know what it’s evidence about.
My tool now weighs each source by how far its best model trails today’s best, using Epoch’s capability score. A source that tested today’s best model counts fully. Its weight halves for every 3 points it trails, and below 0.05 I round it to zero. GPT-5.5, from April, counts about 0.15. Opus 4.6, from February, counts 0.06, and anything that scores lower counts 0.
I picked that rule of thumb, and I checked that I’d reach the same conclusions about code review if I halved every 2, 4 or 6 points instead. My tool gives a source that names no model zero weight, because I can’t tell how far behind its model was.
Under those rules, the sources for “Still read the diff” can’t tell me much about current models. They still point the same way, but they don’t show how well today’s models review code. The benchmark gains give me a reason to test that again.
Before you cite a study
Ask three questions:
- Which model did it test? If the authors don’t say, you can’t learn much about today’s models.
- When did that model come out? Six months is a long time now, and a year is a different era.
- What’s the newest evidence on the same question? METR expects its own slowdown result has changed since early 2025.
This month Ars Technica reported on a study by Harvard researchers Fiona Chen and James Stratton, covering analytics data from more than 700 software firms. Firms that adopted coding agents wrote more code but found “little evidence” of more software output. Code review took longer, and the authors say downstream constraints “absorbed” the coding gains. That supports the review half of “Still read the diff.” The researchers’ data ends in March 2026, before Opus 5.5 shipped, so I can’t tell how well their findings hold for the newest models.
Next I’m testing Opus 5.5 and GPT-6 Astra on real diffs with planted bugs, to see which bugs they catch and which they miss. That test won’t tell me whether review slows down whole teams, but I’ll have current evidence for one part of “Still read the diff.”