Every factual sentence carries assertions checked against the recorded evidence: references must resolve, figures and quotations must be in the cited passage, arithmetic must reproduce, and an independent reviewer must approve the wording. Automated validation says whether the narration is ready for human editorial review; it does not establish truth.
Open a sentence to see its assertions, their status and the passages behind them. Sentences without a marker make no factual claim.
verified s02.a The Echoes of AI study ran a randomised controlled trial on manually evolving code that was or was not developed with AI assistance.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")] Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code th…
verified s03.a A vibe-coding literature review quotes a study noting that code generated through conversational flows often lacks cohesive structure, consistent patterns and documentation, complicating future maintenance.
qualified (qualify): no direct support, 2 qualifying, 1 context only.
warning needs-qualification: Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code. Add the qualification: the evidence supports a narrower or weaker statement.
verified s04.a Razorwire Cybersecurity said that 46% of AI-generated code reportedly contains some form of security weakness.
supported (approved): no direct support, 3 qualifying, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s06.a The trial found no reliable evidence that AI-assisted code differs in manual evolution efficiency; completion times were consistent with a null effect.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")] Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code th…
verified s07.a The authors report at most small and highly uncertain differences between treatment and control.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")] Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code th…
verified s08.a The trial found no reliable frequentist evidence that prior AI-assisted development affects CodeHealth after continued manual evolution.
qualified (qualify): no direct support, 3 qualifying, 1 context only.
warning needs-qualification: Echoes of AI maintainability RCT: Same RCT on quality: no frequentist CodeHealth difference after a second developer evolved the code, a small Bayesian CodeHealth gain when a habitual AI user wrote the original, and identical test coverage. Scope is limited to a small app and short-term effects. Add the qualification: the evidence supports a narrower or weaker statement.
verified s08.b Test coverage was essentially identical between the groups.
qualified (qualify): no direct support, 3 qualifying, 1 context only.
warning needs-qualification: Echoes of AI maintainability RCT: Same RCT on quality: no frequentist CodeHealth difference after a second developer evolved the code, a small Bayesian CodeHealth gain when a habitual AI user wrote the original, and identical test coverage. Scope is limited to a small app and short-term effects. Add the qualification: the evidence supports a narrower or weaker statement.
verified s09.a Bayesian analysis suggests a small CodeHealth improvement when the original solution was co-developed with AI by a habitual AI user.
qualified (qualify): no direct support, 3 qualifying, 1 context only.
warning needs-qualification: Echoes of AI maintainability RCT: Same RCT on quality: no frequentist CodeHealth difference after a second developer evolved the code, a small Bayesian CodeHealth gain when a habitual AI user wrote the original, and identical test coverage. Scope is limited to a small app and short-term effects. Add the qualification: the evidence supports a narrower or weaker statement.
verified s10.a Dave Farley said of the experiment that the AI didn't make it any better and didn't make it any worse.
supported (approved): no direct support, 2 qualifying, 1 context only, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s12.a Shan et al. measured coding maintainability and technical debt with multi-item scales answered by respondents, and found AI use associated with decreased maintainability and increased technical debt.
qualified (qualify): no direct support, 2 qualifying, 1 context only.
warning needs-qualification: Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. Add the qualification: the evidence supports a narrower or weaker statement.
verified s14.a In the vibe-coding literature review, maintainability implications appear in 23% (11) of the sources.
qualified (qualify): no direct support, 2 qualifying, 1 context only.
warning needs-qualification: Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code. Add the qualification: the evidence supports a narrower or weaker statement.
verified s16.a A study of AI co-authored commits found that some AI-introduced issues still survive at HEAD.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: Debt study (302k AI commits): AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells. Add the qualification: the evidence supports a narrower or weaker statement.
verified s17.a AI co-authored commits fix a similar number of code smell issues as they introduce; code smells here are surface-level problems such as formatting and naming.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: Debt study (302k AI commits): AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells. Add the qualification: the evidence supports a narrower or weaker statement.
verified s17.b AI commits introduce more correctness and security issues than they fix.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: Debt study (302k AI commits): AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells. Add the qualification: the evidence supports a narrower or weaker statement.
verified s18.a The commit study does not compare AI-authored commits against a baseline of purely human-written commits.
qualified (qualify): no direct support, 2 qualifying, 1 context only.
warning needs-qualification: Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")] Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code th…
verified s30.a Out of 733 generated code snippets, 27.3% contained security weaknesses.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: 2310.02059.txt: 733 Copilot (and CodeWhisperer, Codeium) snippets found in public GitHub projects: static analysis plus manual filtering flagged CWE weaknesses in 27.3% (about 30% of Python, 25% of JavaScript). Frequent, but these are tool-flagged weaknesses, not confirmed exploits, and there is no human-written baseline. Add the qualification: the evidence supports a narrower or weaker statement.
verified s19.a In the vibe-coding literature review, the most frequently discussed open challenge is security risk in AI-generated code, appearing in 50% (10) of the sources in its table.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: Vibe-coding literature review: Security risk is the most-cited open challenge (10 of 20 RQ8 sources); peer-reviewed and practitioner sources agree generated code may contain weaknesses when it is not rigorously verified, but no prevalence rate is synthesized. Add the qualification: the evidence supports a narrower or weaker statement.
verified s20.a Peer-reviewed and practitioner sources agree that generated code may contain weaknesses when developers rely on automated generation without rigorous verification.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: Vibe-coding literature review: Security risk is the most-cited open challenge (10 of 20 RQ8 sources); peer-reviewed and practitioner sources agree generated code may contain weaknesses when it is not rigorously verified, but no prevalence rate is synthesized. Add the qualification: the evidence supports a narrower or weaker statement.
verified s32.a Participants, recruited as crowd workers and mostly freelance or professional software developers, completed a coding task with or without AI and took a quiz measuring the skill formation gained by completing it.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: 2601.20245v2.txt: Preregistered RCT, 52 crowd-recruited developers (mostly freelance or professional) learning a new Python library: those allowed AI scored 17% lower on a 27-point quiz on its concepts, code reading and debugging (d=0.74), with no significant time saving. Supports the reason behind the norm; no merge gate was tested. Add the qualification: the evidence supports a narrower or weake…
verified s33.a Using AI to complete the coding task did not significantly improve task completion time.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: 2601.20245v2.txt: Preregistered RCT, 52 crowd-recruited developers (mostly freelance or professional) learning a new Python library: those allowed AI scored 17% lower on a 27-point quiz on its concepts, code reading and debugging (d=0.74), with no significant time saving. Supports the reason behind the norm; no merge gate was tested. Add the qualification: the evidence supports a narrower or weake…
verified s33.b Skill formation measured by the quiz was significantly reduced with AI; the 4.15-point gap on the 27-point quiz is a 17% score difference.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: 2601.20245v2.txt: Preregistered RCT, 52 crowd-recruited developers (mostly freelance or professional) learning a new Python library: those allowed AI scored 17% lower on a 27-point quiz on its concepts, code reading and debugging (d=0.74), with no significant time saving. Supports the reason behind the norm; no merge gate was tested. Add the qualification: the evidence supports a narrower or weake…
verified s22.a In the code-ownership experiment, AI assistance was associated with a 5.95 percentage-point decrease in the probability of answering ownership questions correctly.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Code-ownership experiment (Martin-Lopez): Crossover experiment, 69 participants (mostly students): after AI-assisted tasks people answered fewer questions about their own just-written code correctly (median 87.5% vs 100%; AME -5.95 pp, OR 0.58). The authors argue this can stop contributors defending their choices in code review. That supports the reason behind the norm, but no merge gate was teste…
verified s23.a The negative impact of AI assistance on code ownership was robust to alternative specifications, with fluctuations in significance but not in effect direction.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Code-ownership experiment (Martin-Lopez): Crossover experiment, 69 participants (mostly students): after AI-assisted tasks people answered fewer questions about their own just-written code correctly (median 87.5% vs 100%; AME -5.95 pp, OR 0.58). The authors argue this can stop contributors defending their choices in code review. That supports the reason behind the norm, but no merge gate was teste…
verified s24.a The Echoes RCT found no reliable difference in manual evolution efficiency, with at most small and highly uncertain differences.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")] Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code th…
verified s24.b The Echoes RCT found no reliable frequentist evidence of a CodeHealth difference after manual evolution.
qualified (qualify): no direct support, 3 qualifying, 1 context only.
warning needs-qualification: Echoes of AI maintainability RCT: Same RCT on quality: no frequentist CodeHealth difference after a second developer evolved the code, a small Bayesian CodeHealth gain when a habitual AI user wrote the original, and identical test coverage. Scope is limited to a small app and short-term effects. Add the qualification: the evidence supports a narrower or weaker statement.
verified s26.b In the quiz trial, skill formation was significantly reduced when AI was used to complete the task.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: 2601.20245v2.txt: Preregistered RCT, 52 crowd-recruited developers (mostly freelance or professional) learning a new Python library: those allowed AI scored 17% lower on a 27-point quiz on its concepts, code reading and debugging (d=0.74), with no significant time saving. Supports the reason behind the norm; no merge gate was tested. Add the qualification: the evidence supports a narrower or weake…
verified s26.a AI assistance was associated with lower correctness on ownership questions about the participants' code.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Code-ownership experiment (Martin-Lopez): Crossover experiment, 69 participants (mostly students): after AI-assisted tasks people answered fewer questions about their own just-written code correctly (median 87.5% vs 100%; AME -5.95 pp, OR 0.58). The authors argue this can stop contributors defending their choices in code review. That supports the reason behind the norm, but no merge gate was teste…
verified s27.a Simon Willison said he does not think you should ever commit a line of code a language model wrote if you don't understand it yourself.
supported (approved): no direct support, 2 qualifying, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
28 scenes, 5:31 at 150 words per minute including pauses. Narration is copied from the script; every chart value and on-screen citation is checked against a passage.
1. sc01 · 0:00 to 0:08 typography
If an AI writes the code today, who pays for it when somebody has to change it next year?
Visual. Dark field; the question types on in large type, 'next year' held a beat in accent colour.
On screen. Who pays for it when somebody has to change it?
2. sc02 · 0:08 to 0:16 typography
One research team stopped guessing and ran a randomised trial: code written with or without AI, then changed by hand.
Visual. Title card: the trial's design in one line, 'changed by hand' landing last.
On screen. Written with or without AI, then changed by hand
3. sc03 · 0:16 to 0:29 document-excerpt
The worry has a familiar shape: one study collected in a review of the research warns that code built up through conversation often "lacks cohesive structure", which makes later maintenance harder.
Visual. The review's paragraph on maintainability; 'lacks cohesive structure' highlights as it is spoken.
4. sc04 · 0:29 to 0:39 typography
And on security, the Razorwire Cybersecurity channel put it bluntly: "46% of AI generated code is reportedly uh contain some form of security weakness".
Visual. Quote card, quoted words only; '46%' punches in.
On screen. "46% of AI generated code is reportedly uh contain some form of security weakness"
Cite. Razorwire Cybersecurity
5. sc05 · 0:39 to 0:46 typography
So what happened in that trial when people had to change code an AI had helped write?
Visual. The question types on.
On screen. So what happened in that trial?
6. sc06 · 0:46 to 0:52 document-excerpt
On speed, nothing reliable: their completion times were consistent with no effect at all.
Visual. The trial's RQ1 answer; 'consistent with a null effect' highlights.
7. sc07 · 0:52 to 1:03 typography
The authors are careful about what that means: any difference was at most small, and highly uncertain, which is not the same as proof there is none.
Visual. Title card; 'highly uncertain' held in accent colour.
On screen. At most small, and highly uncertain
Cite. Echoes of AI trial, RQ1
8. sc08 · 1:03 to 1:11 document-excerpt
On quality, the code's health score after the changes showed no reliable difference either, and test coverage was essentially identical.
Visual. The trial's RQ2 answer; 'no reliable frequentist evidence' highlights.
9. sc09 · 1:11 to 1:22 document-excerpt
A second, Bayesian analysis even hinted at a small gain in code health when the original was written with AI by someone who uses it habitually.
Visual. Same passage, now 'small CodeHealth improvement' highlights.
10. sc10 · 1:22 to 1:32 typography
Dave Farley, talking about the study, summed it up: "The AI didn't make it any better and it didn't make it any worse."
Visual. Quote card, quoted words only; 'better' and 'worse' balance either side.
On screen. "The AI didn't make it any better and it didn't make it any worse."
Cite. Dave Farley, on the Echoes of AI study
11. sc11 · 1:32 to 1:35 typography
So where does the debt fear get its evidence?
Visual. The question types on.
On screen. So where does the debt fear get its evidence?
12. sc12 · 1:35 to 1:53 typography
Partly from surveys: one study measured maintainability and technical debt through its respondents' answers to questionnaire scales, and AI use went with lower maintainability and more debt. That is an association in people's answers, not a timed test of someone else changing the code.
Visual. Two lines build one per sentence: what was measured, then what it is.
On screen. Questionnaire answers / An association, not a timed test
13. sc13 · 1:53 to 2:08 data-chart
The review of the research counted the worry too: maintainability concerns came up in 11 of its sources, 23 percent. That counts how often the concern is raised, which is not the same as measuring it.
Visual. Two tiles: sources raising maintainability, and their share.
| Sources | 11 | 11 | count | Vibe-coding literature review, lines 1434-L1452 |
| Share of the review's sources | 23% | 23 | percent | Vibe-coding literature review, lines 1434-L1452 |
14. sc14 · 2:08 to 2:21 typography
Then there is code out in the wild: a study that tracked commits made with AI coding assistants found that some of the issues they introduced still survive in the latest code.
Visual. Title card; 'still survive' held in accent colour.
On screen. Some issues still survive in the latest code
15. sc15 · 2:21 to 2:32 document-excerpt
The assistants fixed about as many code smells, surface problems like naming and formatting, as they introduced; but they introduced more correctness and security issues than they fixed.
Visual. The study's discussion paragraph; 'introduce more correctness and security issues than they fix' highlights.
16. sc16 · 2:32 to 2:44 document-excerpt
But the authors did not compare those commits with code written by people alone, so the study cannot say AI code leaves more debt than human code does.
Visual. The study's limitations paragraph; the no-baseline sentence highlights.
17. sc17 · 2:44 to 3:06 data-chart
And the security worry? One study did put a number on it: of 733 code snippets generated by AI, 27.3 percent contained security weaknesses. That is a count of weaknesses in AI code alone; it cannot tell you whether code written by hand would score any better on the same check.
Visual. Two tiles: snippets checked, share with weaknesses.
| Snippets | 733 | 733 | count | 2310.02059.txt, lines 930-L936 |
| Contained security weaknesses | 27.3% | 27.3 | percent | 2310.02059.txt, lines 930-L936 |
18. sc18 · 3:06 to 3:17 data-chart
The review of the research found security risk was the challenge its sources raised most, in 10 of them, half of those it tallied for that question.
Visual. Two tiles: sources raising security risk, and their share.
| Sources | 10 | 10 | count | Vibe-coding literature review, lines 1793-L1803 |
| Share of those tallied | 50% | 50 | percent | Vibe-coding literature review, lines 1793-L1803 |
19. sc19 · 3:17 to 3:27 document-excerpt
What those sources agree on is narrower than a rate: generated code may contain weaknesses when developers rely on it without rigorous checking.
Visual. The review's paragraph; 'generated code may contain weaknesses' highlights.
20. sc20 · 3:27 to 3:35 typography
There is one cost, though, that experiments did measure, and it lands on the person who wrote the code.
Visual. Title card; 'the person who wrote the code' lands last.
On screen. One cost experiments did measure
21. sc21 · 3:35 to 3:49 typography
In one trial, crowd workers recruited online, most of them freelance or professional software developers, did a coding task with or without AI, then took a quiz on the skills the task should have built.
Visual. Title card naming the design.
On screen. A coding task with or without AI, then a quiz
22. sc22 · 3:49 to 3:58 data-chart
Using AI did not make them significantly faster, and their scores came out lower by a gap worth 17 percent of the quiz.
Visual. One big tile: the score gap.
| Lower score, share of the quiz | 17% | 17 | percent | 2601.20245v2.txt, lines 468-L484 |
23. sc23 · 3:58 to 4:20 data-chart
In a separate experiment with different participants, working with AI cut their chance of answering questions about their own code correctly by 5.95 percentage points. The authors re-checked it other ways: each time the drop kept its direction, though in one it was too weak to be sure of.
Visual. One big tile: the drop in the chance of answering correctly.
| Drop, percentage points | 5.95 | 5.95 | percent | Code-ownership experiment (Martin-Lopez), lines 700-L721 |
24. sc24 · 4:20 to 4:37 typography
So, is AI-written code harder to maintain? In the one trial that tested it, over the time it ran, no: no reliable difference in speed or code health, though one trial with that much uncertainty cannot rule out a small one.
Visual. Title card; 'no reliable difference' held in accent colour.
On screen. In the one trial that tested it: no reliable difference
Cite. Echoes of AI trial
25. sc25 · 4:37 to 4:44 typography
The louder warnings rest on questionnaires, counts of concern and issue tallies with nothing human to compare them to.
Visual. Title card listing the kinds of evidence behind the warnings.
On screen. Questionnaires, counts of concern, issue tallies
26. sc26 · 4:44 to 5:04 typography
The cost that was measured is different: in both of those experiments, people working with AI learned less or understood their own code less well, and whether that makes it harder for those same people to change their code later is something none of these studies tested.
Visual. Title card; 'understood their own code less well' held in accent colour.
On screen. Learned less, understood their own code less well
27. sc27 · 5:04 to 5:15 typography
Simon Willison draws his line here: "I don't think you should ever commit a line of code that a language model wrote If you don't understand it yourself".
Visual. Quote card, quoted words only; 'understand it yourself' lands last.
On screen. "I don't think you should ever commit a line of code that a language model wrote If you don't understand it yourself"
Cite. Simon Willison
28. sc28 · 5:15 to 5:31 typography
So when you review AI-assisted code, yours or a colleague's, don't stop at whether it works: ask the author to walk you through it with the AI closed. If they can't, the code isn't finished yet.
Visual. Two lines build one per sentence.
On screen. Walk me through it with the AI closed / If they can't, the code isn't finished yet