Addy Osmani outlines hooks to stop coding agents from weakening tests

• Completion criteria get human approval before coding: Osmani reviews and edits Claude's draft in his workflow.
• Test protection has two layers in his proposal: a hook blocks test edits during implementation, and a separate review checks for weakened assertions.
• Recurring agent mistakes become shared configuration, hooks or tests, according to his post on verification.

Addy Osmani, who works on Claude Code at Anthropic, outlined a particularly sharp control in a reply: locking the tests while the agent implements the change. Claude Code's protected-file example rejects edits with exit code 2, while a 718-firm study found that average pull-request review time increased 49% after coding-agent adoption.

Osmani reviews Claude's proposed definition of done before any code is written, he explains in another reply. The human approves the target the implementation will be judged against.

He names three kinds of checks the agent can run before declaring completion:

• Bash commands.
• Tests.
• Browser click-throughs of the app.

His stopping condition is nothing blocking merge and ship. Without defined completion criteria, he says, a reviewer asked to find bugs may keep finding things that could be improved.

Osmani describes mechanically blocking edits to test files during implementation so the agent cannot quietly move the goalposts.

Claude Code's official file-protection example supplies the underlying mechanics:

1. Registration: a hooks block in .claude/settings.json points to a shell script.
2. Timing: PreToolUse runs before the requested tool executes, with an Edit|Write matcher.
3. Inspection: the script reads .tool_input.file_path from the tool-call JSON and compares it with protected patterns.
4. Rejection: the script writes an explanation to stderr and exits with code 2; Claude receives feedback explaining the blocked edit.

The published example protects .env, package-lock.json and .git/. Osmani's proposal specifies test files and an implementation-phase boundary.

Shell writes through Bash fall outside an Edit|Write matcher. The same guide explicitly documents files being rewritten through Bash, so the matched editing tools define the scope of this particular hook.

The test diff gets examined on its own, with the reviewer asking, “did any assertion get weaker?”, according to the same reply.

Osmani also allows that the test itself may be wrong. An agent changing it has to disclose that fact so a human can decide whether the correction is legitimate.

In a separate exchange, he expressed hope that high-bar verification could become largely a polishing and checking pass after an Opus-generated first pass.

Osmani argues that agents make tests, benchmarks, specifications and real end-to-end runs cheaper to produce. He describes humans retaining ownership of what ships while reviewing evidence rather than every line in his verification post.

Boris Cherny, head of Claude Code at Anthropic, gives a concrete CI optimization example with three inputs:

1. Goal: p95 CI time below 10 minutes.
2. Measurement: read timings from BigQuery after each change.
3. Effort budget: keep experimenting until the goal is reached, potentially over weeks.

He also says he sometimes requests daily progress charts or a live result artifact delivered by DM.

Recurring errors become persistent project controls in Osmani's account: shared configuration, a hook or a test records the correction in his proposal.

He attributes the compounding advantage to years of on-the-job agent use and says senior engineers contribute judgment about system design and failure modes. He also argues in the same conversation that restricting experimentation to after-hours signals that using AI does not count as work.

Commenter apsdehal proposes separating the people who teach or update an enterprise agent from those who verify its work and sign off. The proposed layer would work across models and learn from demonstrations and corrections.

Its decision trail would record:

• What changed and why.
• Who taught the agent.
• Who verified its work.
• How the agent made decisions and the basis for them.

Harvard researchers Fiona Chen and James Stratton's August 4 working paper analyzes 300 million work events across 718 firms, using Jellyfish data from January 2021 through March 2026. It compares adoption timing across firms using a staggered difference-in-differences design.

For coding-agent adoption, the authors report:

• Coding activity: 30% more lines of code, 20% more commits and 23% more pull requests, all statistically significant.
• Review load: average pull-request review time increased 49%; comments per pull request increased 35%; the share with changes requested nearly doubled.
• Completed work: small positive but statistically insignificant changes in resolved Jira issues and epics. The issue-resolution estimate rules out gains larger than 12% of the baseline mean, while leaving room for smaller gains.

A commenter questioned whether the 30% increase in lines of code implied limited actual agent use. The authors identify firm-level adoption as their treatment, which aggregates differences in uptake among individual workers.