Agent skills cause regressions in nearly 6,000 paired office-automation runs

• Agent skills now have a measured regression tax: nearly 6,000 paired office-automation runs found that skills fixed some tasks and broke others, with the paper summary naming skill-description osmosis, grounding displacement, and verification displacement.
• The sharpest community pattern is smaller, scoped skills: the small-skills reply frames specs and tickets as model-facing artifacts for the “smart zone,” while the 600-skill anecdote shows the opposite failure mode.
• Skills are getting easier to create anyway: the VS Code demo shows prompt-to-skill migration, and VS Code’s docs define skills as portable SKILL.md folders that Copilot loads when relevant.
• Reliability work is moving into compaction and eval gates: the intentional compaction quote describes fresh-session markdown artifacts, and the Blume 1.2 launch turns docs coverage into CI failures.

The Regression Tax paper is useful because it decomposes aggregate success into tasks gained and tasks broken. Anthropic’s context-engineering post says Claude Code removed more than 80% of its system prompt for advanced models with no measurable loss on coding evals. VS Code’s Agent Skills docs now describe skills as an open standard across VS Code, Copilot CLI, and Copilot cloud agent.

The controlled comparison is the keeper: the paper ran 5,832 task-condition runs across OfficeQA-Pro and SpreadsheetBench, using OpenCode with MiniMax-M2.7, Codex with GPT-5.4-mini, and Claude Code with Claude Sonnet 4.6.

Its paired decomposition found:

• 553 gain transitions, where a task failed without skills and passed with skills.
• 324 regression transitions, where a task passed without skills and failed with skills.
• Regressions offset 59% of gross gains.
• Every tested library had at least one regression.
• The best-performing libraries separated themselves mostly by breaking fewer tasks, not by producing more gains.

The three named mechanisms are crisp enough to reuse in reviews:

1. Skill-description osmosis: the skill changes behavior just by being present in context.
2. Grounding displacement: the prescribed procedure overrides how the agent reads its inputs.
3. Verification displacement: the procedure suppresses checks the agent would otherwise run.

The community argument is converging on small artifacts that shape the model’s context, not giant process manuals. The small-skills reply says specs define a destination that can later be checked by /code-review, tickets split work into “smart zone” chunks, and Wayfinder breaks planning into context-window-shaped pieces.

The same reply says a spec plus /goal performed worse than pre-scoped chunks for AFK agents. That is the strongest practical defense of tickets in the thread: they are not Jira cosplay, they are context-window packing.

Anthropic’s post says Claude Code removed more than 80% of its system prompt for models such as Claude Opus 5 and Claude Fable 5, while maintaining coding-eval performance. The replacement pattern is “give judgment, design interfaces, use progressive disclosure,” not always-on rules.

One reaction described the failure mode of long prompts as edge-case overfitting: the model follows specific requests while losing the chance to surface “unknown unknowns.” That lines up with the paper’s osmosis result, because passive context can steer behavior even when no skill body runs.

VS Code’s demo shows a new migration flow that turns existing prompts into skills inside the editor. The docs put the format behind that workflow in plain terms: a skill is a directory with a required SKILL.md, YAML frontmatter, instructions, and optional scripts or examples.

The VS Code docs list the portable locations engineers will recognize:

• Project skills: .github/skills/, .claude/skills/, .agents/skills/
• Personal skills: ~/.copilot/skills/, ~/.claude/skills/, ~/.agents/skills/
• Required frontmatter: name and description
• Invocation controls: user-invocable and disable-model-invocation
• Experimental isolation: context: fork, which runs the skill in a dedicated subagent context

A team debugging poor agent performance found that it was loading more than 600 skills at startup. VS Code’s docs describe progressive skill loading, but a huge visible loadout still creates selection and context hazards before the agent starts doing useful work.

The drskill repo frames the tool as “brew doctor” for an agent’s loadout. Its checks include shadowing skills, duplicate loads, near-duplicate skills, broken SKILL.md specs, broken symlinks, lockfile drift, token-heavy skills, duplicate MCP servers, committed secrets, unpinned server packages, missing commands, and tool-description collisions.

Dex Horthy, founder of HumanLayer and coiner of “context engineering,” defines intentional compaction as compressing useful context into a verified markdown artifact, then starting a fresh conversation. The quoted workflow has three passes:

1. Research code, then compact the useful state into a document.
2. Start a new session, read the ticket and intent, then produce a design document with current state, desired end state, and design questions.
3. Start another session with the research and design, then plan from the compressed intent and codebase state.

The follow-up puts the operating rule in one line: there is no perfect skill set or plan shape, only moving context between sessions and across the autocompact boundary.

Boris Cherny, Claude Code creator, gave the least mystical version of the workflow: assign a task that is too hard, give the model the verification tools a human would use, observe where it fails, then patch the failure with prompting, a skill, or an MCP connection.

The linked [orchestrate skill]Orchestrate skill makes that loop multi-agent: keep an orchestrator available to the user, run narrow read-only scouts in parallel, give each worker distinct ownership, prevent leaf workers from delegating further, and leave approvals with the user.

Blume 1.2 adds blume eval, a docs test suite where real user questions and required facts live in evals.yaml. A reader agent answers using only Blume’s private docs MCP, then a separate judge grades the answer.

Failures point back to the source page missing the answer, with JSON output for CI and editor tooling. The Blume repo also lists llms.txt, raw Markdown URLs, “Open in chat,” an optional Ask AI assistant, a hosted MCP server, and agent skills for maintaining the docs site.