Study finds 307 agent-skill failures, including 125 functional failures

• Agent skills caused 307 confirmed failures in a contrastive study, 125 functional failures and 182 high-confidence efficiency regressions, according to rohanpaul_ai's summary of the study.
• Seemingly relevant guidance produced 86 of the 125 functional failures by making agents implement required elements incorrectly or omit them, as rohanpaul_ai's breakdown reports.
• Explicit slash-command invocation removes ambiguous auto-triggering and keeps most skills out of context until called, in mattpocockuk's explanation.
• Forced multi-agent lanes can be expensive: daniel_mac8's 15-task comparison reported lower pass@1, 7.05 times the token volume, and 3.11 times the runtime for an earlier sol-advisor design versus Sol alone.

The paper keeps the task, verifier, agent framework, model, repository or container state, and input data fixed while varying the skill setup. mattpocockuk's post captures a separate design choice: make a human select the procedural prompt instead of asking the agent to decide whether it should fire.

The authors compared a skill-guided target run with either a no-skill run or a semantically matched alternative skill. A functional failure required the target to fail a verifier while its reference passed; an efficiency regression required both to pass while the target used substantially more tokens, time, or both.

That produced 307 confirmed cases across SkillsBench and SWE-Skills-Bench, rather than a count of all runs that happened to fail with a skill loaded. The paper uses a 2.0 threshold on at least one cost metric for its primary high-confidence efficiency-regression analysis.

The largest functional category was Task-Implementation Fault: 86 of 125 failures, or 68.8%. The paper describes agents being pushed to implement or omit a required field, API behavior, calculation, output format, or domain rule incorrectly.

Wrong artifact locations accounted for 24 cases and environment mismatches for 13. The result cuts against a simpler explanation in which bad outcomes mostly come from obviously irrelevant prompts.

Excessive Procedure accounted for 114 of 182 efficiency regressions, or 62.6%, in the paper's taxonomy. Excessive verification contributed 67 cases and heavy implementation pipelines another 30; 43 of the 46 context-overhead cases came from mandatory skill-body text.

The sol-advisor report is a small, separate 15-task TerminalBench 2.1 comparison, but its stated failure mode matches the study's mechanism: its orchestration contract forced subagent lanes even when they were unnecessary. The author said a revised version selects among solo, delegation, audit, and full routes by task risk, while full-benchmark results were still pending in daniel_mac8's follow-up.

mattpocockuk argues for user-invoked skills because the caller never has to ask whether /to-spec fired, why it fired, or why it chose /to-tickets instead. His other claim is cost-related: most skills contribute no tokens until the user invokes them.

That pattern still permits substantial repository-specific guidance. mattpocockuk's code-review workflow says its review skill reads a root CODING_STANDARDS.md, which he says can run to hundreds of lines.

The study did not compare user-invoked selection against automatic triggering. Its evidence concerns what happens after a skill is loaded, so explicit invocation addresses selection and dormant-context cost, not whether the selected procedure fits the particular task.

The authors also built SkillTriage, which normalizes paired executions, extracts differential evidence, and produces root-cause reports. Against manual labels, it matched exact functional-failure causes in 111 of 125 cases, or 88.8%, and exact efficiency-regression causes in 132 of 182, or 72.5%.

Those results depend on the paper's paired reference-run design: the tool has evidence that the same verifier-facing task could succeed, or succeed more cheaply, under a different skill setup.