Claude Code adds `plugin eval` command to compare plugin test runs

• Claude Code now has claude plugin eval, which runs curated prompts with and without a plugin so its contribution can be scored, according to ClaudeDevs.
• claude plugin eval init takes real prompts plus definitions of good and bad output, then drafts test cases and checks before piloting the suite, as ClaudeDevs' setup post describes.
• The evaluator reports each case in the terminal and in HTML, with private artifacts available on supported accounts, per ClaudeDevs' run post.
• Evals consume model tokens and can vary between runs, while plugin hooks and MCP servers execute with the user's permissions, ClaudeDevs cautioned.

Anthropic's skills guide describes the underlying baseline method as fresh sessions with the skill enabled and disabled, so authoring context cannot hide gaps in its instructions. The release's sharpest constraint is in ClaudeDevs' caveat: testing an untrusted plugin also runs its hooks and MCP servers as the user.

plugin eval turns plugin validation into a paired experiment. Each case gets a run with the extension loaded and a baseline run without it, then compares the scores.

The plugin-eval guide names the two arms WITH and W/OUT; the reported difference is Δ = WITH - W/OUT. The setup command starts in a plugin folder, takes real prompts and definitions of good and bad output, then drafts cases and checks, pilots them, and estimates the full-run cost.

The guide lists graders for regex matches, tool use, tool order, and expected files. LLM and baseline graders call a judge model and add usage.

No aggregate plugin scores shipped with the command. The evaluator's documented comparison metrics are:

First-party

• Evaluation arms: old 1.0 -> new 2.0, +1.0 points, from the W/OUT baseline and WITH plugin arm in the plugin-eval guide.
• Run repetitions: old 1.0 -> new 3.0, +2.0 points, from a --runs 1 pilot to the guide's default three-run evaluation.
• Result surfaces: old 1.0 -> new 2.0, +1.0 points, from terminal scoring to terminal scoring plus an HTML report, as ClaudeDevs' run post describes.

The HTML result can be published as a private artifact on supported accounts. Official artifact documentation describes private, organization, and public-link access modes.

Every evaluation calls a model, uses tokens, and returns variable results. ClaudeDevs says a --runs 1 pilot is deliberately cheaper than a full evaluation.

The plugin-eval guide says the displayed cost is a list-price estimate, and that all run, grader, and judge calls count against the user's plan or API bill. It also specifies an isolated run with only the target plugin loaded, but that isolation does not remove the trust boundary: hooks and MCP servers still run as the user.

The stated use case is catching skills that drift after a model release. trq212 said that feedback directly motivated the command because it was hard to tell whether a skill still worked on a new model.

The plugin-eval guide requires Claude Code v2.1.269 or later, a valid plugin directory with plugin.json or a skills-directory plugin, and authentication aligned with the active Claude Code session. ClaudeDevs' release note says claude update makes the feature available.