Claude plugin eval compares with and without your plugin

Proof: Use evals to measure how reliably your plugin steers Claude to the right outcome, to catch regressions when you change the plugin or a new model ships, and to see what the plugin contributes compared with no plugin at all.

Impact: Plugin authors need Claude Code 2.1.269 or later and a plugin manifest. Run claude plugin eval init at the plugin root, then claude plugin eval dot to test prompts against graders. By default each case runs three times with the plugin and three without. These and judge calls consume plan usage or API billing. A high score alone does not prove the plugin helped; compare the two arms.

Watch next: Inspect failed graders and usage-limit errors before interpreting a score change as a regression.