Test Claude Skill Triggers With claude plugin eval (2026)
A. Frans
Published November 9, 2026
Table of Contents
- 011. Check the requirements and the bill
- 022. Wrap your skill in a plugin
- 033. Create the suite
- 044. Write a prompt a real person would type
- 055. Add the grader that detects the trigger
- 066. Write the cases that should not trigger
- 077. Know how the score treats the trigger check
- 088. Run it and read the table
- 099. Fix the description, then re-run
- 1010. Gate your CI on it
- 11When to use skill-creator instead
- 12Security notes
- 13Our take
- 14Frequently Asked Questions
A few small files and one command will tell you how often Claude picks your skill on natural phrasing. The tool is claude plugin eval, it ships inside Claude Code from v2.1.269, and it gives you a score you can gate CI on.
Testing a skill by hand has a built-in flaw. You wrote the description, so you unconsciously phrase requests to match it, and one successful run proves little because the model doesn't behave the same way every time. An eval runs each prompt three times by default in a clean, isolated session, then does it again with your skill absent.
We wrote this from Anthropic's plugin evals documentation, read on November 9, 2026. The sample output is Anthropic's, and we didn't benchmark a skill for this piece. If you're stuck at the earlier stage of a skill that never fires, start with our trigger troubleshooting ladder.
| Method | Measures | Clean session? | Baseline? | Best for |
|---|---|---|---|---|
Typing /skill-name | Whether the skill loads | No | No | First smoke test |
claude plugin validate | Syntax and schema only | n/a | No | Broken frontmatter |
skill-creator evals | Output quality, trigger hit rate | Yes | Yes | Iterating on one skill in a chat |
claude plugin eval | Trigger pass/fail, output score, baseline delta | Yes | Yes | Repeatable suites and CI |
1. Check the requirements and the bill
You need Claude Code v2.1.269 or later and Git 2.31 or later if Git is installed. Every eval run is a real model call on your plan or API bill, so do the arithmetic first.
A suite makes about cases × runs agent runs with the plugin, and the same again for the baseline. Ten cases at the default three runs comes to 60 agent runs. Judge graders add three short calls per grader per run. The four deterministic grader types (regex, tool_used, tool_order, file_exists) cost nothing, and a pure trigger test needs only tool_used.
2. Wrap your skill in a plugin
The command tests plugins, and a bare ~/.claude/skills/my-skill folder isn't one. Runs are isolated, so your personal skills, settings and CLAUDE.md are absent. Make a small wrapper:
mkdir -p commit-helper/.claude-plugin commit-helper/skills
cp -r ~/.claude/skills/commit-message commit-helper/skills/
echo '{"name": "commit-helper"}' > commit-helper/.claude-plugin/plugin.json
cd commit-helper
Anything the skill depends on has to ship inside this plugin too. Then check the syntax, which costs nothing and catches frontmatter that doesn't parse (v2.1.233 or later):
claude plugin validate .
3. Create the suite
The recommended route has Claude interview you about the plugin and draft the cases:
claude plugin eval init
To see every file yourself, generate a blank case instead:
claude plugin eval init --bare writes-commit-message
That creates evals/writes-commit-message/prompt.md and graders/criteria.md.
4. Write a prompt a real person would type
Open prompt.md. Leave the skill's name out, since the test is whether Claude finds it on its own. List Skill in allowed_tools, or Claude can't invoke anything:
---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---
Write me a commit message: I renamed getUser to fetchUser and updated the three call sites.
Each run starts in an empty working directory, and @path mentions aren't expanded. Put what the task needs in the prompt, or seed files with a case.yaml.
Write several phrasings, including the lazy ones. "commit msg pls" is what the description has to survive.
5. Add the grader that detects the trigger
Replace the placeholder with graders/skill-fired.md:
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit-message"'
---
It passes when Claude invoked that skill at least once, including under the namespaced commit-helper:commit-message form. The value is the skill's directory name.
6. Write the cases that should not trigger
A description broad enough to catch everything will hijack requests it has no business touching. Add a case such as ignores-code-review with a prompt about something adjacent, then assert zero calls:
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit-message"'
min: 0
max: 0
arm: both
---
arm: both matters here, for the reason in the next step. We'd start with about two positive prompts for every negative one, then add a case each time the skill misfires in real use.
7. Know how the score treats the trigger check
In a normal two-arm run, a tool_used grader on Skill is reported as a pass or fail indicator and left out of the score, in both arms. The reason is fair: a skill can't fire when it isn't loaded, so counting that check would inflate the difference.
So a case with a result grader plus skill-fired is scored on the result alone. Two exceptions apply:
- If every grader in a case is in the excluded set, they're scored normally.
- With
arm: bothor--ablation none, the grader counts.
For a pure trigger suite, use one tool_used grader per case, or run with --ablation none to skip the baseline and cut the run count in half.
8. Run it and read the table
From the plugin root:
claude plugin eval .
Answer yes to the trust prompt if this is your own code. Anthropic's example output looks like this:
CASE WITH W/OUT Δ RUNS COST NOTES
first-case 1.00 0.33 +0.67 6 $0.41
WITH is the score with your plugin, W/OUT is the score without it, and Δ is what the plugin contributed. A case at 1.00 in both columns means your skill isn't what made it pass, which is a useful thing to learn before you maintain it for a year.
The docs name the most common first finding: Δ near zero with tool_used: Skill failing. That means Claude isn't choosing your skill on natural phrasing.
A signed-in subscription account also publishes the HTML report as a private artifact. Add --no-publish to keep it on your machine.
9. Fix the description, then re-run
Change one thing at a time. Start by moving the key use case to the first sentence, then try adding the words people type. Claude Code truncates description plus when_to_use at 1,536 characters in the listing, and with many skills installed it can drop descriptions from the listing, so the first sentence carries the most weight.
For a quick loop on a single case:
claude plugin eval . --case writes-commit-message --runs 1 --ablation none
One run is noisy, so confirm any improvement at the default three before believing it. If the trigger grader passes but Δ turns negative, suspect the judge model before you suspect the skill. Re-run with --judge-model sonnet and tighten the rubric.
Also check the paths field if your skill sets one, since a skill doesn't activate for files outside Claude's working directory.
10. Gate your CI on it
claude plugin eval . \
--trust-plugin \
--json results.json \
--threshold 0.8 \
--model claude-sonnet-5 \
--judge-model claude-haiku-4-5 \
--no-publish \
--max-cost-usd 20
The command exits 1 when a case falls below the threshold. The default threshold is 1.0, which is too strict for most non-deterministic suites, so set your own. Pin both models so this month's score means the same as last month's, and add evals/results/ to .gitignore.
claude plugin eval init needs a terminal, so create cases locally or use --bare in CI.
When to use skill-creator instead
The official skill-creator plugin runs a similar with-and-without comparison inside a conversation, and it can generate should-trigger and should-not-trigger prompts and propose description edits:
/plugin install skill-creator@claude-plugins-official
Source is on GitHub, and our skill-creator review has more. Its evals/evals.json format and the claude plugin eval case files aren't interchangeable, so pick one per skill. We'd use skill-creator while drafting and move to claude plugin eval once the skill is shared with other people.
Security notes
An eval executes the plugin, so run third-party suites with care. Inside a Git repository, answering yes to the trust prompt trusts the whole repo, for interactive sessions as well. A case's scaffold_script runs as you, outside the agent's sandbox, and only when you pass --scaffold. Pass that flag only for suites you or your organization wrote.
Hooks and real MCP servers also run outside the sandbox and could alter the files graders read. Anthropic says to treat scores as advisory for plugins with hooks you didn't write unless you run them somewhere isolated, like a container. Read our skill audit guide before evaluating anything you downloaded. See also our roundup of LLM eval skills and the tools list for developers.
Our take
Run a trigger suite on any skill that other people will depend on. Ten cases, deterministic graders, no judge. It's cheap, it finds the description bugs manual testing hides, and the no-plugin column will occasionally tell you the skill isn't earning its place in your context. Skip it for a personal skill you invoke by typing its name, because you've already solved the trigger problem.
Frequently Asked Questions
What does claude plugin eval do?
It runs realistic prompts against a Claude Code plugin in isolated sessions, grades the results, and compares them with a run that has no plugin loaded. It requires v2.1.269 or later.
Can I test a skill that isn't in a plugin?
Not directly by path. Wrap it in a folder with a .claude-plugin/plugin.json manifest and a skills directory. Runs are isolated, so personal skills aren't available otherwise.
How much does a skill eval cost?
Each run is a real model call on your plan or API bill. Expect cases × runs × 2 agent runs. The regex, tool_used, tool_order and file_exists graders add no model cost.
Why is my tool_used Skill grader not counted in the score?
In a two-arm run, skill-invocation checks are shown as indicators and excluded from scoring. They count if every grader in the case is excluded, or with arm: both or --ablation none.
How do I test that a skill does not trigger?
Add a case with an adjacent prompt and a tool_used grader for the Skill tool with min: 0, max: 0 and arm: both. It passes only if Claude never invokes that skill.
Share this article
⚙Related Tools
📄Related Articles
Get More AI Tool Guides
New comparisons and guides every week. Join thousands of professionals staying ahead of the AI curve.