Compare with test variants
Compare models, agents, skill revisions, and context using test variants, with examples of how to interpret the results.
Use test variants to try different configurations on the same tasks. You can compare models, choose between skill rewrites, or check whether an agent uses a skill without being prompted.
A test variant specifies the model, agent, and files available to the agent. Each variant solves the same scenarios and is scored against the same rubric. A run can include as many variants as fit in its solve limit.
If you are new to evaluations, start with Prove a skill works using evaluation. The examples below build on that workflow.
Set up a comparison
Declare your variants with --variant-json, as a JSON array or the path to a JSON file:
$ tessl eval run ./my-skill --variant-json ./variants.jsonThe examples below show the JSON array for each comparison. Replace placeholder models, agents, and paths with your own values. The run solves only the variants you list, so add a baseline yourself when you need one. The full field reference is in Comparing test variants.
A fixture is a set of files supplied to the agent. The plugin you evaluate is the fixture named context. The fixtures field changes fixtures by name for one variant:
A value replaces the fixture, or adds it when the name is new. Set
contextto another directory to test a different version of the plugin.nullremoves the fixture. Setcontexttonullto measure the agent without the plugin. Other fixtures remain available.A name you leave out keeps the fixture the scenario and run supply.
Refer to a local directory with { "localPath": "./path" }. Paths are relative to the directory where you run the command. The CLI uploads each distinct path once.
Make the results useful
Run each scenario more than once. Agents can produce different results from the same input, so a single attempt tells you little about consistency. Start with three repeats (-n 3) and increase the count when results vary widely or the decision needs more confidence.
When you want to measure how much a skill helps, include a baseline without it:
This lets you distinguish improvement from a task the agent could already do well. You can omit this variant when you only need to compare versions against each other.
The results below come from example runs. Scores are percentages of the rubric's maximum; differences between scores are percentage points. SD means standard deviation, a measure of how much scores vary. A larger SD calls for a closer look at individual results before choosing a variant. See Read an evaluation's results for the commands and guidance.
1. Compare models
Keep the skill, agent, and judge fixed, and change only the model. This helps you choose a model based on its quality, cost, and speed on your tasks.
In this example, four models used the same skill to work with an admin artifact API. The skill contained about 500 lines of procedure. Each scenario ran three times per model.
A cheaper hosted model
85.0
15.8
$1.572
313.1s
61.3
985,422
A frontier model
84.7
20.0
$0.940
149.6s
17.4
701,643
A mid-tier model
73.0
34.6
$0.683
95.9s
14.8
869,633
A small fast model
72.9
21.8
$0.107
42.7s
10.3
385,330
The small fast model averaged 72.9, compared with the frontier model's 84.7, at about 11% of the cost and under a third of the time. That may be a useful tradeoff for tasks where its quality is sufficient.
The model labeled “cheaper hosted” had a lower list price but cost 67% more per solve than the frontier model. It also took about twice as long and used 61.3 turns on average, compared with 17.4. Use the measured cost of completing a task when comparing prices.
The score differences need more care. Means ranged from 72.9 to 85.0, while standard deviations ranged from 15.8 to 34.6. The mid-tier model's individual scores covered the full range from 0 to 100. Inspect the per-scenario results and failed attempts before treating the average scores as a reliable model ranking.
2. Compare agents
Run the same skill with different coding agents to compare how they complete the work. Keep the model fixed where possible. If two agents do not support the same model, the comparison measures the agent and model together.
This example used a prose editing skill to change files in place. Two variants used Agent A with different models; the other two used separate agents.
Agent A, larger model
99.0
1.1
44.6s
17.7
133,118
Agent A, smaller model
95.3
2.4
31.5s
19.5
137,065
Agent B
92.3
3.6
35.5s
5.0
207,486
Agent C
88.7
1.0
12.7s
3.5
57,456
Agent A with the larger model had the highest mean score, 99.0, and a low SD of 1.1. Agent C was the fastest at 12.7 seconds and used the fewest input tokens, but had the lowest mean score at 88.7. The choice depends on the quality your editing task needs and how long you can wait for it.
3. Compare skill rewrites
Mutation testing compares edited versions of a skill on the same scenarios. It helps you check whether a proposed rewrite improves the agent's work.
Here, a skill for filing and updating tickets gave incorrect CLI guidance. For example, it taught --labels, which replaces the label set, for a task that needed --add-labels, which preserves existing labels. Three other flags also needed correction. Mechanics that were missing or incorrect accounted for 56 of the target scenario's 100 rubric points.
The evaluation compared nine skill versions and a no-skill baseline across three scenarios, with three repeats. Each rewrite supplied its own context directory. This excerpt shows five of the ten variants; the others use the same structure.
The target scenario produced these results:
v7-combined
97.2
2.5
v4-label-semantics-only
91.6
12.1
v5-comment-mechanics-only
89.1
13.5
v1-outcome-recipe
87.0
32.7
v0-shipped
86.4
21.2
v3-route-to-cli
85.2
20.4
v2-unteach-labels
83.0
32.0
v6-correct-pagination
64.7
40.4
no-skill
39.4
24.1
v8-gutted
39.3
29.6
The shipped skill scored 47 points above the no-skill baseline. The highest-scoring rewrite, v7-combined, added another 10.8 points and had the lowest SD, 2.5. It is a candidate to test further across the full scenario set before shipping.
Several smaller edits helped explain what to investigate:
v4-label-semantics-onlycorrected label handling, including adding labels and removing a stale label. It scored 91.6 on a rubric with 29 points for label management.v5-comment-mechanics-onlyadded--body-fileand the idempotency-key format. It scored 89.1, with 19 rubric points covering comment mechanics.v1-outcome-recipecombined those changes with a discovery section and scored 87.0. Combining the same advice in a different form did not produce the same mean score asv7-combined.v2-unteach-labelsremoved incorrect guidance without adding the correct flags and scored 83.0.v3-route-to-cliadded a scoping paragraph without flag details and scored 85.2, close to the shipped version.
Three repeats do not support ranking these rewrites against each other. Most of the standard deviations in this table are large. A variant's range often overlaps the rows above and below it. An overlapping range does not test whether two variants really differ. The score gap between v7-combined and the no-skill floor is 57.8 points, but this run does not tell you the order of the rewrites in between. Run more repeats before treating any single pair in the middle of the table as different.
The run also exposed a rubric problem. v6-correct-pagination documented the CLI's actual pagination behavior, but an eight-point rubric item required an --offset flag that the CLI did not have. Fix that item before using the results to judge pagination guidance. Its SD of 40.4 also makes the mean score a poor basis for a decision on its own.
The v8-gutted variant removed the skill's substantive guidance. It scored 39.3, almost the same as the no-skill baseline's 39.4. A deliberately reduced version like this helps check whether a scenario detects the loss of useful instructions. Keep the other scenarios in the evaluation too, so an improvement on the target task does not hide a regression elsewhere.
4. Test whether a skill activates
Change a skill's frontmatter description while keeping its body identical. This tests whether the description helps the agent recognize when to use the skill.
This example compared five descriptions: the shipped text, internal jargon, wording a user might type, the skill name alone, and a deliberately broad description. It used three scenarios with three repeats each. One task clearly needed the skill, one was ambiguous, and one used similar vocabulary but belonged to another skill. That last scenario checks for unwanted activation.
Run this comparison with --skip-forced-context-activation:
Without that flag, the agent is told to use the context, so you cannot measure whether it would choose the skill itself. Do not set agentMustRead on these variants for the same reason.
Each variant had nine observations: three scenarios multiplied by three repeats.
d2-user-words
6/9
96.0
d4-overbroad
5/9
98.3
d1-jargon
2/9
95.9
d0-shipped
0/9
86.9
d3-name-only
0/9
85.2
The shipped and name-only descriptions recorded no activations. The description written in user language recorded six, making it a useful candidate for further testing.
Check which scenarios activated before choosing a description. More activations are only helpful when they occur on appropriate tasks. The overbroad description scored highest despite activating less often than the user-language version; these nine observations per variant do not explain why, or establish that either description is best.
How activation is measured
For the claude agent, Tessl reads the session transcript and records the skill name when the agent calls the Skill tool. This measures explicit invocation. The agent may read or use the content another way without producing that event, so a zero count does not prove the content had no effect.
Other agents do not report activation yet. Their activation is recorded as not measured, which does not mean the skill failed to activate.
tessl eval view lists activated skill names per scenario, combining names across all variants and repeats. It does not show activation counts. The table above was calculated from the underlying per-solution records.
You can add --skip-scoring to check activation without rubric scoring.
5. Compare a skill with access to the codebase
Test whether an agent needs the skill, the source code, or both. Use four variants so you can measure each combination.
This example paired a CLI skill with a portion of the CLI's source. Some guidance was available in the code. Other guidance covered a server-side route and a required preliminary check that the source did not describe. Two scenarios tested those different needs.
The codebase-only variant removes the plugin and adds the source as a separate fixture:
skill + codebase
100.0
$0.654
skill only
98.0
$0.558
neither
92.0
$1.784
codebase only
87.8
$2.155
The skill-only variant scored 98.0 and cost $0.558 per solve. Adding the source increased the score to 100.0 and the cost to $0.654. Source alone scored 87.8 and cost $2.155, nearly four times as much as the skill alone.
For these tasks, the skill-only setup offered a lower-cost option with a score close to the full setup. Check whether the two-point difference represents an important missed requirement before removing the source. These results apply to the tested tasks; they do not establish that the codebase is unnecessary for every task the skill supports.
6. Remove skills from a plugin
Ablation means removing one part at a time to measure how the result changes. For a plugin with several skills, it can help identify which skills contribute to the tasks you care about.
This example used a dbt review plugin with five skills: incremental correctness, SQL performance, dimensional modeling, test coverage, and SQL style. The scenario contained five planted defects, one for each skill, and the rubric penalized invented findings.
Seven variants ran the scenario three times each: the full plugin, five versions with one skill removed, and a no-plugin baseline. Each reduced version needs its own prepared plugin directory, supplied through the context fixture. Setting a fixture to null removes the entire fixture, so it cannot remove just one skill from a plugin.
full plugin
80.0
3.6
minus-modelling
77.3
6.5
minus-sql-performance
77.3
6.1
minus-testing
73.0
1.7
minus-incremental
72.0
5.2
minus-style
64.7
2.9
no plugin
39.3
4.2
The full plugin had the highest mean score, 80.0, compared with 39.3 without it. Removing the style skill produced the largest drop, 15.3 points. That makes style guidance worth investigating before any attempt to reduce the plugin.
The smaller differences, such as the 2.7-point drop after removing modeling or SQL performance, need more evidence before they justify a change. Skills can also interact, so the drops should not be added together as separate contributions to the full plugin's score. Measure cost and duration as well if the goal is to make reviews cheaper or faster.
7. Check for regressions after an edit
Compare a skill before and after a change to check whether the edit affects behavior.
This example used two commits that differed by a version bump and one restored sentence. The sentence explained that a CLI update command changes only three fields and cannot change instruction text.
The run compared the two versions and a no-skill baseline across two scenarios, with three repeats:
One scenario assigned 45 of its 100 points to requirements affected by the restored sentence. The other tested guidance present word for word in both versions. This control helps check whether a difference is specific to the edited guidance.
The post-edit version scored 100.0, the pre-edit version 95.7, and the no-skill baseline 42.5. The observed gain from restoring the sentence was 4.3 points.
Inspect the relevant rubric items and repeat the comparison before attributing the whole difference to that sentence. Keep a regression scenario when it tests behavior you need to preserve. A tie in one run does not show that the guidance or the test is unnecessary.
View the results
Use view to inspect a run and compare to compare separate runs:
Alongside scores, check cost, duration, tokens, and turns where they are available. Read an evaluation's results explains these fields, missing data, and the differences between the two commands.
Plan the run size
Each combination of a variant, scenario, and repeat creates one solution:
The rewrite comparison above uses 90 attempts, the activation comparison 45, and the ablation comparison 21. Before the run starts, the CLI prints its estimated credit cost and, where available, your remaining daily quota. See pricing for current rates.
A run supports:
At most 2,000 attempts.
Unique variant labels matching
^[a-z0-9][a-z0-9_-]{0,63}$. Dots are not allowed, so a model name such askimi-k2.6needs a different label.1 to 10 repeats, set with
--runsor-n.Up to 50
localPathfixture references, resolving to at most 20 distinct local sources.
An organization can create 1,000 attempts per day, resetting at 00:00 UTC. A run with more than 50 scenarios requires --yes to confirm. Calculate the solution count before submitting so you have room for repeats and follow-up runs.
Next steps
Read an evaluation's results explains the output and how to interpret it.
Prove a skill works using evaluation covers scenario generation and running an evaluation.
Troubleshooting reviews and evaluations covers failed runs and unexpected scores.
Last updated