> For the complete documentation index, see [llms.txt](https://docs.tessl.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tessl.io/improving-your-skills/comparing-with-test-variants.md).

# Compare with test variants

Compare models, agents, skill revisions, and context using test variants, with examples of how to interpret the results.

Use test variants to try different configurations on the same tasks. You can compare models, choose between skill rewrites, or check whether an agent uses a skill without being prompted.

A **test variant** specifies the model, agent, and files available to the agent. Each variant solves the same scenarios and is scored against the same rubric. A run can include as many variants as fit in its [solve limit](#plan-the-run-size).

If you are new to evaluations, start with [Prove a skill works using evaluation](/improving-your-skills/evaluate-skill-quality-using-scenarios.md). The examples below build on that workflow.

## Set up a comparison

Declare your variants with `--variant-json`, as a JSON array or the path to a JSON file:

```bash
tessl eval run ./my-plugin --variant-json ./variants.json
```

The examples below show the JSON array for each comparison. Replace placeholder models, agents, and paths with your own values. Run `tessl eval run --list-agents` to see the agents and models you can use. The run solves only the variants you list, so add a baseline yourself when you need one. See the [`--variant-json` CLI reference](/reference/cli-commands.md#tessl-eval-run) for every field and its label rules. The full fixture reference is in [Comparing test variants](/improving-your-skills/evaluate-skill-quality-using-scenarios.md#comparing-test-variants).

A fixture is a set of files supplied to the agent. The plugin you evaluate is the fixture named `context`. The `fixtures` field changes fixtures by name for one variant:

* A value replaces the fixture, or adds it when the name is new. Set `context` to another directory to test a different version of the plugin.
* `null` removes the fixture. Set `context` to `null` to measure the agent without the plugin. Other fixtures remain available.
* A name you leave out keeps the fixture the scenario and run supply.

Refer to a local directory with `{ "localPath": "./path" }`. Paths are relative to the directory where you run the command. The CLI uploads each distinct path once.

## Make the results useful

Run each scenario more than once. Agents can produce different results from the same input, so a single attempt tells you little about consistency. Start with three repeats (`-n 3`) and increase the count when results vary widely or the decision needs more confidence.

When you want to measure how much a skill helps, include a **baseline** without it:

```json
{ "label": "no-skill", "fixtures": { "context": null } }
```

This lets you distinguish improvement from a task the agent could already do well. You can omit this variant when you only need to compare versions against each other.

The results below come from example runs. Scores are percentages of the rubric's maximum; differences between scores are percentage points. **SD** means standard deviation, a measure of how much scores vary. A larger SD calls for a closer look at individual results before choosing a variant. See [Read an evaluation's results](/improving-your-skills/reading-evaluation-results.md) for the commands and guidance.

## 1. Compare models

Keep the skill, agent, and judge fixed, and change only the model. This helps you choose a model based on its quality, cost, and speed on your tasks.

In this example, four models used the same skill to work with an admin artifact API. The skill contained about 500 lines of procedure. Each scenario ran three times per model.

```json
[
  { "label": "frontier", "model": "<model>" },
  { "label": "mid-tier", "model": "<model>" },
  { "label": "small-fast", "model": "<model>" },
  { "label": "cheaper-hosted", "model": "<model>" }
]
```

| Model                  | Score | SD   | Cost / solve | Duration | Turns | Input tokens |
| ---------------------- | ----- | ---- | ------------ | -------- | ----- | ------------ |
| A cheaper hosted model | 85.0  | 15.8 | $1.572       | 313.1s   | 61.3  | 985,422      |
| A frontier model       | 84.7  | 20.0 | $0.940       | 149.6s   | 17.4  | 701,643      |
| A mid-tier model       | 73.0  | 34.6 | $0.683       | 95.9s    | 14.8  | 869,633      |
| A small fast model     | 72.9  | 21.8 | $0.107       | 42.7s    | 10.3  | 385,330      |

The small fast model averaged 72.9, compared with the frontier model's 84.7, at about 11% of the cost and under a third of the time. That may be a useful tradeoff for tasks where its quality is sufficient.

The model labeled “cheaper hosted” had a lower list price but cost 67% more per solve than the frontier model. It also took about twice as long and used 61.3 turns on average, compared with 17.4. Use the measured cost of completing a task when comparing prices.

The score differences need more care. Means ranged from 72.9 to 85.0, while standard deviations ranged from 15.8 to 34.6. The mid-tier model's individual scores covered the full range from 0 to 100. Inspect the per-scenario results and failed attempts before treating the average scores as a reliable model ranking.

## 2. Compare agents

Run the same skill with different coding agents to compare how they complete the work. Keep the model fixed where possible. If two agents do not support the same model, the comparison measures the agent and model together.

This example used a prose editing skill to change files in place. Two variants used Agent A with different models; the other two used separate agents.

```json
[
  { "label": "agent-a-larger-model", "agent": "<agent>", "model": "<model>" },
  { "label": "agent-a-smaller-model", "agent": "<agent>", "model": "<model>" },
  { "label": "agent-b", "agent": "<agent>", "model": "<model>" },
  { "label": "agent-c", "agent": "<agent>", "model": "<model>" }
]
```

| Test variant           | Score | SD  | Duration | Turns | Input tokens |
| ---------------------- | ----- | --- | -------- | ----- | ------------ |
| Agent A, larger model  | 99.0  | 1.1 | 44.6s    | 17.7  | 133,118      |
| Agent A, smaller model | 95.3  | 2.4 | 31.5s    | 19.5  | 137,065      |
| Agent B                | 92.3  | 3.6 | 35.5s    | 5.0   | 207,486      |
| Agent C                | 88.7  | 1.0 | 12.7s    | 3.5   | 57,456       |

Agent A with the larger model had the highest mean score, 99.0, and a low SD of 1.1. Agent C was the fastest at 12.7 seconds and used the fewest input tokens, but had the lowest mean score at 88.7. The choice depends on the quality your editing task needs and how long you can wait for it.

## 3. Compare skill rewrites

Mutation testing compares edited versions of a skill on the same scenarios. It helps you check whether a proposed rewrite improves the agent's work.

Here, a skill for filing and updating tickets gave incorrect CLI guidance. For example, it taught `--labels`, which replaces the label set, for a task that needed `--add-labels`, which preserves existing labels. Three other flags also needed correction. Mechanics that were missing or incorrect accounted for 56 of the target scenario's 100 rubric points.

The evaluation compared nine skill versions and a no-skill baseline across three scenarios, with three repeats. Each rewrite supplied its own `context` directory. This excerpt shows five of the ten variants; the others use the same structure.

```json
[
  { "label": "no-skill", "fixtures": { "context": null } },
  { "label": "v0-shipped" },
  { "label": "v1-outcome-recipe", "fixtures": { "context": { "localPath": "./variants/v1-outcome-recipe" } } },
  { "label": "v2-unteach-labels", "fixtures": { "context": { "localPath": "./variants/v2-unteach-labels" } } },
  { "label": "v8-gutted", "fixtures": { "context": { "localPath": "./variants/v8-gutted" } } }
]
```

The target scenario produced these results:

| Test variant              | Score | SD   |
| ------------------------- | ----- | ---- |
| v7-combined               | 97.2  | 2.5  |
| v4-label-semantics-only   | 91.6  | 12.1 |
| v5-comment-mechanics-only | 89.1  | 13.5 |
| v1-outcome-recipe         | 87.0  | 32.7 |
| v0-shipped                | 86.4  | 21.2 |
| v3-route-to-cli           | 85.2  | 20.4 |
| v2-unteach-labels         | 83.0  | 32.0 |
| v6-correct-pagination     | 64.7  | 40.4 |
| no-skill                  | 39.4  | 24.1 |
| v8-gutted                 | 39.3  | 29.6 |

The shipped skill scored 47 points above the no-skill baseline. The highest-scoring rewrite, `v7-combined`, added another 10.8 points and had the lowest SD, 2.5. It is a candidate to test further across the full scenario set before shipping.

Several smaller edits helped explain what to investigate:

* `v4-label-semantics-only` corrected label handling, including adding labels and removing a stale label. It scored 91.6 on a rubric with 29 points for label management.
* `v5-comment-mechanics-only` added `--body-file` and the idempotency-key format. It scored 89.1, with 19 rubric points covering comment mechanics.
* `v1-outcome-recipe` combined those changes with a discovery section and scored 87.0. Combining the same advice in a different form did not produce the same mean score as `v7-combined`.
* `v2-unteach-labels` removed incorrect guidance without adding the correct flags and scored 83.0. `v3-route-to-cli` added a scoping paragraph without flag details and scored 85.2, close to the shipped version.

Three repeats do not support ranking these rewrites against each other. Most of the standard deviations in this table are large. A variant's range often overlaps the rows above and below it. An overlapping range does not test whether two variants really differ. The score gap between v7-combined and the no-skill floor is 57.8 points, but this run does not tell you the order of the rewrites in between. Run more repeats before treating any single pair in the middle of the table as different.

The run also exposed a rubric problem. `v6-correct-pagination` documented the CLI's actual pagination behavior, but an eight-point rubric item required an `--offset` flag that the CLI did not have. Fix that item before using the results to judge pagination guidance. Its SD of 40.4 also makes the mean score a poor basis for a decision on its own.

The `v8-gutted` variant removed the skill's substantive guidance. It scored 39.3, almost the same as the no-skill baseline's 39.4. A deliberately reduced version like this helps check whether a scenario detects the loss of useful instructions. Keep the other scenarios in the evaluation too, so an improvement on the target task does not hide a regression elsewhere.

## 4. Test whether a skill activates

Change a skill's frontmatter description while keeping its body identical. This tests whether the description helps the agent recognize when to use the skill.

This example compared five descriptions: the shipped text, internal jargon, wording a user might type, the skill name alone, and a deliberately broad description. It used three scenarios with three repeats each. One task clearly needed the skill, one was ambiguous, and one used similar vocabulary but belonged to another skill. That last scenario checks for unwanted activation.

```json
[
  { "label": "d0-shipped" },
  { "label": "d1-jargon", "fixtures": { "context": { "localPath": "./variants/d1-jargon" } } },
  { "label": "d2-user-words", "fixtures": { "context": { "localPath": "./variants/d2-user-words" } } },
  { "label": "d3-name-only", "fixtures": { "context": { "localPath": "./variants/d3-name-only" } } },
  { "label": "d4-overbroad", "fixtures": { "context": { "localPath": "./variants/d4-overbroad" } } }
]
```

Run this comparison with `--skip-forced-context-activation`:

```bash
tessl eval run ./my-plugin --variant-json ./descriptions.json --skip-forced-context-activation
```

Without that flag, the agent is told to use the context, so you cannot measure whether it would choose the skill itself. Do not set `agentMustRead` on these variants for the same reason.

Each variant had nine observations: three scenarios multiplied by three repeats.

| Test variant  | Activated | Score |
| ------------- | --------- | ----- |
| d2-user-words | 6/9       | 96.0  |
| d4-overbroad  | 5/9       | 98.3  |
| d1-jargon     | 2/9       | 95.9  |
| d0-shipped    | 0/9       | 86.9  |
| d3-name-only  | 0/9       | 85.2  |

The shipped and name-only descriptions recorded no activations. The description written in user language recorded six, making it a useful candidate for further testing.

Check which scenarios activated before choosing a description. More activations are only helpful when they occur on appropriate tasks. The overbroad description scored highest despite activating less often than the user-language version; these nine observations per variant do not explain why, or establish that either description is best.

### How activation is measured

For the `claude` agent, Tessl reads the session transcript and records the skill name when the agent calls the `Skill` tool. This measures explicit invocation. The agent may read or use the content another way without producing that event, so a zero count does not prove the content had no effect.

Other agents do not report activation yet. Their activation is recorded as not measured, which does not mean the skill failed to activate.

`tessl eval view` lists activated skill names per scenario, combining names across all variants and repeats. It does not show activation counts. The table above was calculated from the underlying per-solution records.

You can add `--skip-scoring` to check activation without rubric scoring.

## 5. Compare a skill with access to the codebase

Test whether an agent needs the skill, the source code, or both. Use four variants so you can measure each combination.

This example paired a CLI skill with a portion of the CLI's source. Some guidance was available in the code. Other guidance covered a server-side route and a required preliminary check that the source did not describe. Two scenarios tested those different needs.

The `codebase-only` variant removes the plugin and adds the source as a separate fixture:

```json
[
  { "label": "neither", "fixtures": { "context": null } },
  { "label": "codebase-only", "fixtures": { "context": null, "codebase": { "localPath": "./cli-source" } } },
  { "label": "skill-only" },
  { "label": "both", "fixtures": { "codebase": { "localPath": "./cli-source" } } }
]
```

| Test variant     | Score | Cost / solve |
| ---------------- | ----- | ------------ |
| skill + codebase | 100.0 | $0.654       |
| skill only       | 98.0  | $0.558       |
| neither          | 92.0  | $1.784       |
| codebase only    | 87.8  | $2.155       |

The skill-only variant scored 98.0 and cost $0.558 per solve. Adding the source increased the score to 100.0 and the cost to $0.654. Source alone scored 87.8 and cost $2.155, nearly four times as much as the skill alone.

For these tasks, the skill-only setup offered a lower-cost option with a score close to the full setup. Check whether the two-point difference represents an important missed requirement before removing the source. These results apply to the tested tasks; they do not establish that the codebase is unnecessary for every task the skill supports.

## 6. Remove skills from a plugin

**Ablation** means removing one part at a time to measure how the result changes. For a plugin with several skills, it can help identify which skills contribute to the tasks you care about.

This example used a dbt review plugin with five skills: incremental correctness, SQL performance, dimensional modeling, test coverage, and SQL style. The scenario contained five planted defects, one for each skill, and the rubric penalized invented findings.

Seven variants ran the scenario three times each: the full plugin, five versions with one skill removed, and a no-plugin baseline. Each reduced version needs its own prepared plugin directory, supplied through the `context` fixture. Setting a fixture to `null` removes the entire fixture, so it cannot remove just one skill from a plugin.

| Test variant            | Mean | SD  |
| ----------------------- | ---- | --- |
| full plugin             | 80.0 | 3.6 |
| `minus-modelling`       | 77.3 | 6.5 |
| `minus-sql-performance` | 77.3 | 6.1 |
| `minus-testing`         | 73.0 | 1.7 |
| `minus-incremental`     | 72.0 | 5.2 |
| `minus-style`           | 64.7 | 2.9 |
| no plugin               | 39.3 | 4.2 |

The full plugin had the highest mean score, 80.0, compared with 39.3 without it. Removing the style skill produced the largest drop, 15.3 points. That makes style guidance worth investigating before any attempt to reduce the plugin.

The smaller differences, such as the 2.7-point drop after removing modeling or SQL performance, need more evidence before they justify a change. Skills can also interact, so the drops should not be added together as separate contributions to the full plugin's score. Measure cost and duration as well if the goal is to make reviews cheaper or faster.

## 7. Check for regressions after an edit

Compare a skill before and after a change to check whether the edit affects behavior.

This example used two commits that differed by a version bump and one restored sentence. The sentence explained that a CLI update command changes only three fields and cannot change instruction text.

The run compared the two versions and a no-skill baseline across two scenarios, with three repeats:

```json
[
  { "label": "no-skill", "fixtures": { "context": null } },
  { "label": "pre-edit", "fixtures": { "context": { "localPath": "./plugins/pre-edit" } } },
  { "label": "post-edit" }
]
```

One scenario assigned 45 of its 100 points to requirements affected by the restored sentence. The other tested guidance present word for word in both versions. This control helps check whether a difference is specific to the edited guidance.

The post-edit version scored 100.0, the pre-edit version 95.7, and the no-skill baseline 42.5. The observed gain from restoring the sentence was 4.3 points.

Inspect the relevant rubric items and repeat the comparison before attributing the whole difference to that sentence. Keep a regression scenario when it tests behavior you need to preserve. A tie in one run does not show that the guidance or the test is unnecessary.

## View the results

Use `view` to inspect a run and `compare` to compare separate runs:

```bash
# One run, with a summary for each test variant
tessl eval view <run-id>

# Detailed statistics in the per-scenario tables
tessl eval view <run-id> --full

# Compare two to ten separate runs
tessl eval compare <run-id> <run-id> --full
```

Alongside scores, check cost, duration, tokens, and turns where they are available. [Read an evaluation's results](/improving-your-skills/reading-evaluation-results.md) explains these fields, missing data, and the differences between the two commands.

## Plan the run size

Each combination of a variant, scenario, and repeat creates one solution:

```
solutions = test variants x scenarios x repeats
```

The rewrite comparison above uses 90 attempts, the activation comparison 45, and the ablation comparison 21. Before the run starts, the CLI prints its estimated credit cost and, where available, your remaining daily quota. See [pricing](https://tessl.io/pricing/) for current rates.

A run supports:

* At most 2,000 attempts.
* Unique variant labels matching `^[a-z0-9][a-z0-9_-]{0,63}$`. Dots are not allowed, so a model name such as `kimi-k2.6` needs a different label.
* 1 to 10 repeats, set with `--runs` or `-n`.
* Up to 50 `localPath` fixture references, resolving to at most 20 distinct local sources.

An organization can create 1,000 attempts per day, resetting at 00:00 UTC. A run with more than 50 scenarios requires `--yes` to confirm. Calculate the solution count before submitting so you have room for repeats and follow-up runs.

## Next steps

* [Read an evaluation's results](/improving-your-skills/reading-evaluation-results.md) explains the output and how to interpret it.
* [Prove a skill works using evaluation](/improving-your-skills/evaluate-skill-quality-using-scenarios.md) covers scenario generation and running an evaluation.
* [Troubleshooting reviews and evaluations](/improving-your-skills/troubleshooting-reviews-and-evals.md) covers failed runs and unexpected scores.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.tessl.io/improving-your-skills/comparing-with-test-variants.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.