Troubleshooting reviews and evaluations

Most problems with reviews and evaluations come down to a few causes. Work through these when something does not behave as expected.

Takeaway Most problems with reviews and evaluations come down to a few causes: scenarios in the wrong place, missing permissions, a skill that never activated, or normal model variance read as a real result. Work through the questions below when something does not behave as expected.

Why does my evaluation say "no scenarios found"?

tessl scenario download writes scenarios into the plugin's evals/ directory when run from the plugin root or any subdirectory of it. The plugin root is the directory that contains .tessl-plugin/plugin.json. If you ran it from outside the plugin tree, scenarios landed somewhere else and tessl eval run <path> cannot find them.

Re-download from inside the plugin (any subdirectory works):

cd <plugin-root>          # or any subdirectory under it
tessl scenario download --last

Or pass the location explicitly:

tessl scenario download --last --output <plugin-root>/evals

Why do I only see baseline results, with no "with context" column?

The evaluation completed, but every scenario shows a baseline score and nothing for the skill. This means the run could not find the skill to inject, usually because the plugin layout is wrong.

Check that evals/, .tessl-plugin/, and the skill directory all sit under the same plugin root, and that skills[] in plugin.json points at the skill directory correctly, relative to the plugin root.

Why does my run fail on permissions?

Evaluations require at least publisher access to the workspace. Member access is not enough to run tessl eval run or tessl scenario generate. A review needs only member access, so a skill you can review may still fail to evaluate.

Confirm who you are logged in as with tessl whoami, and check your access to the workspace the plugin is attributed to in .tessl-plugin/plugin.json.

Why do scores look worse with the skill than without?

A small negative difference is usually normal model variance, not evidence the skill is harmful. An evaluation runs a real agent, so results vary from run to run. Run each scenario several times with --runs and read the overall pattern rather than a single result. A large or consistent negative difference is a real signal that the skill is confusing the agent, and the content needs work.

Why does the skill make no difference at all?

If the "with context" score sits at zero improvement across repeated runs, the skill most likely never activated, so the agent never used it. That is a different problem from variance. Check activation: run tessl review run and look at the description score, and make sure the skill's description states clear trigger conditions for when it should be used. See Check a skill's quality using review.

A scenario keeps failing to complete

It is normal for an agent not to solve every scenario on every attempt. Retry the run, or adjust the scenario if it is consistently failing for the wrong reason.

tessl eval retry abc123
tessl eval retry --last

Still using tessl skill review?

tessl skill review has been removed and replaced by tessl review run, which requires authentication. If a script or workflow still calls the old command, see Migrate from skill review for what changed and how to move across.

Getting help

If none of the above surfaces the cause, share the output of tessl doctor --json when you reach out, since it reports your authentication status. See Giving feedback.