Troubleshooting reviews and evaluations

Most problems with reviews and evaluations come down to a few causes. Work through these when something does not behave as expected.

Takeaway Most problems with reviews and evaluations come down to a few causes: scenarios in the wrong place, missing permissions, a skill that never activated, or normal model variance read as a real result. Work through the questions below when something does not behave as expected.

Why does my evaluation say "no scenarios found"?

tessl scenario download writes scenarios into an evals/ directory relative to your current directory, but tessl eval run <path> looks for evals/ inside the plugin directory you point it at. If you downloaded scenarios somewhere else, they are not where the run expects them.

Run tessl scenario download from inside the plugin root, the directory containing .tessl-plugin/, or pass the location explicitly:

tessl scenario download --output <plugin-root>/evals

Why do I only see baseline results, with no "with context" column?

The evaluation completed, but every scenario shows a baseline score and nothing for the skill. This means the run could not find the skill to inject, usually because the plugin layout is wrong.

Check that evals/, .tessl-plugin/, and the skill directory all sit under the same plugin root, and that skills[] in plugin.json points at the skill directory correctly, relative to the plugin root.

Why does my run fail on permissions?

Evaluations require at least publisher access to the workspace. Member access is not enough to run tessl eval run or tessl scenario generate. A review needs only member access, so a skill you can review may still fail to evaluate.

Confirm who you are logged in as with tessl whoami, and check your access to the workspace the plugin is attributed to in .tessl-plugin/plugin.json.

Why do scores look worse with the skill than without?

A small negative difference is usually normal model variance, not evidence the skill is harmful. An evaluation runs a real agent, so results vary from run to run. Run each scenario several times with --runs and read the overall pattern rather than a single result. A large or consistent negative difference is a real signal that the skill is confusing the agent, and the content needs work.

Why does the skill make no difference at all?

If the "with context" score sits at zero improvement across repeated runs, the skill most likely never activated, so the agent never used it. That is a different problem from variance. Check activation: run tessl review run and look at the description score, and make sure the skill's description states clear trigger conditions for when it should be used. See Check a skill's quality using review.

A scenario keeps failing to complete

It is normal for an agent not to solve every scenario on every attempt. Retry the run, or adjust the scenario if it is consistently failing for the wrong reason.

tessl eval retry abc123
tessl eval retry --last

Still using tessl skill review?

tessl skill review is the local, unauthenticated review that ran before tessl review. It has been replaced. If you are still on it, see Migrate from skill review for what changed and how to move across.

Getting help

If none of the above surfaces the cause, share the output of tessl doctor --json when you reach out, since it reports your authentication status. See Giving feedback.