Verify agent changes in CI

Write and calibrate a verifier, then run it as an advisory or blocking check on pull requests.

Takeaway Start a verifier as a warning, inspect the files it covers, and promote it to an error only after it produces reliable findings on real changes.

This guide creates a verifier for a repository invariant, connects it to tessl.json, and runs it against pull request changes. If you are deciding whether a verifier fits your standard, start with Verifiers overview.

1. Write the verifier

Create verifiers/actionable-errors.json:

{
  "name": "User-facing errors include a next step",
  "instruction": "When code returns a user-facing error, it should tell the user what to do next.",
  "relevant_when": "TypeScript code that constructs a UserFacingError or prints an error for a CLI command.",
  "context": "CLI errors should be actionable for users running commands in a terminal.",
  "scope": "file",
  "scope_reason": "Each command file contains the error and its recovery guidance.",
  "checklist": [
    {
      "name": "actionable-error",
      "rule": "Each user-facing error includes the command to run next, the flag to change, or the file to inspect."
    }
  ]
}

Keep relevant_when specific. It should name the observable trait that makes a target applicable, rather than a broad category such as "TypeScript file."

Validate the file before running it:

$ tessl change verify lint verifiers/actionable-errors.json

2. Configure its scope as a warning

Add a group to the verify block in tessl.json:

{
  "name": "engteam/my-project",
  "verify": {
    "groups": {
      "error-ux": {
        "path": "verifiers/actionable-errors.json",
        "level": "warn",
        "include": ["src/**/*.ts"],
        "exclude": ["**/*.test.ts"]
      }
    }
  }
}

Severity belongs in tessl.json, not in the verifier file. Starting at warn reports findings without blocking the pull request.

3. Inspect the matched files

Preview a full-repository run before spending judge calls:

$ tessl change verify --dry-run --all --show-files

Check that the output contains the files you intend to cover and excludes unrelated files. Adjust the group's include and exclude globs or the verifier's relevant_when if the scope is too broad or narrow.

Then judge a small sample:

$ tessl change verify --all --sample 5

This sends the selected source content to Tessl's LLM judge. Limit the verifier's scope to content your organization is authorized to send, and exclude credentials, personal data, and other sensitive material.

Review each result against the invariant. Refine the verifier until the same evidence produces consistent pass or fail decisions.

4. Run it on pull requests

CI needs a Tessl API key in the TESSL_TOKEN environment variable. For key creation and storage guidance, see Authenticating in CI. A pull request can change its workflow, verifier, and manifest configuration, so its branch location does not establish trust. Use a pull_request workflow, not pull_request_target, and protect the credentialed job with a GitHub environment that requires approval from reviewers who are authorized to approve the configuration and source changes.

The following job and steps assume the workflow runs on pull_request, has checked out the repository, and installed Tessl with tesslio/setup-tessl@v2. The job skips an external head repository by comparing repository identities, so it still runs for same-repository pull requests when the repository itself is a fork. Create the tessl-verify environment in GitHub with required reviewers, store TESSL_TOKEN as an environment secret, and do not approve it until the pull request's workflow, verifier, and manifest changes have been reviewed:

verify:
  runs-on: ubuntu-latest
  if: github.event.pull_request.head.repo.full_name == github.repository
  environment: tessl-verify
  env:
    TESSL_TOKEN: ${{ secrets.TESSL_TOKEN }}
  steps:
    - name: Fetch base branch
      env:
        BASE_REF: ${{ github.event.pull_request.base.ref }}
      run: git fetch --no-tags origin "$BASE_REF:refs/remotes/origin/$BASE_REF"

    - name: Inspect verifier scope
      env:
        BASE_REF: ${{ github.event.pull_request.base.ref }}
      run: tessl change verify --base "origin/$BASE_REF" --dry-run --show-files --max-calls 200

    - name: Verify changed files
      env:
        BASE_REF: ${{ github.event.pull_request.base.ref }}
      run: tessl change verify --base "origin/$BASE_REF" --github --max-calls 200

--base "origin/$BASE_REF" compares against the pull request's actual target branch, which the fetch step records as a remote-tracking ref. For source-only changes, the command verifies the pull request's changed files. When the pull request changes a verifier JSON file or the verify configuration, the affected verifier runs across its full configured scope. --github emits annotations for findings. A warn finding is advisory, while an error finding makes the command exit non-zero.

--max-calls 200 is the gate's call budget. The command stops before judging if its planned scope exceeds that limit; narrow the group's scope or choose and document a larger limit before changing the workflow.

If you run the command from a monorepo root with linked child manifests, add --recursive to each invocation.

5. Promote the verifier to a blocking check

Leave the verifier at warn while it runs on real pull requests. Track false positives, missed applicable files, and unclear findings. When the verifier is reliable, change the group level:

{
  "verify": {
    "groups": {
      "error-ux": {
        "path": "verifiers/actionable-errors.json",
        "level": "error",
        "include": ["src/**/*.ts"],
        "exclude": ["**/*.test.ts"]
      }
    }
  }
}

The existing CI command now blocks pull requests with failing findings. You do not need to change the workflow or add --fail-on-warn.

Next

• Read the verify block reference for group filters and per-verifier overrides.
• Add Tessl Code Review when you also want general pull request review beyond named repository invariants.