> For the complete documentation index, see [llms.txt](https://docs.tessl.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tessl.io/codifying-and-enforcing-your-skill-standards/verify-agent-changes-in-ci.md). # Verify agent changes in CI Write and calibrate a verifier, then run it as an advisory or blocking check on pull requests. {% hint style="info" %} **Takeaway** Start a verifier as a warning, inspect the files it covers, and promote it to an error only after it produces reliable findings on real changes. {% endhint %} This guide creates a verifier for a repository invariant, connects it to `tessl.json`, and runs it against pull request changes. If you are deciding whether a verifier fits your standard, start with [Verifiers overview](/codifying-and-enforcing-your-skill-standards/verifiers-overview.md). ## 1. Write the verifier Create `verifiers/actionable-errors.json`: ```json { "name": "User-facing errors include a next step", "instruction": "When code returns a user-facing error, it should tell the user what to do next.", "relevant_when": "TypeScript code that constructs a UserFacingError or prints an error for a CLI command.", "context": "CLI errors should be actionable for users running commands in a terminal.", "scope": "file", "scope_reason": "Each command file contains the error and its recovery guidance.", "checklist": [ { "name": "actionable-error", "rule": "Each user-facing error includes the command to run next, the flag to change, or the file to inspect." } ] } ``` Keep `relevant_when` specific. It should name the observable trait that makes a target applicable, rather than a broad category such as "TypeScript file." Validate the file before running it: ```bash $ tessl change verify lint verifiers/actionable-errors.json ``` ## 2. Configure its scope as a warning Add a group to the `verify` block in `tessl.json`: ```json { "name": "engteam/my-project", "verify": { "groups": { "error-ux": { "path": "verifiers/actionable-errors.json", "level": "warn", "include": ["src/**/*.ts"], "exclude": ["**/*.test.ts"] } } } } ``` Severity belongs in `tessl.json`, not in the verifier file. Starting at `warn` reports findings without blocking the pull request. ## 3. Inspect the matched files Preview a full-repository run before spending judge calls: ```bash $ tessl change verify --dry-run --all --show-files ``` Check that the output contains the files you intend to cover and excludes unrelated files. Adjust the group's `include` and `exclude` globs or the verifier's `relevant_when` if the scope is too broad or narrow. Then judge a small sample: ```bash $ tessl change verify --all --sample 5 ``` This sends the selected source content to Tessl's LLM judge. Limit the verifier's scope to content your organization is authorized to send, and exclude credentials, personal data, and other sensitive material. Review each result against the invariant. Refine the verifier until the same evidence produces consistent pass or fail decisions. ## 4. Run it on pull requests CI needs a Tessl API key in the `TESSL_TOKEN` environment variable. For key creation and storage guidance, see [Authenticating in CI](/codifying-and-enforcing-your-skill-standards/gate-skill-quality-in-ci.md#authenticating-in-ci). A pull request can change its workflow, verifier, and manifest configuration, so its branch location does not establish trust. Use a `pull_request` workflow, not `pull_request_target`, and protect the credentialed job with a GitHub environment that requires approval from reviewers who are authorized to approve the configuration and source changes. The following job and steps assume the workflow runs on `pull_request`, has checked out the repository, and installed Tessl with `tesslio/setup-tessl@v2`. The job skips an external head repository by comparing repository identities, so it still runs for same-repository pull requests when the repository itself is a fork. Create the `tessl-verify` environment in GitHub with required reviewers, store `TESSL_TOKEN` as an environment secret, and do not approve it until the pull request's workflow, verifier, and manifest changes have been reviewed: ```yaml verify: runs-on: ubuntu-latest if: github.event.pull_request.head.repo.full_name == github.repository environment: tessl-verify env: TESSL_TOKEN: ${{ secrets.TESSL_TOKEN }} steps: - name: Fetch base branch env: BASE_REF: ${{ github.event.pull_request.base.ref }} run: git fetch --no-tags origin "$BASE_REF:refs/remotes/origin/$BASE_REF" - name: Inspect verifier scope env: BASE_REF: ${{ github.event.pull_request.base.ref }} run: tessl change verify --base "origin/$BASE_REF" --dry-run --show-files --max-calls 200 - name: Verify changed files env: BASE_REF: ${{ github.event.pull_request.base.ref }} run: tessl change verify --base "origin/$BASE_REF" --github --max-calls 200 ``` `--base "origin/$BASE_REF"` compares against the pull request's actual target branch, which the fetch step records as a remote-tracking ref. For source-only changes, the command verifies the pull request's changed files. When the pull request changes a verifier JSON file or the `verify` configuration, the affected verifier runs across its full configured scope. `--github` emits annotations for findings. A `warn` finding is advisory, while an `error` finding makes the command exit non-zero. `--max-calls 200` is the gate's call budget. The command stops before judging if its planned scope exceeds that limit; narrow the group's scope or choose and document a larger limit before changing the workflow. If you run the command from a monorepo root with linked child manifests, add `--recursive` to each invocation. ## 5. Promote the verifier to a blocking check Leave the verifier at `warn` while it runs on real pull requests. Track false positives, missed applicable files, and unclear findings. When the verifier is reliable, change the group level: ```json { "verify": { "groups": { "error-ux": { "path": "verifiers/actionable-errors.json", "level": "error", "include": ["src/**/*.ts"], "exclude": ["**/*.test.ts"] } } } } ``` The existing CI command now blocks pull requests with failing findings. You do not need to change the workflow or add `--fail-on-warn`. ## Next * Read the [`verify` block reference](/reference/configuration.md#verify-block) for group filters and per-verifier overrides. * Add [Tessl Code Review](/tutorials/setting-up-agentic-code-review.md) when you also want general pull request review beyond named repository invariants. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.tessl.io/codifying-and-enforcing-your-skill-standards/verify-agent-changes-in-ci.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.