> For the complete documentation index, see [llms.txt](https://docs.tessl.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tessl.io/codifying-and-enforcing-your-skill-standards/verify-agent-changes-in-ci.md). # Verify agent changes in CI Write and calibrate a verifier, then run it as an advisory or blocking check on pull requests. {% hint style="info" %} **Takeaway** Start a verifier as a warning, inspect the files it covers, and promote it to an error only after it produces reliable findings on real changes. {% endhint %} This guide creates a verifier for a repository invariant, connects it to `tessl.json`, and runs it against pull request changes. If you are deciding whether a verifier fits your standard, start with [Verifiers overview](/codifying-and-enforcing-your-skill-standards/verifiers-overview.md). ## 1. Write the verifier Create `verifiers/actionable-errors.json`: ```json { "name": "User-facing errors include a next step", "instruction": "When code returns a user-facing error, it should tell the user what to do next.", "relevant_when": "TypeScript code that constructs a UserFacingError or prints an error for a CLI command.", "context": "CLI errors should be actionable for users running commands in a terminal.", "scope": "file", "scope_reason": "Each command file contains the error and its recovery guidance.", "checklist": [ { "name": "actionable-error", "rule": "Each user-facing error includes the command to run next, the flag to change, or the file to inspect." } ] } ``` Keep `relevant_when` specific. It should name the observable trait that makes a target applicable, rather than a broad category such as "TypeScript file." Validate the file before running it: ```bash tessl change verify lint verifiers/actionable-errors.json ``` ## 2. Configure its scope as a warning Add a group to the `verify` block in `tessl.json`: ```json { "name": "engteam/my-project", "verify": { "groups": { "error-ux": { "path": "verifiers/actionable-errors.json", "level": "warn", "include": ["src/**/*.ts"], "exclude": ["**/*.test.ts"] } } } } ``` Severity belongs in `tessl.json`, not in the verifier file. Starting at `warn` reports findings without blocking the pull request. ## 3. Inspect the matched files Preview a full-repository run before spending judge calls: ```bash tessl change verify --dry-run --all --show-files ``` Check that the output contains the files you intend to cover and excludes unrelated files. Adjust the group's `include` and `exclude` globs or the verifier's `relevant_when` if the scope is too broad or narrow. Then judge a small sample: ```bash tessl change verify --all --sample 5 ``` This sends the selected source content to Tessl's LLM judge. Limit the verifier's scope to content your organization is authorized to send, and exclude credentials, personal data, and other sensitive material. Review each result against the invariant. Refine the verifier until the same evidence produces consistent pass or fail decisions. ## 4. Run it on pull requests Run verifiers on pull requests with the Tessl Reviewer GitHub App, or with a workflow you own. The App needs no workflow and no stored token. ### With the Tessl Reviewer GitHub App 1. Sign in to [tessl.io](http://tessl.io), select your organization, and open **Reviewer** in the sidebar. 2. Open the **Verify PRs** tab. If the App is not installed yet, select **Connect GitHub** and install it as described in [Set up Tessl Code Review](/tutorials/setting-up-agentic-code-review.md#step-1-install-the-app). 3. Select **Turn on** next to the repository, switch on **Turn on for this repository**, and select **Save settings**. Turning Verify PRs on and changing its settings takes an organization admin. Verify PRs runs when a pull request opens or gets new commits, and when someone comments `@tessl-verify`. It checks the pull request against its base branch the same way as the workflow below run with `--recursive`, so it covers every linked manifest in the repository, and reports the result as a **Tessl Verify** GitHub Check. A finding from a `warn` verifier shows as a warning and the Check still passes. A finding from an `error` verifier fails the Check. If `tessl.json` declares no verifiers, the Check is skipped. The repository's settings also set: * **Run on draft pull requests**: when off, a draft gets a skipped Check until it is marked ready for review or someone comments `@tessl-verify`. * **Judge calls per run**: the most judge calls one run makes. A run that needs more stops without judging, and its Check says it did not finish. * **Time limit, seconds**: how long a run may spend judging. The **Verify PRs** tab shows each repository's last result. When a repository has no verifiers yet, its settings offer **Write verifiers with an agent**: a prompt that has a coding agent propose verifiers for the repository and open a pull request. ### In your own GitHub workflow CI needs a Tessl API key in the `TESSL_TOKEN` environment variable. For key creation and storage guidance, see [Authenticating in CI](/codifying-and-enforcing-your-skill-standards/gate-skill-quality-in-ci.md#authenticating-in-ci). A pull request can change its workflow, verifier, and manifest configuration, so its branch location does not establish trust. Use a `pull_request` workflow, not `pull_request_target`, and protect the credentialed job with a GitHub environment that requires approval from reviewers who are authorized to approve the configuration and source changes. The following job and steps assume the workflow runs on `pull_request`, has checked out the repository, and installed Tessl with `tesslio/setup-tessl@v2`. The job skips an external head repository by comparing repository identities, so it still runs for same-repository pull requests when the repository itself is a fork. Create the `tessl-verify` environment in GitHub with required reviewers, store `TESSL_TOKEN` as an environment secret, and do not approve it until the pull request's workflow, verifier, and manifest changes have been reviewed: ```yaml verify: runs-on: ubuntu-latest if: github.event.pull_request.head.repo.full_name == github.repository environment: tessl-verify env: TESSL_TOKEN: ${{ secrets.TESSL_TOKEN }} steps: - name: Fetch base branch env: BASE_REF: ${{ github.event.pull_request.base.ref }} run: git fetch --no-tags origin "$BASE_REF:refs/remotes/origin/$BASE_REF" - name: Inspect verifier scope env: BASE_REF: ${{ github.event.pull_request.base.ref }} run: tessl change verify --base "origin/$BASE_REF" --dry-run --show-files --max-calls 200 - name: Verify changed files env: BASE_REF: ${{ github.event.pull_request.base.ref }} run: tessl change verify --base "origin/$BASE_REF" --github --max-calls 200 ``` `--base "origin/$BASE_REF"` compares against the pull request's actual target branch, which the fetch step records as a remote-tracking ref. For source-only changes, the command verifies the pull request's changed files. When the pull request changes a verifier JSON file or the `verify` configuration, the affected verifier runs across its full configured scope. `--github` emits annotations for findings. A `warn` finding is advisory, while an `error` finding makes the command exit non-zero. `--max-calls 200` is the gate's call budget. The command stops before judging if its planned scope exceeds that limit; narrow the group's scope or choose and document a larger limit before changing the workflow. If you run the command from a monorepo root with linked child manifests, add `--recursive` to each invocation. ## 5. Promote the verifier to a blocking check Leave the verifier at `warn` while it runs on real pull requests. Track false positives, missed applicable files, and unclear findings. When the verifier is reliable, change the group level: ```json { "verify": { "groups": { "error-ux": { "path": "verifiers/actionable-errors.json", "level": "error", "include": ["src/**/*.ts"], "exclude": ["**/*.test.ts"] } } } } ``` The **Tessl Verify** Check, or your own workflow's command, now fails on pull requests with failing findings. You do not need to change any settings or the workflow, or add `--fail-on-warn`. To stop those pull requests from merging, make the check required in the branch's protection rules. ## Next * Read the [`verify` block reference](/reference/configuration.md#verify-block) for group filters and per-verifier overrides. * Add [Tessl Code Review](/tutorials/setting-up-agentic-code-review.md) when you also want general pull request review beyond named repository invariants. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.tessl.io/codifying-and-enforcing-your-skill-standards/verify-agent-changes-in-ci.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.