Test health monitor

Catch flaky tests before everyone learns to ignore them. A scheduled test health monitor finds flaky or repeatedly failing tests and files a ticket for each one.

Goal: Catch flaky tests before everyone learns to re-run and ignore them.

What it does

Daily, a schedule checks for flaky or repeatedly failing tests. It files a ticket for each test that either failed and then passed on a re-run with no code change (flaky), or failed more than once in the window with no code change to explain it. This catches test rot early, before a flaky test becomes something everyone just re-runs and ignores.

The plugin

tessl/test-health reads recent CI history, finds the unstable tests, and files a ticket for each one worth fixing. Install it with:

$ tessl install tessl/test-health

Define the schedule

Run it every day at 6:00 UTC. Add this entry to the schedules block in your project's tessl.json, and keep any other entries already in that block. Replace production with the name of your environment (see the Schedules overview for the prerequisites).

1

Add the schedule entry

{
  "schedules": {
    "test-health": {
      "skill": "tessl/test-health",
      "schedule": "daily at 06:00",
      "environment": "production"
    }
  }
}

2

Validate and push

Validation always checks the local file. For an authenticated linked project that can reach Tessl, it also gives an advisory warning when current workspace capacity may not accommodate additional schedules. Commit and push the change, then confirm Tessl applied the schedule in your workspace before checking its run history.

$ tessl schedule validate
$ git add tessl.json
$ git commit -m "chore(docs): add test health schedule"
$ git push

3

Check the runs

Open the schedule in the web app, or run tessl schedule runs <id> to see its run history.

Customize with instructions

Set the bar for what counts by adding instructions to the same schedule entry. Keep all other entries in the schedules block:

{
  "schedules": {
    "test-health": {
      "skill": "tessl/test-health",
      "schedule": "daily at 06:00",
      "environment": "production",
      "instructions": "Only file a ticket if a test failed 3+ times this week with no related code change"
    }
  }
}

Create your own

Want a different behavior? Create your own plugin that reads test results over time and flags the unstable ones your way.