Your Tests Passed. That Does Not Mean Your Pull Request Is Safe.

Every pull request ends the same way. A row of green ticks, a merge button, a small hit of confidence. The tests passed, so the change must be fine.

Except that the green tick answers a much narrower question than most people assume. It tells you the suite ran and nothing blew up. It does not tell you whether the lines you just changed were actually exercised, whether the assertions checked anything real, or whether the one risky file in the diff had a single test pointed at it.

That gap is the whole problem. A passing suite and a safe change are not the same thing, and treating them as the same is how regressions ship with a clean board behind them.

The two signals everyone trusts both miss the point

When someone asks, "Are we testing enough?" the answer is usually a coverage number. 80%, 90%, a badge in the README. The trouble is that coverage is measured across the entire codebase, while a pull request only touches a handful of files. A repo can sit at 90% overall, and your specific change can be the uncovered 10%.

So the two signals every team already trusts, pass or fail and a global coverage figure, both quietly skip the question that actually matters on a pull request. Did the code that changed get tested, and did those tests prove anything?

That is the question I wanted a tool to answer on every merge, automatically, with evidence attached.

A gate that scores the change, not the repo

So I built a small GitHub Action. It is not a linter, and it is not another coverage badge. It reads the test results and coverage from your CI run, identifies which production files the pull request changed, and then looks for the test evidence specific to each of those files.

From that, it produces a verdict with three states.

Green means the evidence supports the change. Red means a blocking rule failed, usually a real test failure or coverage on the changed lines dropping below your threshold. The interesting one is orange.

Orange indicates the evidence is incomplete and tells you exactly where. Not "something failed" but "this file changed, and nothing in your test results touches it." Orange is the status reviewers actually learn from, because it points a finger at the precise spot where confidence runs out. Most gates only know how to pass or block. The gap in between is where the useful information lives.

A little under the hood

The plumbing is deliberately boring, which is the point.

You run your tests first and produce a JUnit XML report, a format that almost every test runner across every language already emits. Coverage arrives as LCOV, which Jest, pytest, Istanbul and most others write out the moment you ask for it. Then the gate step runs and maps each changed production file to the tests and coverage that reference it.

- uses: gilav115/pr-test-quality-gate@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}

Out of the box, failures turn it red, and everything else is informational, so you can drop it in without a config file and learn from what it reports. A short config, then, lets you raise the bar where it counts. Demand a minimum coverage on changed lines. Mark critical paths, such as authentication or payments, to apply a stricter standard. Ignore the noise like docs and markdown. The output is a plain status plus a full report in Markdown, JSON and standalone HTML, so it reads the same whether a human or another job is consuming it.

There is an optional layer of AI reviewers on top. One classifies each changed file by risk category. One flags where the evidence looks thin against that risk. One reads the production code and the test code together and catches things coverage cannot see, like a mock so shallow it proves nothing, or an assertion that checks an implementation detail rather than behaviour.

And here is the design decision I care about most.

The trust model: the machine flags, the rules decide

The AI reviewers can raise concerns, but they can never set the final verdict on their own. Every finding must cite the exact file and line. Anything vague gets stripped before the verdict is computed. The worst an AI finding can do is push the status to orange. A deterministic engine, plain rules you can read and predict, always makes the actual call.

That boundary is not a limitation I tolerate. It is the feature. A gate that blocks your team needs to be explainable, repeatable and impossible to spook with a hallucination. So the model gets to point at problems, and the rules get to decide. If you build anything that puts a language model in a position of authority, this is the pattern worth copying.

The tool raises the floor, not the ceiling

A gate like this catches the obvious misses. The file nobody tested. The assertion that checks nothing. The risky path that slipped in with no coverage. That is real value because humans are genuinely bad at noticing the absence of something.

But a quality gate can quietly turn into a crutch. The moment you hand the question "Is this change safe?" entirely to a badge, you stop building the instinct that lets you answer it yourself. The skill of opening a diff and feeling, in your gut, what could break and what you would want to prove before merging, that skill is still yours to grow, and no scorer hands it to you.

So treat the relationship simply. Let the automation make your judgment faster and more consistent. Do not let it make your judgment for you. The engineers who stay valuable through the next decade are the ones who use tools like this to clear the routine work, then spend the freed attention getting sharper at the part the machine cannot do, which is understanding what a change actually risks.

A green board should be where your thinking starts, not where it stops.

Try it, then go deeper

The action is open source. The full rule reference, config options and workflow examples live in the repo at github.com/gilav115/pr-test-quality-gate, and it takes one step in a workflow to start seeing orange where you used to see a misleading green.

If the wider problem interests you, validating systems you cannot fully see inside, including AI ones, that is the work I do at squarenumbers.co.uk.

Ship fast. Ship with confidence. Just make sure the confidence is earned.

This article also appears on Medium