The five checks, in order
- Is there a test that failed before the change? This is the strongest check there is. If a test fails on the old code and passes on the new code, the change did something real. If no such test exists, you are relying on reading alone.
- Did that test fail for the right reason? A test can fail because of a typo, a missing import or a wrong file path. That proves nothing. The failure message should name the bug you are fixing.
- Does everything else still pass? Run the whole suite, on a clean checkout. A fix that turns one test green and three others red has moved the bug.
- Did it change only what it needed to? Look at the list of files before the lines. A fix for one crash that touches the build config, the lockfile and an unrelated module needs a reason for each.
- Do you understand why it works? Read the diff last. If you cannot say in a sentence why the change fixes the problem, do not merge it yet.
Why a clean environment matters
An AI agent that runs code on your laptop can pass a test because of something on your laptop: a cached file, an environment variable, a service that happens to be running. A check that only passes there is not a check.
Run the tests somewhere that starts empty, such as your CI or a sandbox. Bugstack runs its proof in an isolated sandbox with the network cut and none of your secrets, so a passing test means the code passed and nothing else helped.
Things that look like verification and are not
- The model saying the code works. It has no way to know.
- Tests the model wrote in the same step as the code, and that have only ever passed. A test is evidence once it has failed.
- A second model reviewing the first. Useful for catching obvious mistakes, and it can share the first model’s blind spots.
- A demo. The happy path working once says little about the edges.
How Bugstack does it
Bugstack does autonomous software maintenance: it fixes crashes, failing builds and failed deploys, and opens each fix as a pull request. It runs the checks above on itself before you see anything.
- A tester writes a test that fails on your current code, for the bug’s own reason.
- A fixer makes the change. The test has to pass and the rest of your suite has to stay green, or no pull request opens.
- The pull request carries the test, so the proof travels with the fix and your CI runs on it.
- You merge it, or you set the repository to Autopilot and let proven fixes merge once your proof setting is met.
If a bug cannot be reproduced, the pull request says so. That tells you which changes still need your own check.
Questions
What is the single most useful check?
A test that failed before the change and passes after it. Everything else is weaker evidence. If you can only do one thing, ask for that test.
Should I read every line an AI writes?
Read every line of what merges into code you are responsible for, until you have a track record with the tool. Reading is the slowest check and the easiest to do badly, so let tests and CI narrow down what needs your attention first.
Is passing CI enough?
Passing CI tells you the change broke nothing your tests cover. It does not tell you the change fixed the problem, unless one of those tests was failing because of it.