All pages
Close the loop · Lesson 1.1 · 12 min · Free
Tests are the contract
Make a failing test the definition of done and let the agent run it.
After this lesson you write the definition of done as a failing test before the agent writes any code, have the agent run that test itself, and read failures as structured errors instead of scrolling through raw output.
Time: 12 minutes · Works on: macOS app.
Done means a test passes
When you ask an agent for a change, it tells you when it thinks it's finished. That report is a claim, not evidence. An agent can say "fixed" after an edit that never ran, after a test run it misread, or after it quietly changed the test so the broken code passes.
The fix is to agree on what "done" means before work starts, in a form a machine can check. A test is that form. It is specific, it runs the same way every time, and it doesn't care how confident the agent sounds. Put it in the brief and the rest of the run has a target:
Done when
slugify("Hello, World!")returns"hello-world", the new test inslugify.test.tspasses, and the rest of the suite still passes.
This is test-driven development with an agent doing the typing. The part that matters most is unchanged: the test comes first and you read it before any implementation exists.
Red, then green, in two turns
Split the work into two turns, and review between them.
Turn one, red. Ask the agent to write a test that captures the behaviour you want, run it, and show you that it fails. Tell it not to touch the code under test. A test that passes before the fix is written proves nothing: it either tests the wrong thing or the bug isn't where you think.
Read the test. This is the cheapest review you'll do all day. Ask yourself:
- Does it check the behaviour you described, or a detail of one possible implementation?
- Would it fail for the reason you expect? The failure message should name the bug, not a typo or a missing import.
- Does it cover the edge you care about? For the slug example, that might be repeated punctuation, leading spaces or non-ASCII letters.
Turn two, green. Now ask for the smallest change that makes the test pass, and have the agent run the test again, then the whole suite. The full run matters. An agent that only reruns the one test it was given can break three others and still report success.
Rules the agent must follow
Write these into the brief, or into your project instructions so every run gets them:
- Don't edit the test to make it pass. If the agent thinks the test is wrong, it stops and says why. Changing an assertion is a decision for you.
- Run the tests yourself. "This should pass" is not a result. The reply quotes the run's actual summary line.
- Run the full suite before reporting done. A filtered run is for the loop, not for the finish.
- Report a failure as a failure. If the suite has failures that existed before the change, the agent names them and says they were already there.
Rule one is the one agents break most, and usually with good intentions. A test that fails because of a timezone looks like a bad test to a model trying to finish. Sometimes it is. You decide, not the agent.
When there is no test yet
Some work is hard to test first: a layout change, a log message, a
configuration file. Don't skip the contract; change its form. "The build
passes and npm run lint prints no warnings" is a check the agent can run.
For anything a user sees, the next lesson, Verify in the
browser, turns "looks right" into assertions.
A worked example
A bug report says that slugify leaves a trailing hyphen on titles that end
in punctuation. Your first message to the agent:
slugify("Done!") returns "done-" but should return "done".
Write a failing test for this in src/slugify.test.ts. Run it and show me
the failure. Don't change src/slugify.ts yet.
The agent writes the test, runs it, and quotes the failure: expected "done",
received "done-". You read the test and add one request: also cover
"Done!!!". It adds the case and both fail.
Second message:
Make the smallest change to src/slugify.ts that makes both cases pass.
Then run the full test suite and tell me the result.
Don't edit the test file.
The agent changes one regular expression, reruns, and reports the suite's summary line with every test passing. You have a regression test, a fix, and proof, and you read about twenty lines to get them.
Do it in TRMNLY
- Brief the red turn. Open the right sidebar with
⇧⌘\and choose Agent. Send a message like the first one above, naming the behaviour, the test file, and "don't change the code yet". - Narrow the agent for the red turn. In the agent's gear menu, under
Persona, choose Tester. It's a testing role with a narrowed tool
set that includes
run_tests, and it is told to keep its edits to test files. That instruction lives in its prompt, not in a path check, so still read the diff. Switch back to Default for the green turn. - Let it run
run_tests. The agent'srun_teststool finds your test runner by itself:swift testfor aPackage.swift,xcodebuild testfor an Xcode project,npm testfor apackage.json,pytest, orbundle exec rake testfor aGemfile. It takes afilterto run one test, and a timeout that defaults to 300 seconds. Output is capped at 256 KB. Under the default Approve posture,run_testswaits for your approval each time. Once you trust the loop, open Always… on the approval card and allow the tool for this project. - Use a pane when the runner is custom. If your tests run with
something like
make test,run_testscan't detect it. Ask the agent to run the command in your terminal instead, where you can watch it. - Read failures as structured errors. When a command fails in a pane,
TRMNLY parses its output. Ask the agent to call
get_command_errors. It returns each recent failure's exit code, working directory and parsed errors, with kind, message, andfile:linewhen the toolchain prints one, so the agent goes straight to the right line instead of re-reading scrollback. Parsing only happens on a failed command, so a passinggrep errorisn't mistaken for a broken build. - Use the error pill. After a failed command, a pill appears at the bottom of the pane with Explain and Fix. Either one hands the error to the agent. To skip the click on real build failures, turn on Auto-ask agent on build failures under Automation in the gear menu.
- Spot flaky tests. TRMNLY keeps a history of every command it has
seen finish. It flags a command that fails for the first time ever, runs
sharply slower than usual, or prints a warning it has never printed. Ask
the agent to call
command_statsfor your test command to see its typical duration and failure rate. A test command with a steady trickle of failures is a flaky suite, and a flaky suite is a contract the agent can't keep. - Check who ran what. Press
⇧⌘Pand run Toggle Command Provenance. The panel lists the pane's recent commands, who produced each one (you, the agent, the queue, a runbook or a replay), and its exit code. It answers "did the agent actually run the suite, or only say it did?" - Confirm the test wasn't weakened. Before you commit, open More ▾ → Review and read the test file's diff. If an assertion changed, that was the agent breaking rule one. Review the diff covers this step in full.
Check yourself
- You've seen the new test fail, for the reason you expected, before any implementation was written.
- The agent's final reply quotes the full suite's real result, not a prediction.
- You can say which runner
run_testspicked for your project, or why it couldn't pick one. get_command_errorsgave you afile:linefor a failure in a pane.- The test file's diff shows no weakened assertions.
Go further
- Terminal intel — command provenance, anomaly flags and every introspection tool
- Write a task brief — put the done-when line in every brief
- Your first task — approval cards and Always… rules
- Watch the terminal — catch failures in long-running commands