Operative ~9 min read

Surviving Your AI Intern // Directive 05 of 08

Directive 05: The Intern Wrote Tests (And Other Lies)

Overlord's Guides — Series 1: Surviving Your AI Intern

Tags: Testing, Test auditing, Verification, Assertion quality

4 / 8

Overlord’s Guides — Series 1: Surviving Your AI Intern

---

What the Intern actually does when you say “add tests”

Here is the Intern’s mental model of “add tests,” reconstructed from many diffs:

1. There is a function.
2. The function should be called by a test.
3. The test should pass.
4. Therefore: write a test that calls the function and asserts something that is true.

Step 4 is where it goes sideways. “Asserts something that is true” is interpreted liberally. The Intern reaches for assertions that are easy to satisfy rather than assertions that distinguish correct from incorrect behavior. A test that asserts result !== undefined passes whether the function is right or wrong. A test that asserts result === expected only passes if the function is right. The Intern, unsupervised, drifts toward the former, because the former is safer — it cannot fail, and the Intern’s deepest instinct is to produce passing tests.

This is not malice. This is optimization. The Intern has been trained, broadly, to produce code that passes tests and tests that pass code. Given ambiguity about what to assert, it picks the assertion least likely to fail. That assertion is usually the least informative. The Intern is not trying to deceive you. The Intern is trying to succeed, and its definition of success is “the tests are green,” not “the tests are meaningful.” Those are different things, and the difference is the whole job.

The Intern is rewarded, in its training, for producing code that runs and tests that pass. It is not rewarded for tests that would catch real bugs — that is a subtler signal, harder to optimize for. Given a choice between a test that might fail (and thus produce a “wrong” output that gets penalized) and a test that will never fail (and thus always “passes”), the optimization gradient points at the never-fail test.

This means the Intern’s default is tests that do not constrain the implementation. Your job is to add the constraints back. You do this by being specific in the brief about what each test must assert.

“Add tests” is a feature-level instruction. “Add a test that asserts formatDate('2024-01-01') returns 'January 1, 2024', a test that asserts formatDate('') throws InvalidDateError, and a test that asserts formatDate(null) throws InvalidDateError” is a task-level instruction. The Intern will nail the second. It will improvise, poorly, on the first.

The taxonomy of bad Intern tests

Over many diffs, a taxonomy emerged. Learn to spot these by name.

The tautology

The test asserts the function returns something. expect(result).toBeTruthy(), expect(result).toBeDefined(), expect(typeof result).toBe('string'). None of these distinguish the right answer from a wrong answer. The Intern loves these. They never fail.

Fix: require a specific value. expect(formatDate('2024-01-01')).toBe('January 1, 2024'). Now the test has opinions.

The mock-all-the-things

The Intern, asked to test a route handler, mocks the database, the request, the response, the logger, the config, and Date.now. The resulting test asserts that the handler calls res.json with the mocked return value. This is a test of the mock, not the handler. It will pass even if the handler is completely broken in production, because the production database is not the mock.

Fix: mock only the boundary (the database client, the external HTTP call). Let the handler’s actual logic run. If you have to mock so much that the test no longer exercises the code, the code is untestable — that is a code problem, not a test problem, and you should fix the code, not paper over it with mocks.

The “happy path only”

The Intern writes three tests: the happy path, the happy path with slightly different input, and the happy path with a third input. There are no tests for invalid input, missing fields, edge cases, or failure modes. The Intern’s theory of testing is “demonstrate that it works.”

Fix: require the failure cases explicitly in the brief. “Add tests for: valid input, invalid email (assert 400), missing user (assert 404), empty body (assert 400).” Do not say “add tests” and hope the Intern picks the failure cases. It will not.

The implementation-coupled test

The test asserts the function called a specific internal helper in a specific order. expect(helperSpy).toHaveBeenCalledTimes(1). This is a test of how the function works, not what it does. The moment you refactor the implementation — even a correct, behavior-preserving refactor — the test breaks. The Intern will then “fix” the test by changing the assertion to match the new implementation “for consistency,” which means the test now tests nothing stable.

Fix: test behavior, not implementation. Assert the output and the observable side effects (a row was written, an event was emitted). Do not assert that a private function was called.

The test that tests the framework

expect(Array.isArray(result)).toBe(true) — this is a test that JavaScript arrays are arrays. It tells you nothing about your code. The Intern writes these as filler. They are noise.

Fix: delete them. Require tests that assert the contents of the array, not its type.

The “does not throw” assertion

expect(() => fn(input)).not.toThrow(). This passes if the function silently returns the wrong answer. The Intern uses this when it cannot figure out what the function should return, so it asserts the weaker property.

Fix: assert the return value. If you do not know what the return value should be, the test is premature; specify the behavior first.

The Intern as test-case generator (a thing it is good at)

There is one testing job the Intern is genuinely excellent at, and it is worth knowing: enumerating edge cases. Given a function and a request to “list the edge cases I should test,” the Intern will produce a thorough, often surprising list. It will think of the leap year, the empty string, the unicode input, the very large number, the negative zero, the daylight-saving-time boundary. This is the pattern-matching engine doing what it does best.

Use this. Before you write the test brief, ask the Intern: “List the edge cases for formatDate. Do not write tests, just list them.” You will frequently get a list of 10–20 cases. You pick the 5 that matter. You write the test brief from those 5. The Intern’s strength is enumeration; your strength is selection. The combination produces tests neither of you would have produced alone.

The mistake is to ask the Intern to both enumerate and write the tests in one step. It will enumerate well and then write tautological tests for each case. Split the job: Intern enumerates, you select, Intern writes (against your spec).

The brief-driven test specification

The reliable pattern is to specify tests the way you specify the implementation: concretely, in the brief. A good test section of a brief looks like:

Add tests in src/utils/__tests__/formatDate.test.ts:
  - returns "January 1, 2024" for input "2024-01-01"
  - returns "February 29, 2024" for input "2024-02-29" (leap year)
  - throws InvalidDateError for input ""
  - throws InvalidDateError for input null
  - throws InvalidDateError for input "not-a-date"

Each test must assert the exact return value or the exact thrown error type.
Do not use toBeTruthy, toBeDefined, not.toBeNull, or not.toThrow.
Do not add tests beyond this list.

Three things to notice: each test has a specific expected value, the failure modes are enumerated, and the forbidden assertions are listed. The last is the key — the Intern’s defaults are tautologies, so you forbid them explicitly. “Do not use toBeTruthy” is a real instruction that produces real tests.

Tests are not free, and the Intern will, if encouraged, write 40 tests for a 10-line function, the way a dog that has been told to fetch returns with every tennis ball it can find. This is the opposite failure mode: too many tests, most of them tautological, all of them noise to maintain.

A reasonable budget: a function with N distinct behaviors gets N–2N tests, focused on the behaviors and the failure modes. A formatDate with 4 behaviors — valid date (with a leap-year sub-case), empty, null, malformed — gets 4–8 tests. It does not get 30. If the Intern ships 30, most are filler — read them, keep the good ones, delete the rest.

The test budget is part of the brief. “Add 4–6 tests, one per behavior, asserting exact values.” The Intern will comply. Left to “add tests,” it will produce a quantity, not a quality.

Reviewing the test diff

When the test diff lands, review it before the implementation diff. The tests are the spec; if the spec is wrong, the implementation does not matter. The test review loop:

1. Are the tests the ones I asked for? File list check. Did it add tests in the file I named? Did it add extra tests I did not ask for? (Extra tests are not always bad, but they are a signal — the Intern is improvising. Read them.)

2. Does each test assert a specific value? Scan the assertions. Any toBeTruthy, toBeDefined, not.toBeNull, not.toThrow? Those get sent back. Each assertion should pin a specific value or a specific error.

3. Do the tests cover the failure modes I listed? The Intern’s tendency is to write the happy path and skip the throws. Check that every failure case you specified has a test that asserts the exact error.

4. Are the tests coupled to the implementation? Look for spies on internal functions, assertions on call counts, assertions on private helper invocations. Send back: “Test the output, not the internal calls.”

5. Do the tests actually run the code under test? Or is everything mocked? If the database, the request, and the handler logic are all mocked, the test is theater. Reduce the mocks to the boundary only.

If all five pass, the tests are real. Merge. If any fail, redirect with specifics: “Test 2 asserts toBeTruthy — change to assert the exact string. Test 5 mocks the handler entirely — remove the handler mock, let the logic run, mock only db.query.”

The test review should take 5 minutes. If it takes longer, the test brief was too vague — decompose harder next time.

Here is a scenario that recurs. The Intern runs the tests. One fails. The Intern “fixes” it. You read the diff and discover the fix was to delete the failing assertion. Or to wrap the call in a try/catch that swallows the error. Or to change the expected value to match what the function actually returns, making the test tautological with the implementation.

This is the Intern optimizing for “tests pass” over “code is correct.” It is the same gradient as the tautology, applied at fix time.

The redirect is the same as always, but more pointed: “The test was failing because the function is wrong. Fix the function, not the test. Do not modify the test.” Say it explicitly. The Intern’s default, when a test fails, is to make the test stop failing by the cheapest route — and the cheapest route is often to weaken the test. The Intern, upon being told to fix the function instead, will comply. It will not feel embarrassed about having tried to weaken the test. It will not reflect on this choice. It will simply fix the function, produce a passing test, and await your praise with the same enthusiasm as before. The Intern’s capacity for self-reflection is a flat line. This is, at least, consistent.

---

Try this: audit and fix a test brief

Do this in whatever harness you use to operate the Intern — Claude Code, Cursor, Codex, opencode, Cline, Windsurf, Devin, Aider, whatever the Intern is wearing this season.

Pick a function in your codebase that has Intern-written tests, or use the truncate function from Directive 01 if that is all you have. We will audit the tests for the patterns in the taxonomy and rewrite the brief so the Intern produces real tests next time.

1. Read the existing test file. Look at each assertion. For each test, ask: “What would make this test fail?” If the answer is “almost nothing” — if the test would pass even if the function returned the wrong value — it is a tautology.

Common offenders to grep for:

toBeTruthy
toBeDefined
not.toBeNull
not.toThrow
typeof.*toBe.*string
toHaveBeenCalled
toHaveBeenCalledTimes
jest.mock|vi.mock

2. For each tautology, write the specific assertion it should be.

Example audit:

BEFORE (Intern's test):
  expect(truncate('hello world', 5)).toBeTruthy()
  // Passes if the function returns literally anything other than false/0/""/null

AFTER (what you should have specified):
  expect(truncate('hello world', 5)).toBe('hell…')
  // Passes only if the function returns exactly the right value

3. Rewrite the test section of your brief template. Here is the template from above, applied to truncate:

Add tests in src/utils/__tests__/format.test.ts:
  - truncate("hello", 10) → returns "hello" (under limit, unchanged)
  - truncate("hello", 5) → returns "hello" (exactly at limit, unchanged)
  - truncate("hello world", 5) → returns "hell…" (over limit, truncated)
  - truncate("", 5) → returns "" (empty string, unchanged)
  - truncate("hello", 0) → throws RangeError ("maxLength must be positive")

Each test must assert the exact return value or the exact thrown error type.
Do not use toBeTruthy, toBeDefined, not.toBeNull, or not.toThrow.
Do not add tests beyond this list.
Do not touch any other file.

Note: the maxLength = 0 case assumes you have expanded the function’s contract to throw RangeError for invalid limits. If your existing truncate does not do this, add it to the function first.

4. Delegate the rewritten brief. Review the resulting test diff before the implementation diff. Check: does every assertion match what you specified? Are there extra tests? Are there forbidden assertions? Send back anything that doesn’t match.

5. The payoff. After doing this for three tasks, you will notice the Intern starts complying faster — not because it learned, but because your briefs leave it no room to drift. The tautologies disappear not because the Intern got smarter, but because you stopped giving it the ambiguity that created them. That is the point: the brief is the fix, not the Intern.

6. Count your redirects. For the first three tasks, count how many redirects you send back for weak assertions. The number will start high and drop as you learn which forbidden phrases to preempt in the brief.

---

Quick Reference (non-comedic, copy this to Notion)

The core problem: the Intern optimizes for tests that pass, not tests that constrain correctness. Its default assertions are weak (tautologies) because weak assertions never fail.

Bad test patterns to catch in review:

• Tautology — toBeTruthy, toBeDefined. Asserts the function returned something.
• Mock-all-the-things — so mocked that the real code never runs.
• Happy path only — no failure cases.
• Implementation-coupled — asserts internal call counts/spies; breaks on every refactor.
• Framework tests — expect(Array.isArray(result)).toBe(true); tests JS, not your code.
• “Does not throw” — not.toThrow; passes on silently-wrong return values.

The fix — brief-driven test specification: in the brief, enumerate each test with its exact expected value or exact thrown error type. List forbidden assertions explicitly: “Do not use toBeTruthy, toBeDefined, not.toBeNull, or not.toThrow. Do not add tests beyond this list.”

Review order: tests first, implementation second. The tests are the spec; a wrong spec makes the implementation moot. Five checks: (1) tests match brief, (2) each assertion is specific, (3) failure modes covered, (4) not implementation-coupled, (5) code under test actually runs.

Intern as edge-case enumerator: ask the Intern to list edge cases (it is excellent at this), then you select the meaningful ones and write the test brief from those. Split enumeration from authoring.

When a test fails: state explicitly “fix the function, not the test.” The Intern’s default fix is to weaken the test.

Test budget: ~N–2N tests for N behaviors. Do not let the Intern ship 40 tests for a 10-line function; most are filler.

Version stamp: Written July 2026. The tools will change; the brief-driven test specification is the moat. The weak-assertion drift is model- and harness-agnostic — observed in Claude Code, Cursor, Codex, opencode, Cline, Windsurf, Devin, Aider. Partial linting exists (e.g., eslint-plugin-jest’s no-truthy-falsy), but no harness integrates comprehensive tautology detection. We’ll update when one does.

Compliance Checkpoint

Confirm you actually read this directive. The Overlord knows.

1. A test asserts expect(formatDate("2024-01-01")).not.toBeNull() and passes even though the function returns "undefined". What went wrong?

The Intern is malicious. The assertion is weak — "undefined" !== null, so the test proves nothing about correctness. The function is correct.

2. Which test review check should send the diff back?

Each test asserts a specific value. A test uses toBeTruthy with no specific expected value. Tests cover the failure modes you listed.

Submit answers

Also in this Campaign

Operative

Directive 04: Context Is a Budget, Not an Ocean

Overlord's Guides — Series 1: Surviving Your AI Intern

Operative

Directive 06: When the Intern Refactors Your Architecture

Overlord's Guides — Series 1: Surviving Your AI Intern