Test Case Generation

Sonnet 4 produces the most thorough and least tautological tests in 2026 benchmarks when given an explicit edge-case checklist. Haiku 4 is 5x cheaper and adequate for simple utility functions. DeepSeek R1 shines when the function has subtle invariants that require step-by-step reasoning about edge cases.

Alternatives: GPT-4o, DeepSeek R1 for edge-case reasoning, Claude Haiku 4 for simple functions

Coverage alone is a weak signal — a test can execute every line without catching any bug. Mutation testing is the ground truth for whether tests catch regressions. Expect 10-30 minutes added per module; run it in CI for generated tests only, not the full suite.

Alternatives: PIT for Java/JVM, Skip for prototypes, Lightweight manual mutation checks

Generated tests can contain infinite loops, resource leaks, or network calls to anywhere. Firecracker-based sandboxes isolate each test run. GitHub Actions works but is slow (90-180s overhead per run); microVM sandboxes run in under 2 seconds.

Alternatives: GitHub Actions matrix, Docker-in-Docker, Local subprocess with ulimit

LLMs love generating the same test three ways. AST matching on the assertion + setup combination catches exact duplicates cheaply. Embedding similarity on the test body catches semantic duplicates. Both together prevent 95+ percent of test-suite bloat.

Alternatives: Full-text similarity, No dedup (accept bloat), Manual review

Use whatever the project already runs; adding a new coverage tool for test generation is friction. Branch coverage is more useful than line coverage — aim for branch-level targets in the generator prompt.

Alternatives: Codecov integration, Branch coverage via llvm-cov for Rust, Custom coverage aggregator

Mutation-kill rate is the best automated measure of generated test quality. Pair with coverage delta to ensure you are targeting the right code. Sample 5-10 percent of generated tests for human review to catch systematic quality issues the automated metrics miss.

Alternatives: Coverage only (misleading), Braintrust eval harness, Custom benchmark on held-out modules

Coverage targets vs business-logic targets

Generating tests to hit coverage targets is fast but produces lots of tautological tests on untested code that was probably dead anyway. Targeting business-critical modules (billing, auth, permissions) catches real bugs but requires human judgment to pick modules. A hybrid — coverage for the long tail, human-prioritized for hot paths — beats either alone.

Example-based vs property-based tests

Example-based tests are easy to read and debug; property-based tests cover input space much more thoroughly but fail in harder-to-interpret ways. For pure functions with clean inputs, generate both. For UI components, side-effect-heavy code, or tests with complex setup, stick to example-based.

Mutation testing cost vs coverage confidence

Mutation testing adds 10-30 minutes per module, which is painful in an interactive workflow. Running it only on generated tests (not the whole suite) and only on merge (not on every save) keeps cost bounded. Skipping it entirely means you cannot tell real tests from tautologies — pay the cost.

Generated tests are tautological (assert the code does what it does)

Mitigation: Run mutation testing on every merge. Any test that passes under all mutations is flagged and rejected. Track mutation-kill rate per module; a sudden drop usually means a regression in the generation prompt or model version.

Tests depend on implementation details, break on refactors

Mitigation: Instruct the generator to test observable behavior through the public API, not internal state. After generation, scan tests for references to private methods or internal module structure and downgrade-score or reject those. Pair with a style guide snippet in the system prompt showing good and bad examples.

Test-suite bloat from duplicate or near-duplicate tests

Mitigation: Run AST-matching deduplication against existing tests before committing. Use embedding similarity (Voyage-3) with a 0.85 threshold as a second pass for semantic duplicates. Track test-suite growth rate; an explosion in test count without a matching mutation-kill improvement is a red flag.

Flaky tests from non-deterministic inputs (timestamps, random, network)

Mitigation: Prompt explicitly against Date.now, Math.random, and network calls; require test fixtures for dates and seeded randomness. Post-generation, scan for these patterns with regex and retry if found. Add a CI job that runs new tests 20 times; any test failing 1+ run out of 20 is flaky and rejected.

Tests passing locally but failing in CI due to environment drift

Mitigation: Generate and run tests in the same sandbox image used by CI. Pin test framework versions and node/python/go versions via the sandbox config. Any test that passes in the sandbox but fails in CI is an environment-drift bug in the sandbox config — track and alert on the gap.