QA Testing Agent

Sonnet 4 produces tests that match existing project style better than GPT-4o and generates more meaningful edge cases. DeepSeek R1 is a strong cheap alternative but more verbose.

Alternatives: GPT-4o, DeepSeek R1

Root-cause analysis on a failed test is a reasoning task. Opus 4 with extended thinking finds real bugs that Sonnet misses, worth the cost for the ~20% of runs that fail.

Alternatives: Claude Sonnet 4, GPT-4o reasoning

Ephemeral Firecracker microVMs start in <1s and isolate untrusted AI-generated code. Critical if the agent is writing tests that exec, hit the network, or touch the filesystem.

Alternatives: Daytona, E2B, Modal, Docker-in-Docker on K8s

Tests need related source context. Embedding search returns the caller, the types, and similar existing tests — far better than file-path heuristics.

Alternatives: pgvector, Sourcegraph

AI-generated tests will be flaky — it's a feature of LLM variance. Auto-detect and quarantine flakes or you'll train the team to ignore the bot.

Alternatives: CircleCI, Custom SQL on test history

Coverage delta is the key signal for 'did this test add value'. Use the language-native tool and parse the output — avoid coverage-as-a-service latency.

Alternatives: Codecov, Coveralls

This is CI-shaped work. GitHub Actions gives you free compute for open-source and matches the existing workflow. LangGraph overcomplicates a linear DAG.

Alternatives: Temporal, LangGraph

Generate-then-run vs iterate-until-passes

Letting the agent loop 'write test, run, fix until passes' is tempting but usually produces tautological tests that just assert whatever the code currently does. Better: generate tests with an explicit oracle (requirements doc, spec, or behavioral description), run once, and diagnose real failures — don't let the agent rewrite the test to make it pass.

Sonnet 4 vs Opus 4 for generation

Sonnet 4 is fine for standard unit tests on well-specified code. Opus 4 is worth the extra cost for complex state machines, concurrency, and security-sensitive paths. Route by file tag — mark auth, payment, and state-machine files as 'opus-tier' in a config.

Sandbox per test run vs shared environment

Per-run sandbox is isolated and repeatable but adds 5–15s cold start. Shared environment is fast but risks state leakage between runs. Use per-run for PR checks, shared (with reset-between) for fast local dev loops.

Agent writes tautological tests (assert that code does what it does)

Mitigation: Require tests to reference an external oracle — the PR description, a requirements doc, or a previous stable test. Reject tests whose assertions only restate the code under test. Run an LLM-as-judge critique pass that scores test 'meaningfulness'.

Flaky tests blocked PRs and team loses trust

Mitigation: Every failing test runs 3 times before being reported as failed. Tests that pass ≥1 and fail ≥1 out of 3 get quarantined automatically. Track flake rate per test file and alert when it exceeds 2%.

AI-generated test code runs arbitrary commands or exfiltrates data

Mitigation: Sandbox every run in an ephemeral microVM with no network access except to npm/PyPI mirrors. No env vars from the main project. Static-analyze generated test code for network calls, subprocess spawns, and filesystem writes outside /tmp before executing.

Tests hit real external services (databases, APIs)

Mitigation: The sandbox has no network access by default. Generated tests must use project's mocking fixtures; reject tests that import `requests`, `fetch`, or DB clients without a mock wrapper. Maintain an allowlist of mock-friendly modules.

Agent generates 50 low-value tests and clutters the suite

Mitigation: Enforce per-file test budget (max 8 tests per file per run). Planner prioritizes by estimated defect-finding value using a rubric (edge case > error path > happy path). Deduplicate against existing tests before writing — embed existing tests and check cosine similarity > 0.9.