Code Review Agent

Sonnet 4 has the best tool-use reliability for fetching repo context, and its false-positive rate on code review evals is ~30% lower than GPT-4o in 2026 benchmarks.

Alternatives: GPT-4o, Gemini 2.5 Pro

Reserved for security-sensitive files. Opus catches subtle logic bugs Sonnet misses but costs 5x more — only invoke on risk signals.

Alternatives: GPT-4o reasoning, Gemini 2.5 Pro Thinking

Filters out trivial diffs before spending on Sonnet. Also used for the comment filter pass.

Alternatives: GPT-4o-mini, Gemini 2.0 Flash

Voyage-code-3 is tuned for code and outperforms general-purpose embeddings by ~15% on code retrieval evals.

Alternatives: OpenAI text-embedding-3-large, Jina Code v2

Self-hostable next to your code, supports filtering by path/language, and cheap at repo scale.

Alternatives: pgvector, Turbopuffer

GitHub Apps get proper review API access (line comments, suggestions, request-changes) without a PAT.

Alternatives: Probot, Danger.js

Essential — run every prompt change against a golden set of historical PRs with known bugs before shipping.

Alternatives: Langfuse, Self-hosted + SWE-bench

Coverage vs noise

Reviewing every line catches more bugs but drowns the signal. The better pattern is to only comment when confidence is high and severity is ≥ warning — shipping 3 real findings beats 30 nits every time. Tune the severity threshold against your team's tolerance.

Per-file vs whole-PR prompts

Per-file prompts are cheaper and parallelizable but miss cross-file bugs (a caller passing the wrong type). A hybrid works best: per-file Sonnet pass, then a whole-PR synthesis prompt only when files touch the same module.

Agentic retrieval vs static context

Letting the model call a 'read file' tool mid-review catches cross-file issues but adds 2–5s and sometimes loops. Pre-fetching related files via embedding retrieval is faster and deterministic, at the cost of occasionally missing obscure references.

False-positive nitpicks train devs to ignore the bot

Mitigation: Ship with a strict severity floor (only 'bug' and 'security', never 'style'). Track comment acceptance rate per rule in Braintrust — auto-disable any rule under 40% acceptance after 100 impressions.

Agent hallucinates APIs or imports

Mitigation: Ground every code suggestion by requiring the model to cite the exact file and line it read. Reject any suggestion whose cited path is not in the repo (validate against a file manifest before posting).

Repo context retrieval returns wrong files

Mitigation: Rerank top-20 results with Cohere Rerank before passing to the reviewer. Filter retrieval by the PR's changed-language and ±2 directory levels from touched files.

Agent loops re-fetching files on large PRs

Mitigation: Hard cap at 6 tool-call rounds per file and 20 rounds per PR. If hit, fall back to static pre-fetched context and emit a metric — usually a sign the diff is too big and should be split.

Secrets or proprietary code leak to the LLM provider

Mitigation: Run a secret scanner (gitleaks) on the diff before sending. Route repos marked 'sensitive' through a self-hosted Llama 3.3 70B via Groq or an enterprise-tier AWS Bedrock endpoint with zero-retention.