Regression Testing for LLM Applications (2026)

When to Use

• ✓Before deploying any prompt change, even small wording tweaks that seem innocuous
• ✓When upgrading to a new model version (gpt-4o-mini → gpt-4o-mini-2026) to verify no degradation
• ✓After changing the retrieval strategy in a RAG system
• ✓When a user reports a quality issue — add their case to the regression suite and verify the fix doesn't break other cases
• ✓Before rolling out a new feature that touches the LLM pipeline

How It Works

1. 1Build a fixed regression suite: cases from historical failures, edge cases you've discovered, and representative positive examples. This suite never changes (only grows). Version-control it with your code.
2. 2Run tests against a baseline (current production) and the change candidate simultaneously. Compare scores statistically rather than by absolute values.
3. 3Use appropriate scoring for the task: exact match for structured outputs, LLM-as-judge for open-ended, schema validation for JSON, functional tests for code. Choose the most deterministic scoring method available.
4. 4Set a regression threshold: block deployment if candidate scores drop more than 5% below baseline on any dimension, or if the drop is statistically significant (p < 0.05 with N=5 runs per test case).
5. 5Categorize tests by severity: P0 (must not regress — core functionality), P1 (should not regress — important features), P2 (track but allow regression with explanation). Block deploys only on P0 failures.

Examples

# promptfoo config: promptfooconfig.yaml
prompts:
  - 'prompts/v1/system.txt'
  - 'prompts/v2/system.txt'  # candidate change

models:
  - anthropic:claude-3-5-haiku-20241022

tests:
  - description: 'Extracts invoice number correctly'
    vars:
      document: '{{invoice_text}}'
    assert:
      - type: javascript
        value: 'output.includes("INV-2024")'
  - description: 'Handles missing date field gracefully'
    vars:
      document: '{{invoice_missing_date}}'
    assert:
      - type: llm-rubric
        value: 'Response acknowledges the missing date and does not hallucinate one'

# Run: promptfoo eval --no-cache
# Compare v1 vs v2 scores side by side

from scipy import stats
import numpy as np

def detect_regression(baseline_scores: list[float], candidate_scores: list[float], threshold: float = 0.05) -> dict:
    '''
    Returns whether candidate is a statistically significant regression from baseline.
    '''
    baseline_mean = np.mean(baseline_scores)
    candidate_mean = np.mean(candidate_scores)
    
    # Welch's t-test (handles unequal variance)
    t_stat, p_value = stats.ttest_ind(baseline_scores, candidate_scores)
    
    pct_change = (candidate_mean - baseline_mean) / baseline_mean
    
    is_regression = (
        p_value < 0.05 and  # Statistically significant
        pct_change < -threshold  # Meaningfully worse (>5% drop)
    )
    
    return {
        'is_regression': is_regression,
        'baseline_mean': baseline_mean,
        'candidate_mean': candidate_mean,
        'pct_change': pct_change,
        'p_value': p_value,
        'significant': p_value < 0.05
    }

Common Mistakes

• ✗Using exact string matching for open-ended outputs — 'The answer is 42' and '42' are the same answer. Use semantic equivalence checking (LLM judge or embedding similarity) rather than exact match for natural language outputs.
• ✗Not running enough trials per test case — LLM outputs vary between runs. Running each case once gives unreliable scores. Run each case 3–5 times and compare means, especially for cases near the pass/fail threshold.
• ✗Regression tests that are too sensitive — if 50% of your regression tests fail every deploy due to trivial rewording, the team stops trusting them. Calibrate test sensitivity: important behaviors should fail hard; stylistic preferences should have wide tolerance.
• ✗Not adding production failures to the regression suite — every time a user reports a quality issue, add that case to the regression suite before fixing it. This ensures the fix works and prevents recurrence.

FAQ