Output Validation: Ensuring LLM Outputs Are Safe and Correct (2026)

When to Use

• ✓Any structured output extraction (JSON, XML) that flows into a database or downstream system
• ✓Agent tool calls where invalid parameters could cause data corruption or security issues
• ✓Customer-facing outputs that must comply with brand voice, content policies, and accuracy requirements
• ✓RAG outputs that must be grounded in retrieved context to prevent hallucination
• ✓Outputs that include numerical values, dates, or other machine-interpretable data

How It Works

1. 1Schema validation: for JSON outputs, use Pydantic (Python) or Zod (TypeScript) to validate against a defined schema. If validation fails, retry with explicit correction feedback: 'Your previous output was invalid. Error: [specific error]. Please output valid JSON matching [schema].'
2. 2Content validation: check outputs against prohibited content lists, required inclusions, length limits, and format requirements. Use regex for deterministic checks; use a fast LLM for nuanced content policy checks.
3. 3Grounding validation: for RAG outputs, verify that factual claims in the output are supported by the retrieved context. Use an LLM-as-judge to check: 'Is every claim in this response verifiable from the provided context?'
4. 4Action validation: for agent tool calls, validate that action_name is in the allowlist, parameters match the expected schema, and parameter values satisfy business rules (e.g., date in valid range, amount within limits).
5. 5Retry with feedback: when validation fails, don't crash or return an error to the user. Include the validation error in a retry prompt: 'Your previous response contained [specific issue]. Please regenerate the response, fixing [issue].' Allow up to 2 retries before falling back.

Examples

from pydantic import BaseModel, validator
from anthropic import Anthropic
import json

client = Anthropic()

class ExtractedInvoice(BaseModel):
    invoice_number: str
    date: str
    total_amount: float
    vendor_name: str
    line_items: list[dict]
    
    @validator('total_amount')
    def amount_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError('Amount must be positive')
        return v

def extract_with_validation(invoice_text: str, max_retries: int = 2) -> ExtractedInvoice:
    error_context = ''
    
    for attempt in range(max_retries + 1):
        response = client.messages.create(
            model='claude-3-5-haiku-20241022', max_tokens=500,
            messages=[{
                'role': 'user',
                'content': f'Extract invoice data as JSON.{error_context}\n\nInvoice:\n{invoice_text}'
            }]
        )
        
        try:
            data = json.loads(response.content[0].text)
            return ExtractedInvoice(**data)  # Pydantic validation
        except (json.JSONDecodeError, ValueError) as e:
            error_context = f' Previous attempt failed: {str(e)}. Fix this in your response.'
    
    raise ValueError(f'Failed to extract valid invoice after {max_retries} retries')

def validate_grounding(query: str, context: str, response: str, threshold: float = 0.7) -> dict:
    validation_prompt = f'''
Query: {query}
Context provided: {context}
Response to validate: {response}

For each factual claim in the response:
1. Identify the claim
2. Find its source in the context (or note it's unsupported)
3. Rate support: supported/partially-supported/unsupported

Return JSON: {{"overall_grounding": 0-1, "unsupported_claims": ["..."]}}'''
    
    result = client.messages.create(
        model='claude-3-5-haiku-20241022', max_tokens=300,
        messages=[{'role': 'user', 'content': validation_prompt}]
    )
    
    data = json.loads(result.content[0].text)
    return {
        'valid': data['overall_grounding'] >= threshold,
        'grounding_score': data['overall_grounding'],
        'unsupported_claims': data['unsupported_claims']
    }

Common Mistakes

• ✗Validating only JSON structure, not business logic — a response with valid JSON schema can still contain semantically wrong values (negative prices, invalid dates, impossible combinations). Include business rule validation alongside schema validation.
• ✗Infinite retry loops — don't retry indefinitely. Cap at 2-3 retries with degrading quality tolerance. After max retries, return a safe default response or escalate to human review rather than crashing.
• ✗Over-validation that creates false positives — excessive validation rejects valid outputs that don't match overly strict patterns. Calibrate validation rules against a sample of known-good outputs to measure false positive rate before deploying.
• ✗No logging of validation failures — validation failures are signals about model behavior and prompt quality. Log every failure with the original output and the validation error. Review weekly to identify systematic patterns.

FAQ