Prompt Compression: Reducing Token Count Without Losing Quality (2026)

When to Use

• ✓High-volume RAG pipelines where retrieved context is long (2K+ tokens) and you run thousands of queries per day
• ✓Fitting long documents into smaller, cheaper models to reduce cost while maintaining quality
• ✓Reducing latency in latency-sensitive applications where time-to-first-token matters
• ✓Extracting only the information-dense portions of boilerplate-heavy documents (legal contracts, terms of service)
• ✓A/B testing whether a compressed prompt maintains quality before permanently switching to a cheaper model

How It Works

1. 1Manual compression: remove verbose instructions, collapse repeated whitespace, shorten few-shot examples, use abbreviations. Often reduces prompts by 20–40% with minimal quality impact on well-written prompts.
2. 2Selective truncation: rank sentences or chunks by relevance to the query (BM25 or embedding similarity), keep only the top-k. Simple but effective — relevant sentences are usually 30% of a retrieved document.
3. 3LLMLingua (Microsoft, 2023–2025): a small LM (125M parameters) scores each token's importance given the query and removes low-importance tokens. Achieves 5–20x compression with 5–10% quality drop on reading comprehension tasks.
4. 4Semantic summarization: use a cheap model (GPT-4o-mini, Claude Haiku) to summarize retrieved chunks before passing to the expensive model. Effective for narrative documents; less effective for technical content where exact wording matters.
5. 5Caching compressed versions: pre-compress static documents in your corpus and store the compressed form in the vector store alongside original text. This amortizes compression cost across all queries.

Examples

BEFORE:
You are a helpful assistant. Your task is to carefully read the user's question and provide a comprehensive, accurate answer. Please make sure to consider all aspects of the question before responding. If you are not certain about something, please indicate that clearly.

AFTER:
Answer accurately. Flag uncertainty explicitly.

# Python: filter retrieved chunks to top-3 most relevant sentences
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def compress_context(query_emb, chunks, chunk_embs, top_k=3):
    scores = cosine_similarity([query_emb], chunk_embs)[0]
    top_idx = np.argsort(scores)[-top_k:][::-1]
    return ' '.join(chunks[i] for i in top_idx)

Common Mistakes

• ✗Compressing without measuring quality — always compare compressed vs. full-context performance on a held-out eval set. Some tasks (exact quote extraction, legal analysis) degrade severely with compression.
• ✗Applying the same compression ratio to all document types — code and structured data should not be compressed (removing tokens breaks syntax); narrative text can handle 70%+ compression.
• ✗Using compression instead of prompt caching — if your context is static or semi-static, prompt caching (90% cost reduction) is better than compression (lossy). Try caching first.
• ✗Over-compressing system prompts — system prompts are read-heavy; the model relies on every instruction. Compressing them aggressively causes instruction-following failures.

FAQ