Realtime Content Moderation Pipeline

PhotoDNA is industry standard for CSAM detection and free for qualifying platforms through Microsoft. GIFCT HSP covers terrorist content. Both run in under 5ms per image. You want these even if you also have model-based moderation.

Alternatives: Apple NeuralHash, Internal MD5 + pHash

Self-hosted embedding + classifier gives you policy-specific control and 20-40ms latency. OpenAI's Moderation API is free and good enough to start. Switch to fine-tuned once you have 10k+ labeled examples and need custom policies.

Alternatives: OpenAI Moderation API (free), Perspective API, Azure Content Safety

Rekognition has the best NSFW and violence recall on real-world UGC at a reasonable $0.001 per image. Hive is better on cartoon/anime content. Azure Content Safety has the strongest CSAM detection integration for Microsoft-aligned stacks.

Alternatives: Hive Moderation, Google Vision SafeSearch, Azure Content Safety Image

Sonnet 4 follows complex policy documents better than GPT-4o for moderation. Haiku 4 is good enough for lower-stakes policies (spam, soft NSFW) at one-fourth the cost. Keep Opus 4 out of the hot path - use only for appeals.

Alternatives: GPT-4o, Gemini 2.5 Pro, Claude Haiku 4

OPA lets non-engineers (Trust & Safety leads) edit policy rules without deploys. Critical when regulators change requirements mid-year (DSA, UK Online Safety Act).

Alternatives: Pure Python rules, AWS Step Functions, Temporal

You need precision/recall dashboards sliced by policy, geo, and model version. ClickHouse handles billions of rows for realtime dashboards. Braintrust runs the daily golden-set evals.

Alternatives: Arize Phoenix, Langfuse, Custom dashboard

LLM-only vs tiered pipeline

Routing every item through Claude Sonnet 4 would cost 50-100x more and blow P95 latency past 1 second. A tiered pipeline (hash + small classifier + LLM on the tail) handles 95% of decisions in under 100ms at one-tenth the cost. The LLM only touches the ambiguous middle where its judgment actually improves outcomes.

Precision vs recall tradeoff

High-recall (catch everything) policies like CSAM require low thresholds and large human queues - false positives are cheap, false negatives are catastrophic. High-precision policies like misinformation need high thresholds to avoid over-blocking speech. Tune thresholds per policy, not globally.

Managed API vs self-hosted classifier

Perspective API and OpenAI Moderation are free and fine at low volume. Self-hosting a fine-tuned DistilBERT becomes cheaper around 10M items/month and gives you policy-specific control. Self-hosting also lets you train on your own labeled data, which is the only way to beat generic APIs on platform-specific content.

Adversarial inputs (leetspeak, homoglyphs, image perturbations) slip past classifiers

Mitigation: Run a text-normalization pass (unicode NFKC, leetspeak dictionary, homoglyph detection) before the classifier. For images, use perceptual hashes (pHash, wavelet hash) not just cryptographic hashes so minor edits still match known-bad content.

Model over-blocks legitimate speech (false positive spike after model update)

Mitigation: Shadow-deploy every model update for 24-48h before switching traffic. Compare precision/recall vs current prod on a golden set of 10k+ labeled items. Alert if precision drops more than 2% or if false-positive rate on any protected category (race, religion, LGBTQ+ terms used positively) spikes.

Human review queue overflows and SLAs slip

Mitigation: Priority-tier the queue: credible-threat and CSAM get 15-minute SLA, hate speech 2h, soft NSFW 24h. Auto-apply soft actions (shadow-remove, lowered reach) while waiting. Track queue depth per tier and auto-escalate staffing when depth grows beyond throughput.

Policy drift - enforcement differs from written policy

Mitigation: Treat the policy document as ground truth and run weekly evals of the full pipeline against it. Maintain a labeled golden set of 2-5k items per policy category and require precision/recall targets before deploying changes.

Regulatory exposure (DSA, UK Online Safety Act, COPPA)

Mitigation: Log every decision with model version, scores, reviewer, jurisdiction, and rationale. Produce monthly transparency reports automatically. For EU traffic, route through a DSA-compliant appeals workflow with 14-day response SLA. Treat under-13 (COPPA) and under-18 (various) users with stricter thresholds.