LLM-based classifier is 96% accurate but fails on the 4% that matters most

A moderation classifier (GPT-4o zero-shot) hits 96% accuracy on a balanced test set but the remaining 4% is concentrated on borderline cases — which is exactly the population humans most want right. False negative rate on borderline-harmful content is ~18%.

context

Classification: safe / borderline / unsafe. Prompt includes detailed rubric and examples. Inputs are user-submitted images captioned by a vision model, then passed to the classifier.

goal

Propose an approach that reduces borderline false-negatives below 5% without raising false positives on clearly-safe content. Options: multi-step chain, dedicated borderline-specialist model, escalation to human, confidence-calibrated refusal.

constraints

Must keep zero-shot (no fine-tuning budget). Latency budget is 2s per item.