Hook: Every false positive is a manual review burden. At scale, that's an invisible compliance tax that erodes the ROI of automation.
The Challenge
ML-only PII detection systems produce unacceptable false positive rates in production environments. The Presidio GitHub (Discussion #1071) documents a specific pattern: TFN (Tax File Number) and PCI recognizers with checksum validation produce confidence scores of 1.0 even for non-PII numbers that happen to pass the checksum — because context words are checked after the checksum step, not before. In spreadsheets and log files with numeric data, this creates a flood of false positives. A 2024 study found that even with score_threshold=0.7, 38 out of 39 DICOM images still had false positive entities. Over-detection creates its own compliance risk: over-redacted documents hide relevant evidence, slow workflows, and destroy data utility.
By the Numbers
- Microsoft Presidio GitHub issue #1071 (2024): systematic false positives for German words
- Presidio false positive rate in multilingual production: 3 errors per 1 real entity (Alvaro et al. 2024)
- 22.7% precision rate in mixed-language enterprise datasets
Real-World Scenario
A data engineering team at a healthcare company running Presidio on clinical notes exported to JSON. The raw Presidio output flags hundreds of numeric sequences as SSNs and phone numbers that are actually medical record numbers, dosage amounts, and procedure codes. Manual review of false positives consumes 3+ hours per batch. anonym.legal's hybrid system with configurable thresholds and the MRN entity type reduces false positives by ~70% while maintaining PHI recall.
Technical Approach
The hybrid three-tier architecture separates structured data (regex with 100% reproducibility) from contextual detection (NLP) from cross-lingual detection (transformers). Confidence thresholds are configurable per entity type. Context-aware enhancement boosts scores when context words appear near matches and suppresses false positives when context is absent. The result is dramatically lower false positive rates than Presidio defaults.
Comments (0)