[D] Asymmetric consensus thresholds for multi-annotator NER — valid approach or methodological smell?

Context

I’m training a Spanish legal NER model (RoBERTa-based, 28 PII categories) using curriculum learning. For the real-world legal corpus (BOE/BORME gazette), I built a multi-annotator pipeline with 5 annotators:

Consensus rule: an entity is accepted if ≥N annotators agree on span (IoU ≥80%) AND category.

The problem

Not all annotators can detect all categories. DATE is only detectable by GLiNER + RoBERTa-v2. ADDRESS is similar. So I use asymmetric thresholds:

Actual data (the cliff effect)

I computed retention curves across all thresholds. Here’s what the data shows:

Key observations:

• DATE and ADDRESS drop to exactly 0 at ≥3. A uniform threshold would eliminate them entirely.
• LOCATION is the only category reaching ≥4 (gazetteer + flair + gliner + v2 all detect it).
• No entity in the entire corpus gets 5/5 agreement. The annotators are too heterogeneous.
• Even PERSON_NAME only retains 18% at ≥3.


My concerns

1. ≥2 for DATE/ADDRESS essentially means “both annotators agree”, which is weaker than a true multi-annotator consensus. Is this still meaningfully better than single-annotator?
2. Category-specific thresholds introduce a confound — are we measuring annotation quality or annotator capability coverage?
3. Alternative approach: Should I add more DATE/ADDRESS-capable annotators (e.g., regex date patterns, address parser) to enable a uniform ≥3 threshold instead?

Question

For those who’ve worked with multi-annotator NER pipelines: is varying the consensus threshold per entity category a valid practice, or should I invest in adding specialized annotators to enable uniform thresholds?

Any pointers to papers studying this would be appreciated. The closest I’ve found is Rodrigues & Pereira (2018) on learning from crowds, but it doesn’t address category-asymmetric agreement.