EDO optimizes ensemble weights and cardinality to improve calibration on subjective NLP tasks

Researchers introduce Ensemble Diversity Optimization (EDO), a prediction-space framework that jointly optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective. EDO learns ensemble composition end-to-end via Gumbel-Softmax relaxation and incorporates a signed diversity regularizer to steer optimization toward preserving or suppressing disagreement.

• Experiments on four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) show EDO reduces cross-entropy by 40-78% relative to baselines like Soft-CE and Top-5 Voting.
• The method lowers Brier scores while maintaining competitive F1 scores and better alignment with annotator distributions.

These results demonstrate that jointly optimizing ensemble structure with a signed diversity regularizer provides an efficient, model-agnostic approach for modeling human subjectivity in supervised learning.