EDO framework optimizes ensemble diversity to improve calibration on subjective NLP tasks

Researchers introduce Ensemble Diversity Optimization (EDO), a prediction-space framework designed to handle systematic annotator disagreement in subjective NLP tasks by jointly optimizing ensemble weights, cardinality, and calibration. The method uses Gumbel-Softmax relaxation for end-to-end learning and incorporates a signed diversity regularizer to control the utility-calibration trade-off.

• EDO integrates a soft F1 surrogate, class-weighted cross-entropy, and reliability-weighted diversity to regulate intra-ensemble variability.
• Experiments on ArMIS, ConvAbuse, HS-Brexit, and MD-Agreement benchmarks show reduced cross-entropy (40-78%) and lower Brier scores compared to Soft-CE, Soft-MD, Top-5 Voting, and WEL.
• The approach maintains competitive F1 scores while achieving better alignment with annotator distributions.

This model-agnostic approach provides an efficient method for modeling human subjectivity in supervised learning by preventing ensemble collapse.