RVL-CDIP revisited: fixing label errors and test-train overlap

A new analysis of the RVL-CDIP document classification dataset reveals significant quality issues, identifying 12% label errors and approximately 35% test-train duplication. The authors address these problems by producing corrected variations of the dataset to benchmark their impact on model performance.

• The corpus contains 12% label error and roughly 35% test-train overlap.
• Removing label errors improves classification accuracy, while removing duplicates decreases it.
• Training on error-corrected data substantially improves out-of-distribution generalization on RVL-CDIP-N.
• Supervised models gain an average of 8.1 percentage points in accuracy, with improvements up to 14 percentage points.

Correcting label errors in RVL-CDIP is shown to significantly enhance the ability of supervised models to generalize to out-of-distribution data.