The DiaLLM study reveals that dialectal robustness and generation are dissociated in large language models, as benchmarks shaped by continual pretraining do not capture how alignment reshapes actual output. The authors continually pretrain three open-weight model families on the International Corpus of English and apply implicit and explicit post-training paradigms combined with three alignment strategies for Australian, Indian, and Northern British English.
- Explicit variety-targeted adaptation produces output reliably recognized as dialectal and preferred over broad alignment.
- The method that most aggressively optimizes the dialectal reward is not preferred by human evaluators, indicating a reward-quality gap.
- Independent linguistic analysis corroborates this gap, particularly for two of the three model families.
- No single alignment method dominates, suggesting richer reward designs are needed to close the gap.
The authors release all code, checkpoints, and preference datasets to support further research into dialectal resources.