An experimental fine-tuning run of the Dolphin 3.0 Llama 3.1 8B model using the AGIO framework reveals that GPQA Diamond scores stop improving after an initial rise, even as training loss continues to drop sharply.
- The starting model achieved a GPQA Diamond score of 49/198 (24.75%), which peaked at 60/198 in checkpoints CP24 and CP36 before fluctuating between 54 and 56 for the remainder of the run.
- Training utilized a learning rate schedule that started at 1.50e-05, decreased to 1.15e-05 after four epochs, and remained constant thereafter.
- While external accuracy plateaued, internal metrics like the GAP increased from 0.7615 to 1.2999, suggesting the model may be committing more strongly to its choices rather than gaining general capability.
- The author notes that reasoning capabilities were maintained without degradation into "parrot" behavior, despite the divergence between loss reduction and benchmark performance.
The experiment highlights the importance of monitoring learning rate trajectories during fine-tuning, as continued training does not necessarily yield proportional improvements in external evaluation metrics.