TimeCapsuleLLM: training a 500M parameter model on 1800s English texts

A developer has completed a 160GB dataset of 40B tokens containing English data from England and the United States between 1800 and 1875. The author trained a 500M parameter evaluation model on a 5B token sample from this collection.

• The dataset covers texts from the 19th century, specifically 1800 to 1875.
• A smaller 2B parameter model is planned for future training using the full dataset.
• The current evaluation model was fine-tuned on synthetic questions and answers derived from the source material.
• The project includes a GitHub repository named TimeCapsuleLLM and a Hugging Face model card.

The author notes that while the results are promising, the model currently performs better with London-specific content and requires further refinement for accuracy.