Srijan Srivastava trains 117M-parameter Silia model on H100 in 5 hours

Srijan Srivastava has trained a 117 million parameter model using the custom Silia architecture, leveraging compute sponsorship to complete training in just five hours on an NVIDIA H100 GPU.

• The model utilizes the Silia architecture, featuring multi-headed attention with rotary positional embeddings, QK normalization, and weight-tied output projections.
• Training was conducted on the synth-100M dataset, processing approximately 82 million tokens with a batch size of 8 and a context length of 1024.
• The author notes the model is severely under-trained due to the limited token count relative to parameter size and a low learning rate, resulting in higher loss than potentially achievable.
• Code for the architecture and inference scripts are available on GitHub, while the research paper and model weights are hosted on Hugging Face and Zenodo.

The release provides an open-source implementation of the Silia transformer variant for further experimentation, though the author explicitly states that benchmarking has not yet been performed.