Independent researcher develops 270M parameter language model from scratch

An independent researcher has created a 270 million parameter language model entirely from scratch. The project is available on Hugging Face under the name Wiki-SmartBotLM-Instruct.

The model utilizes a custom Transformer architecture designed for efficiency and local inference. Key technical features include Rotary Positional Embeddings, RMSNorm, SwiGLU feed-forward layers, grouped query attention, and an optimized autoregressive decoder.

This work demonstrates the feasibility of building functional large language models independently without relying on pre-existing weights or massive infrastructure.