media Hugging Face Forums · 7h ago · open_models

Tutorial: Build and Train a Byte-Level BPE Tokenizer From Scratch

from English

A new tutorial documents the step-by-step process of building and training a byte-level BPE tokenizer from first principles. The guide covers key implementation details such as Unicode-aware pretokenization, incremental pair-frequency updates, and using a max heap with lazy deletion.

Training starts from 256 raw byte tokens and learns merges on multilingual text.
A pair-to-pretoken inverted index is implemented to optimize the training process.
The model was trained on 100 MiB of mC4 data across eight languages.
It results in an 8,192-token vocabulary with 7,936 learned merges.
Training took roughly one hour on a 12-core M2 Pro with 32 GB of memory.

The author provides the full source code and explanations via a GitHub repository and includes a live demo for users to interact with the custom tokenizer.

Importance 1/3 Trust 1/3 Hugging Face Forums Training data Training methods

Read original

media Hugging Face Forums · 11d ago · 9 views

Discussion on Parsewave and quality over scale in post-training data

The article discusses the diminishing returns of increasing post-training data volume as models improve, suggesting a shift toward higher-quality datasets. It highlights Parsewave, which generates post-training data from real-world engineering problems along with evaluation traces.

media Hugging Face Forums · 46d ago · 7 views

Discussion on handling data bottlenecks in scientific ML simulation

A user shares their experience with the computational expense of generating training datasets from physics simulations, such as fluid flow and structural fields. They highlight that the primary challenge is not the model architecture but the difficulty of building a large and diverse dataset when each sample is costly to produce.

arxiv arXiv cs.LG · 49d ago · 5 views

Study derives empirical formula to estimate training set size for inertial sensor classification

Researchers present a systematic empirical evaluation of learning curve convergence rates to address the lack of data-driven guidelines for determining minimum sample sizes in inertial sensor-based classification tasks.

arxiv arXiv cs.AI · 51d ago · 7 views

Study reveals overlap-gap trade-off in synthetic ASR training data

A study investigates conversational timing as a controllable variable for training synthetic multi-speaker conversations in automatic speech recognition (ASR). Researchers parameterized pause and overlap distributions using an exponential-tilting family and explored the resulting four-dimensional space via Latin hypercube sampling and multi-objective Bayesian optimization.

media r/LocalLLaMA · 52d ago · 3 views

TimeCapsuleLLM: training a 500M parameter model on 1800s English texts

A developer has completed a 160GB dataset of 40B tokens containing English data from England and the United States between 1800 and 1875. The author trained a 500M parameter evaluation model on a 5B token sample from this collection.