Researchers identify a "countdown subcircuit" in Llama-3.1-70B-Instruct that enables language models to track remaining tokens across various tasks, such as writing sentences of fixed length or formatting tables.
The study isolates this mechanism in a controlled setting and finds it uses an identical motif previously discovered in other frontier LLMs, suggesting the structure is shared across models. Unsupervised probing on natural language datasets reveals the subcircuit handles tasks where goal lengths are inferred from context rather than explicitly stated.
This work demonstrates that reverse-engineering subcircuits helps explain how specific behaviors generalize from single examples to diverse tasks and different models.