LLMs linearly encode their remaining output length in hidden states

A study reveals that large language models maintain an internal estimate of their remaining response length, which can be decoded from frozen hidden states. Researchers trained minimal-capacity linear probes on three open-weight 7-8B models across seven completion-style datasets to investigate this phenomenon.

• Total response length is linearly decodable from the prompt's last hidden state before any output is generated.
• Probe directions transfer broadly to controlled synthetic completions, outperforming statistical baselines.
• On high-loss completions, the probe's estimate shifts upward when the model retracts and restarts a solution.

The findings suggest that LLMs maintain a plan-like internal representation of output length, distinct from exact-counting impossibility results for transformers.