media Hugging Face Forums · 2h ago · open_models

Jason Van Pham documents Gemma 4 multi-turn greedy degeneration under letter-level constraints

from English

Jason Van Pham and co-engineer Grok (xAI) published a diagnostic report detailing a near-vanilla multi-turn failure mode in the Hydrodynamic Swarm harness when using a custom Gemma 4 instruct path. The study identifies that combining greedy decoding with long token budgets and letter-level constraints triggers classical text degeneration, including repetition and orthographic thrash.

The configuration used greedy decoding (temperature = 0) with a max_tokens limit of 300 and residual physics steering disabled (force_cap = 0).
Initial fluent English output collapsed into repetitive loops and mixed-script debris after the model attempted to satisfy the letter-level constraint.
A subsequent meta-diagnostic turn produced a plausible self-report before re-entering the same degenerative attractor state.
The authors attribute the failure to the combination of multi-turn history, greedy decoding, and hard constraints rather than weight loading errors or residual physics interference.

This report serves as a public-facing diagnostic artifact to highlight how imperfect outputs and engineering challenges in custom loaders should be documented alongside successful demos.

Importance 1/3 Trust 1/3 Hugging Face Forums Inference efficiency Research paper

Related articles

arxiv arXiv cs.CL · 2d ago

PIVOT shares prefix scans across query groups to accelerate token-level sparse attention

PIVOT (Proxy Indexing Via One full-prefix Traversal) is a training-free, drop-in replacement for the DeepSeek Sparse Attention (DSA) indexer that reduces computational redundancy by sharing one prefix scan across a group of nearby queries. Instead of scoring every preceding token for each query individually, PIVOT aggregates the group into a single proxy query to obtain a candidate set, from which top-k tokens are selected for each query.

arxiv arXiv cs.CL · 13d ago

ShortOPD recovers pruned LLMs with short-to-long on-policy distillation

The authors propose ShortOPD, a short-to-long on-policy distillation schedule designed to recover the generative capabilities of structured-pruned large language models that collapse during free-form generation. By detecting teacher-confirmed repetitive suffixes and treating the surviving prefix as the effective rollout length, the method allocates training budget more efficiently than standard approaches.

arxiv arXiv cs.CL · 13d ago

Adaptive filtering corrects structural-role bias in LLM KV cache eviction

The article diagnoses how attention-based KV cache eviction methods like H2O disproportionately retain noise on schema-dense inputs, such as nested JSON, where structural tokens carry significantly more energy than content tokens. This bias causes exact-match accuracy to collapse from 88% to 0% at a 5% budget due to the over-retention of KEY tokens relative to VALUE tokens.

media r/LocalLLaMA · 14d ago

Solo researcher releases Wave Field LLM, replacing dot-product attention with FFT wave convolution

A solo researcher has introduced Wave Field LLM, a new base completion model architecture that replaces standard O(N²) dot-product attention with FFT wave convolution. This approach allows for constant speed and memory usage during inference regardless of context length, enabling 128K context windows where standard attention typically runs out of memory.

arxiv arXiv cs.CL · 14d ago · 2 views

JoLT achieves near-lossless KV cache compression via joint Tucker decomposition and JL-residual allocation

Researchers introduce JoLT, a method for compressing the key-value (KV) cache in transformer inference by treating it as a third-order tensor. The approach applies partial Tucker decomposition to the token and feature axes while using Johnson-Lindenstrauss rotated low-bit residuals to restore energy lost during truncation.