The author reports that the extGemma4-40_5B model, an extended version of Gemma 4, successfully overcomes previous failures in layer insertion by using a new initialization strategy. By initializing new layers as a blend of their neighbors rather than as no-ops, the model avoids signal starvation and allows for effective training.
- The author stopped inserting layers at awkward spots and kept per-layer volume settings neutral to preserve signal integrity.
- Training occurred in two rounds: first freezing the original model to teach only the new layers, then unfreezing everything for joint settlement.
- Benchmarks show the healed model recovers most ground lost from surgery, landing close to the original on GPQA-Diamond.
- In side-by-side comparisons across 10 diverse topics, the expanded model tied with the parent on 9 questions and outperformed it on a physics question involving de Broglie wavelength.
The experiment demonstrates that major structural surgery on an already fine-tuned model can heal successfully rather than collapsing into gibberish.