A Reddit user has extended the Google Gemma 4 31B model, which has 60 layers, into a larger 44B parameter version containing 88 layers. This modification was undertaken because Google has not released a larger dense version of the model for use on Lyzr Architect.
- The expansion increases the layer count from 60 to 88 using a Gemma-specific scaling strategy inspired by LLaMA Pro.
- New layers were initialized using knowledge from the base model and structural patterns learned during training.
- Early results indicate the expanded model retains the original identity while gaining additional capacity.
The author notes that further testing is required to observe how the model behaves when context gets stacked up.