An empirical study found that projecting MLP weights from one transformer model into another fails to transfer semantic capability. Every tested variant performed worse than the unmodified host model, indicating a structural limitation in weight projection. The results challenge public claims about model capabilities based on benchmarks, showing such claims do not reflect actual internal weight geometry.
Capability Is Not in the Weights: Empirical Negative Result on MLP Weight Projection
from English