The authors introduce a method to build legible transformers using bounded, named units that function as fuzzy set operations rather than dense activations. To prevent the crispness penalty from collapsing operators into dead constants, they implement a per-channel variance floor as a target legibility metric.
- The approach eliminates the need for hand-set reserved-GELU partitions by learning per-unit fractions, routing 87% of computation through crisp operators.
- The resulting model achieves 78% legible feed-forward operands and 50% legible attention value channels, with per-head legibility rising from 18% in shallow layers to 78% in deep ones.
- Edits to these units are significantly more local, offering 50-184x improvement in deep layers where edit sites concentrate.
- Between-unit decorrelation pressure allows trading circuit reuse for independence without quality cost, turning concepts into surgically editable units.
This method enables edits that target explicit conjunctions a single neuron cannot express while maintaining quality parity with conventional baselines.