The authors propose a parameter-neutral replacement for the standard feed-forward network (FFN) in transformers, called NC-FFN, which uses explicit fuzzy set operations like intersection and bounded positive negation. This design allows every hidden unit to carry an explicit logical form while tying the GELU baseline's perplexity at scale.
To address logic localization and eroding performance, the model incorporates a block of sequence quantifiers with learned forgetting rates. This approach recovers grammatical deficits early in training and modestly improves LAMBADA scores. The resulting FFN becomes legible, with units acting as grammatical licensing detectors that fire on licensors like comparatives or negative-polarity items to predict licensed words. This work provides a readable, interpretable-by-construction mechanism for how transformer FFNs license language, offering an account of both representation and function.