WideNDepth separates knowledge storage from iterative reasoning

A researcher proposes WideNDepth (WND), a neural architecture that decouples knowledge storage from the reasoning process. The design uses a Wide Layer and Encoder to generate rich representations stored in a Feature Bank, while a separate Depth Layer handles iterative reasoning via compressed states.

• The Wide Layer holds richer information, processed by an Encoder into a representation stored in a Feature Bank.
• A Compressor reduces this representation into a smaller state for the Depth Layer.
• The Depth Layer performs step-by-step reasoning over the compressed state, using attention-based retrieval to access the Feature Bank when needed.
• On graph reasoning tasks involving long chains and medium difficulty, a ~100K parameter WND model outperforms a ~600K Transformer baseline in performance, training speed, sample efficiency, and footprint.

The author notes that the Wide Layer and Encoder currently dominate the workload, leaving the Depth Layer underutilized, and seeks methods to constrain these components to focus solely on producing useful information for the reasoning layer.