Percepta has introduced a new neural network architecture called Spotlight, which replaces traditional attention mechanisms with a writable, unbounded memory system. This design allows the model's knowledge and skills to grow without requiring changes to the model's weights or increasing access costs per token.
- Spotlight separates an intelligence module for computation from memory that holds knowledge and procedures.
- The architecture is arbitrarily sparse, allowing the fraction of used memory to shrink as capacity grows.
- The model decides what to load and overwrite token by token, enabling new capabilities without retraining.
- Unlike mixture-of-experts models, Spotlight touches a small number of cells regardless of total memory size.
This separation allows the intelligence module to remain fixed in size while the memory expands, theoretically enabling infinitely growing memory without increasing computational overhead during access.