The developer liventruth has open-sourced the Unified Latent-State Memory Fabric (UL-SMF), a hardware-software co-designed architecture that compresses standard FP32 Key-Value caches into a 16-dimensional discrete latent space using Finite Scalar Quantization (FSQ).

  • Achieves a 384x compression ratio, reducing a 48 MB FP32 block to 0.12 MB.
  • Maintains 94.15% - 95.84% cosine similarity for semantic retention on real LLM attention tensors.
  • Demonstrates pipeline latency of approximately 14.1 ms to 19.6 ms in end-to-end CUDA event verification.
  • Provides a self-contained Python package allowing users to monkey-patch standard Transformer attention layers with two lines of code.

UL-SMF aims to resolve the memory bottleneck in long-context Transformer inference by combining FSQ with dynamic latent mapping, and is dual-licensed under AGPLv3 with commercial options available.