media Hugging Face Forums · 1h ago · open_models

ELIZA with Sparse Autoencoder

from English

A new implementation of the ELIZA chatbot utilizes a sparse autoencoder architecture to store conversation entries in a latent memory layer, aiming to improve upon traditional keyword-matching methods. The system employs a T5 encoder-decoder structure with a 32,768-neuron memory layer, where similar entries are activated via contrastive learning.

The model achieves logarithmic memory scaling (log(n)) and strong resistance to catastrophic forgetting by using monosemantic neurons and activating only top-k connections.
Training on 10,000 QA pairs from the 'naklecha/minecraft-question-answer-700k' dataset allowed for latent memorization of all entries within three hours.
Unlike standard ELIZA which outputs exact stored sequences, the decoder can slightly modify responses, though it still requires a similar entry to exist for every question.

The author notes that while the approach demonstrates speed and memory efficiency, current limitations prevent scaling to larger datasets to fully verify the compression and forgetting mitigation claims.

Importance 1/3 Trust 1/3 Hugging Face Forums Retrieval & RAG Training methods

Related articles

media Hugging Face Forums · 6d ago

ShinBay-UCTF proposes a conceptual framework for privacy-preserving continuous learning and compressed knowledge

K7007 presents ShinBay-UCTF, a conceptual framework addressing limitations in AI systems regarding multilingual redundancy, persistent episodic memory, and continuous adaptation. The proposal outlines a unified architecture consisting of three interacting layers: Universal Compressed Training Format (UCTF), Biologically Inspired Episodic Memory, and Environment-Driven Continuous Learning.

arxiv arXiv cs.CL · 15d ago · 2 views

Counterfactual supervision improves LLM search routing for Gemma and Qwen

Researchers address the inefficiency of search-augmented language models by formulating an instance-level search-routing problem that decides whether external evidence is needed to improve task success. They derive supervision by comparing no-search and forced-search outcomes to construct an oracle, which serves as both an evaluation criterion and a learning signal for training policies via supervised fine-tuning and preference optimization.

arxiv arXiv cs.CL · 15d ago · 1 view

LongCrafter uses evidence-graph-guided synthesis to improve long-context understanding

The authors propose LongCrafter, a structured framework for synthesizing long-context supervised fine-tuning data that addresses limitations in task coverage and faithfulness. It couples a hierarchical taxonomy of 32 task types with an evidence-grounded pipeline to generate instruction-response pairs strictly grounded in located evidence spans.

arxiv arXiv cs.CL · 16d ago

TR-RAG uses teacher-regularized RL to fix cross-lingual RAG language drift

Researchers propose TR-RAG, a teacher-regularized reinforcement learning method that addresses language drift and unreliable evidence usage in English-evidence cross-lingual retrieval-augmented generation. The approach couples reward optimization with on-policy distillation, using a frozen teacher to provide prefix-wise reverse-KL anchors for answers sampled by a compact student.

media r/LocalLLaMA · 26d ago · 2 views

Book Review: Domain-Specific Small Language Models by Guglielmo Iozzia

This review evaluates Guglielmo Iozzia's book "Domain-Specific Small Language Models," which advocates for a paradigm shift from generalist large language models to specialized, fine-tuned small language models (SLMs). The reviewer argues that SLMs offer superior control, visibility, and cost-efficiency for narrow tasks compared to the hype surrounding artificial general intelligence.