PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMs

This paper introduces PASTA, a framework designed to integrate detailed factual information from news articles into Large Language Models (LLMs) to address the challenge of knowledge updating. The approach combines data augmentation, question-answering generation, and a novel self-learning Direct Preference Optimization (DPO) process to enable knowledge overwriting and hallucination suppression.

PASTA utilizes a combination of data augmentation and question-answering generation alongside a self-learning DPO process. The method simultaneously enables knowledge overwriting and suppresses hallucinations in the updated models. Experimental evaluation on web articles published after the base model's knowledge cutoff showed accuracy improvements from 0.02 to 0.82. The framework maintains general language capabilities while creating domain-specialized LLMs.

PASTA demonstrates effectiveness for building specialized models that accurately answer questions about specific factual information, such as news articles, which LLMs typically struggle with.