LLMs Use Difference-Making Logic to Learn Causal Structure

Large language models learn causal structure through a difference-making logic during training, identifying which word sequences influence others. This approach mirrors the experimental method, using variation in text to infer causal relationships, and is supported by analyses of token embeddings and self-attention mechanisms.