Quantifying Agreement Between Data-Influence and Data-Similarity in LLMs

This study quantifies the agreement between data-similarity and data-influence measures used for tracing LLM outputs back to training data, revealing a significant overlap with an asymmetry where data-influence ranks top similar documents more consistently. Experiments across models including OLMo2-1B, Qwen3-1.7B, LlaMa3.2-1B, Gemma3-1B, and GPT2 demonstrate that this asymmetry allows for a favorable cost-accuracy trade-off by using data-influence to refine cheaper data-similarity results.

• The two ranking measures agree significantly, but data-influence assigns more consistent ranks to the top documents of data-similarity than vice versa.
• This finding holds across multiple models: OLMo2-1B, Qwen3-1.7B, LlaMa3.2-1B, Gemma3-1B, and GPT2.
• The asymmetry is exploited to improve cost-accuracy trade-offs by using costly data-influence to refine the results of cheaper data-similarity.

This result helps users achieve better accuracy in output tracing by leveraging the complementary nature of these two measures.