RPAM: principled metric for evaluating associations in language models

Researchers introduce the Relative Probability Association Metric (RPAM), an upstream evaluation method for generative language models that analyzes embeddings or continuation probabilities to assess underlying associations.

• RPAM is designed to overcome the generalization limits of downstream metrics by examining models at a fundamental level rather than relying on variable generated text.
• The metric was tested on Mistral-7B-Instruct, Mistral-7B, and GPT-2 using datasets including WEAT-WS, Bellezza, WS-353, and SST2.
• Results show a strong relationship between RPAM measurements and both human implicit/explicit associations and biases measured in downstream tasks, outperforming prior record values where applicable.

RPAM addresses the gap in upstream metrics by providing a method that uncovers strong relationships with real-world associations, enabling principled analysis across different language models.