Empirical study evaluates RAG metrics against human annotators

This paper reports an empirical study evaluating the relevance of several Retrieval-Augmented Generation (RAG) metrics. The experiment uses a question-answering dataset created by human annotators from business data.

• Generated responses and retrieved spans were scored using evaluation metrics from four libraries: Ragas, DeepEval, RAGChecker, and Opik.
• These scores were compared to ratings given by two human evaluators as well as standard metrics such as recall.
• The study conducts an analysis of correlations between the automated metrics and human evaluations.
• The authors highlight limitations of their methodology, compare it to existing literature, and suggest avenues for future research.

This work is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).