AMALIA fails to validly measure authority construct despite high human agreement

A study evaluates Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, finding it cannot reliably measure the theoretical construct of "authority" despite achieving high agreement with human coders. The research demonstrates that while AMALIA agrees with trained human annotators within six F1 points, it relies on surface correlates like moral outrage rather than the underlying theory.

• The study uses a "recovery gap" metric to test validity by decomposing holistic prompts into atomic clauses and recombining them via explicit rules.
• Decomposition recovers only about half of AMALIA's holistic performance, indicating the model fails to follow the construct's theoretical framework.
• An open multilingual LLM successfully closes this gap on the same Portuguese corpus, suggesting the failure lies with AMALIA rather than the corpus.
• The authors argue that sovereign-LLM benchmarks must test the evidential route of agreement, not just the agreement score itself.

The findings suggest that while AMALIA can screen and pre-code at scale, it cannot yet measure this construct well enough to stand alone as a valid instrument.