AMALIA-9B fails to validly measure moral authority despite high human agreement

A study evaluates Portugal's AMALIA, a 9B-parameter European Portuguese model, finding it cannot validly measure the moral foundation of authority despite high agreement with human coders. While AMALIA matches open models eight to thirteen times its size in raw agreement scores, it fails to recover performance when prompts are decomposed into atomic clauses, suggesting reliance on surface correlates like moral outrage rather than theoretical constructs.

• AMALIA agrees with trained human coders within six F1 points of larger open models.
• Decomposition recovers only about half of AMALIA's holistic performance, indicating a failure to follow the construct's theory.
• An open multilingual LLM closes this gap on the same Portuguese corpus, pointing away from the corpus as the main explanation.

The study argues that sovereign-LLM benchmark batteries should test not only agreement with human coders but the evidential route by which that agreement is warranted.