A study evaluated whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses from second-year Computer Engineering students.
- The research utilized a four-level cognitive taxonomy ranging from information retrieval to advanced system management across 1200 real student responses graded by three experts.
- Gemini 3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement, with an ICC(3,1) of 0.888 and a Mean Absolute Error (MAE) of 0.10.
- Agreement declined as taxonomy level increased, with the largest discrepancies occurring at higher complexity levels.
- Across all models, the quality of the rubric had a larger effect on performance than the model provider choice, with structured prompts consistently improving agreement.
These results establish a principled framework for determining which questions are suitable for AI-assisted grading and provide a transferable evaluation protocol and prompt templates.