Evaluations show Fongbe translations achieve poor quality (1.0-2.2/5) compared to Hausa's acceptable scores (4.0-4.5/5), with a consistent 3x BLEU gap. Automatic metrics like BERTScore show embedding collapse and weak human correlation, especially for Hausa, while Gemini outperforms others for Fongbe and GPT-4o for Hausa in human judgments. Minimum sample sizes of 2,500 sentences are needed for stable model rankings.
arxiv
arXiv cs.CL
·
47d ago
·
src: 50d ago
·
research
Large Language Models Fail to Translate Fongbe Accurately
from English
Related articles
arxiv
arXiv cs.CL
·
37d ago
Automated grading of Linux/bash examinations using large language models
This study evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The research demonstrates that structured prompts significantly improve agreement with human graders, establishing a framework for AI-assisted assessment in computing education.
arxiv
arXiv cs.CL
·
53d ago
Routing Accuracy Degradation and Recovery in Enterprise Agent Systems
As enterprise agent tool catalogs scale from 10 to 110 agents, routing accuracy drops 16--23 percentage points on under-specified requests. An oracle analysis identifies retrieval and confusion gaps, with embedding-based shortlisting recovering +10--11pp F1. A human-annotated study of 1,435 utterances confirms real-world recovery of +10--17pp despite lower absolute performance.
arxiv
arXiv cs.CL
·
35d ago
GPT-5 outperforms humans in inducing belief states via planning
A new study evaluates Large Language Models' ability to induce specific belief states in other agents through actions rather than conversation, a capability termed Non-Conversational Planning ToM (NCP-ToM). Using the NCP-ExploreToM framework, researchers tested six frontier models and human participants on 600 task instances where agents had to move objects or direct characters to achieve belief goals.
arxiv
arXiv cs.CL
·
35d ago
Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy
A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.
arxiv
arXiv cs.AI
·
41d ago
·
1 view
When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs
A study reveals that large language models systematically suppress 'Causal Caution'—the tendency to refrain from causal judgment without sufficient evidence—when shifting from academic to practical advisory contexts. This suppression occurs despite the models retaining the underlying capability, as evidenced by the ability to restore cautious reasoning through specific prompts.