media AIBase · 608d ago

Google's Gemini-Exp-1206 surpasses ChatGPT on LMArena leaderboard

from English

Google has launched an experimental language model named Gemini-Exp-1206, which has achieved the highest Arena Score on the LMArena (Chatbot Arena) leaderboard. The model reached 1379 points, slightly exceeding ChatGPT-4.0's score of 1366 points.

Gemini-Exp-1206 scored 1379 on LMArena, compared to ChatGPT-4.0's 1366.
ChatGPT-4.0 received significantly more votes (21,929) than Gemini-Exp-1206 (5,052), indicating greater reliability.
The model demonstrates stronger performance than the previous Gemini-Exp-1114 version.
Google describes these experimental models as temporary prototypes for testing and feedback, not suitable for production.

The model is available for free via Google AI Studio, offering developers a chance to experience Google's latest AI advancements ahead of time.

Importance 3/3 Beats a top-lab benchmark AIBase Google DeepMind Benchmark results

Benchmarks

Benchmark	Model	Score
LMSYS Arena (Elo)	Gemini-Exp-1206	1379Elo

Read original

arxiv arXiv cs.AI · 34d ago

TestEvo-Bench: Executable live benchmark for test and code co-evolution

Researchers introduce TestEvo-Bench, a live benchmark designed to evaluate how well agents handle the co-evolution of software tests and code. Unlike existing benchmarks that rely on static metadata, this framework uses real commit histories and environment configurations to verify executable metrics like pass rate and coverage.

media r/LocalLLaMA · 34d ago

Meta tested ChatGPT, Gemini, and Character.AI with minor-perspective crisis prompts

Meta reportedly had hundreds of contractors pose as minors and send suicide, sex, and drug-related prompts to chatbots from OpenAI, Google, and Character.AI. In a single testing round, more than 45,000 prompts were sent.

arxiv arXiv cs.CL · 34d ago · 1 view

QLoRA fine-tuning of Gemma 4 31B closes cross-lingual relation extraction gap for Romanian

The study investigates cross-lingual relation extraction for Romanian by translating the SemEval-2010 Task 8 benchmark and evaluating Gemma 4 31B against encoder baselines. Results show that QLoRA fine-tuning improves macro F1-Score by more than 22 percentage points, reducing the cross-lingual gap from 3.3 to 1.4pp.

arxiv arXiv cs.CL · 35d ago

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 35d ago · 2 views

LLM evaluators lack clinical caution despite statistical alignment in MedQADE benchmark

The study introduces MedQADE, a standardized open-response clinical benchmark for German comprising 3,800 items annotated by ten physicians and nine LLM evaluators. It investigates whether automated LLM-as-a-Judge approaches replicate the calibration and caution of human clinicians.