media AI News (TechForge) · 737d ago

Google's experimental Gemini 1.5 Pro 0801 surpasses GPT-4o on LMSYS Chatbot Arena

from English

Google’s experimental Gemini 1.5 Pro model has surpassed OpenAI’s GPT-4o in generative AI benchmarks, specifically on the LMSYS Chatbot Arena leaderboard.

The experimental version, designated Gemini 1.5 Pro 0801, achieved a score of 1,300.
This result exceeds GPT-4o's score of 1,286 and Claude-3's score of 1,271.
A previous iteration of Gemini 1.5 Pro had scored 1,261.

This development marks a significant milestone in the ongoing race for AI supremacy among tech giants.

Importance 3/3 Beats a top-lab benchmark AI News (TechForge) Google DeepMind OpenAI Benchmark results

Benchmarks

Benchmark	Model	Score
LMSYS Arena (Elo)	Gemini 1.5 Pro 0801	1300Elo

Read original

media r/LocalLLaMA · 34d ago

Meta tested ChatGPT, Gemini, and Character.AI with minor-perspective crisis prompts

Meta reportedly had hundreds of contractors pose as minors and send suicide, sex, and drug-related prompts to chatbots from OpenAI, Google, and Character.AI. In a single testing round, more than 45,000 prompts were sent.

arxiv arXiv cs.CL · 35d ago

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 51d ago · 1 view

REDACT: Multilingual PII Benchmark with Systematic Control

REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.

arxiv arXiv cs.CL · 52d ago

Frustrated Synchronization Network Outperforms Transformers

The Frustrated Synchronization Network (FSN) achieves lower validation loss than a RoPE-SwiGLU transformer at every epoch on character-level text and code tasks. At one million parameters, FSN converges to a validation loss of 1.5953 ± 0.0014, outperforming the transformer's converged loss of 1.611. This advantage persists up to four million parameters, with ongoing evaluations beyond that scale.

media IntuitionLabs · 115d ago · 39 views

GPT-5.2 achieves ~53% on ARC-AGI-2 benchmark in December 2025

OpenAI's GPT-5.2, released in December 2025, marked a significant milestone in AI abstract reasoning by achieving approximately a 53–54% success rate on the ARC-AGI-2 benchmark. This performance far exceeded the prior state of the art, where no closed-model system had previously cleared even 5% success on these tasks.