VibeThinker-3B is a small 3-billion-parameter model that performs exceptionally well on the MathQA benchmark, achieving results comparable to models with around 30 billion parameters. The model's strong performance has sparked discussion about its efficiency and capabilities in mathematical reasoning.
media
r/LocalLLaMA
·
66d ago
·
open_models
VibeThinker-3B: What Is This Witchcraft?
from English
Related articles
arxiv
arXiv cs.AI
·
63d ago
·
4 views
QMFOL: Benchmarking LLM Reasoning with Controllable Logical Complexity
QMFOL is an automated framework that generates monadic first-order logic reasoning tasks with quantifiable complexity. It produces 2880 benchmark instances across 960 configurations, evaluating six large reasoning models and two LLMs, showing performance degradation and increased computational cost as logical complexity rises.
arxiv
arXiv cs.CL
·
64d ago
·
7 views
CombEval: Benchmark for Combinatorial Counting in LLMs
CombEval is a dynamic benchmark that generates natural-language counting problems with verified answers using typed Cofola specifications. It evaluates 11 large language models and reveals persistent failures in handling ordered objects, indistinguishable elements, positional constraints, and nested dependencies, with errors rooted in constraint interpretation and counting principles.
arxiv
arXiv cs.CL
·
64d ago
·
4 views
ForecastBench-Sim: Simulated World Forecasting Benchmark
ForecastBench-Sim is a simulated-world forecasting benchmark using Freeciv game rollouts. It enables continuous or binary forecasts at arbitrary horizons, with intervention worlds for causal questions and rare outcomes, and provides immediate, resolvable feedback for evaluating probabilistic reasoning in dynamic environments.
media
r/LocalLLaMA
·
2h ago
·
3 views
NVIDIA AVO achieves 100% on ARC-AGI-3
NVIDIA's AVO model has achieved a perfect score on the ARC-AGI-3 benchmark. It successfully completed all 183 levels across 25 public environments without any instructions, explicit rules, or stated goals.
lab
Hugging Face Blog
·
3h ago
Artificial Analysis finds ASR models reproduce benchmark transcripts via acoustic cues
Artificial Analysis research reveals that several high-scoring open-source automatic speech recognition (ASR) models optimize for benchmarks by reproducing reference transcripts even when the audio contradicts them. The study evaluated 11 widely used models and identified three specific behaviors: reproducing erroneous reference text, recovering silenced numbers, and switching orthographic variants to match benchmark conventions.