Benchmark · agentic

MLE-bench

0 results 0 models

MLE-bench measures how well autonomous AI agents can carry out end-to-end machine learning engineering across 75 real Kaggle competitions; the headline metric is the fraction of competitions in which the agent's submission earns a Kaggle medal (bronze or better).

Example: A single task is one Kaggle competition handed to the agent as a problem description, a training dataset, and a required submission format — for example, an image-classification competition where the agent must train a model on labeled images and output predictions for a held-out test set.
Scoring: Each competition is scored with its own original Kaggle metric (e.g., AUC, accuracy, or RMSE) on a held-out private test set; that raw score is placed on a locally reconstructed leaderboard and converted to a medal (gold/silver/bronze/none) using Kaggle's percentile thresholds. The primary reported number is the percentage of competitions earning any medal, typically averaged over several runs.
Verification: A submission counts only if it is a validly formatted prediction file produced within the run's compute and time budget; it is then graded locally against the private labels and compared to the medal cutoffs derived from the competition's real participant counts. Runs are checked for rule violations such as reading the answer labels.
Why it matters: Kaggle competitions are open-ended, multi-hour ML engineering problems — data cleaning, feature work, model selection, training, and debugging — so success requires sustained autonomous effort rather than a single answer, making MLE-bench a realistic test of whether agents can do the job of an ML engineer.

Worked example

Task

Competition — binary image classification. You are given a training set of small (32×32) aerial photographs, each labeled 1 (contains a columnar cactus) or 0 (does not), plus an unlabeled test set. Produce submission.csv with columns id and has_cactus giving each test image's predicted probability of containing a cactus. Evaluation metric: area under the ROC curve (AUC).

Solution

Key steps: (1) load the labeled images and split into train/validation; (2) build a data pipeline with augmentation (flips, rotations, normalization); (3) fine-tune a pretrained CNN (e.g., ResNet) with a binary-cross-entropy head, using early stopping on validation AUC; (4) predict cactus probabilities on the test set and write id,has_cactus rows to submission.csv. Final deliverable: a valid submission.csv whose private-leaderboard AUC clears the bronze-medal percentile cutoff.

Walkthrough

There is no single 'right answer' — any pipeline that generalizes well enough to push test AUC above the competition's medal threshold succeeds; a fine-tuned CNN with augmentation is a strong, standard approach for small-image binary classification and reliably reaches high AUC. Grading: the submission is scored by AUC against the held-out private labels and mapped to a medal via Kaggle's percentile cutoffs.

No verified scores reported yet for this benchmark.