The authors introduce MTEB-PT, a dedicated benchmark for evaluating text embeddings in Brazilian Portuguese that excludes translated data. The benchmark comprises 22 native tasks across seven categories and evaluates 93 models ranging from 23M to 27B parameters.
- The evaluation includes 73 open-weight models and 20 closed commercial APIs, with a statistical layer providing bootstrap confidence intervals and discrimination analysis.
- The benchmark separates models into distinct tiers, identifying an openly licensed, self-hostable model that reaches the leading tier.
- A moderate correlation (Spearman rho = 0.75) between global multilingual leaderboard ranks and Portuguese ranks indicates that native benchmarks measure different capabilities.
The release of tasks, code, and a public leaderboard allows practitioners to select embedding models based on native evidence rather than translated corpora.