The article presents aider polyglot benchmark results for the new Qwen3 models, highlighting how inference settings and API providers impact open source model performance. It compares pass rates using both "diff" and "whole" edit formats against various configurations.

  • Qwen3-235B-A22B running locally with VLLM, bfloat16, and /no_think settings achieved a 65.3% pass rate on the aider benchmark.
  • The same model accessed via the official Alibaba API yielded a lower pass rate of 61.8%.
  • Results include detailed metrics such as test case counts, cost, and error outputs for each configuration.

The data illustrates that stable inference settings are critical for open source models to match proprietary performance levels.