The authors introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases designed to evaluate text-to-SQL models on AI-native SQL functions within the Snowflake platform.

  • The benchmark covers six types of AI functions, including classification, filtering, sentiment analysis, extraction, similarity search, and aggregation.
  • Instances were constructed via an agent-based pipeline that rewrites source tasks into AI-native form while refining natural language instructions to reduce ambiguity.
  • All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release.
  • Evaluations show proprietary models reach 67-70% execution accuracy, while the best open-source model achieves 58.1%, with gaps driven by errors in predicate specification and schema grounding.
  • Traditional agent frameworks for text-to-SQL do not transfer effectively to AI-native SQL, as minimal setups consistently match or outperform more elaborate alternatives.

This benchmark provides the first signal on whether models can generate AI-native SQL, addressing the limitation of existing benchmarks that evaluate only conventional SQL.