SQuaD-SQL enables small language models to approach LLM performance on Text-to-SQL via knowledge distillation

Researchers introduce SQuaD-SQL, a method that allows small language models (SLMs) to achieve near-LLM performance on the Text-to-SQL task while significantly improving efficiency through knowledge distillation and synthetic data generation.

The approach comprises three key components: LLM-based synthetic data generation using structured knowledge extraction, parameter-efficient fine-tuning for training on a single consumer-grade GPU, and domain-adaptive fine-tuning to enhance performance in targeted domains. Experiments on the WikiSQL dataset demonstrate that SQuaD-SQL achieves an execution accuracy of 86.9% on the test set.

These results suggest that with proper training strategies, SLMs can serve as practical and efficient alternatives for Text-to-SQL applications in resource-limited environments.