ServiceNow CoreAI introduces AutoSynthData, a pipeline that generates and validates synthetic training tasks for enterprise agents by leveraging target model failures and stronger teacher successes. The system identifies capability gaps through diagnostic evaluations and creates executable tasks with verifiers to address specific weaknesses.
- Tasks are defined by system specifications, user prompts, and verifiers ensuring feasibility, realism, and difficulty.
- A "target" phase generates core samples while a "multiply" phase expands them into novel variants.
- Quality control includes positive/negative verification and critique-driven repair for individual candidates.
- Batch-level review monitors coverage and diversity to prevent overrepresentation of easy task families.
The approach treats synthetic data generation as a search near the model's capability boundary, allowing the curriculum to shift toward remaining difficulties as the agent improves.