Researchers introduce CausalDS, a benchmark designed to evaluate causal reasoning within agentic data-science workflows. Unlike existing benchmarks that separate symbolic reasoning from realistic analysis, CausalDS uses sampled structural causal models paired with synthetic natural-language stories grounded in realistic domains.
- Each instance combines an observational dataset derived from the structural causal model with a narrative story.
- Tasks span all three of Judea Pearl's rungs, allowing for evaluation of complex causal inference beyond simple prediction.
- The benchmark requires models to use multiple tools and coding skills to handle imperfect observations generated by an observation model.
- It treats abstention as a first-class scored outcome, evaluating uncertainty quantification alongside symbolic reasoning and tool use.
This approach jointly evaluates symbolic causal reasoning, data science capabilities, uncertainty quantification, and tool usage while reducing the risk of models merely parroting curated examples.