A new framework enables zero-shot active feature acquisition by leveraging LLMs to elicit only discriminative statistics like unary deviations and pairwise co-variations. Using maximum-entropy closure, it resolves ambiguity in feature selection and outperforms LLMs itself, especially on challenging IBD patient cases where diagnostic uncertainty is high.