Cross-survey transfer shows zero-shot LLMs reach 52% accuracy on unseen survey items

The article introduces cross-survey transfer, a rigorous evaluation framework for silicon sampling where large language models predict answers to entirely different questions based on a respondent's previous answers. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, the study evaluates three open-weight LLMs with 27B-120B parameters against supervised machine learning baselines.

• Zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points of a supervised random forest trained on same-population data.
• A stable construct predictability hierarchy emerges, ranging from 67% for partisan attitudes down to 23% for sovereignty.
• Variance collapse and safety alignment effects are found to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families.

These findings clarify both the promise and boundaries of silicon sampling by demonstrating that LLMs can effectively augment traditional survey research through individual-level prediction.