A study examines activations at the final prompt token in twelve instruction-tuned models from the Bielik, PLLuM, Gemma-4, and Qwen3 families to determine if they can estimate familiarity with an entity before generating an answer. Using a dataset of 1,440 Polish entities, the researchers found that familiarity-probe scores effectively separate real from fabricated entities across all model families.
- Familiarity-probe scores track entity popularity in Polish-adapted Bielik and PLLuM models (Spearman ρ 0.28-0.57), a pattern less evident in Gemma-4 and Qwen3.
- Probes retain 96-101% of within-language AUROC when the question stem is switched from Polish to English, demonstrating robustness to prompt language.
- In Gemma-4-12B, adding a one-dimensional familiarity direction at a single layer allows monotonic control over refusal rates for both known and unknown entities.
- Calibrated familiarity probes are competitive as pre-generation abstention gates, though post-generation detectors better predict behavioral error.
The results support the existence of graded pre-generation entity-familiarity readouts and highlight a separation between representational familiarity and the policy that converts it into abstention.