This article addresses the challenge of training-free source selection for large language models with shared vocabularies in scientific domains like SMILES and genomics, where classical metrics are either uninformative or computationally prohibitive. The authors demonstrate that representation similarity metrics are non-identifiable for transfer because models can share identical representations yet have orthogonal head updates.
- Head Fisher alignment is shown to be exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors without requiring a materialized Fisher matrix.
- FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature and a 192 KB per-task streaming state.
- Llama-3.1-8B verbalizer-shift experiments confirm that FisherSketch remains informative for source selection even when activation similarity cannot distinguish tasks.
The proposed method provides a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling, enabling efficient and accurate model selection without training overhead.