This study evaluates the ability of large language models to detect and classify reports of antisemitic events using fine-grained labels. The authors tested OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on expert-annotated datasets derived from news articles, civil society reports, and official records.
- GPT-4o demonstrates potential for this task, though substantial performance improvements are required across the board.
- Providing clear term definitions in prompts significantly improves detection of rhetoric-oriented events, such as classical antisemitic tropes.
- Including in-context examples is more effective for labeling action-oriented events, such as physical assault.
- A case study involving college newspapers shows that LLMs can help surface relevant real-world events to support early monitoring and intervention.
The findings highlight critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among developers, policymakers, and civil society to design robust evaluation frameworks.