Papers with Code has implemented a hybrid search system combining keyword and semantic retrieval to help users find relevant AI research papers. The architecture leverages Hugging Face Jobs, Storage Buckets, and Inference Endpoints to manage the embedding pipeline and serve low-latency queries.
- Hugging Face Jobs provides burstable GPU compute for embedding over 110,000 papers using the Qwen/Qwen3-Embedding-0.6B model.
- Storage Buckets act as durable storage between database exports, job processing, and index importation.
- Inference Endpoints serve low-latency embeddings for live queries, falling back to full-text retrieval if unavailable.
- The system uses reciprocal rank fusion (RRF) to combine lexical PostgreSQL search with dense vector search from pgvector.
- A strict embedding contract ensures reproducibility by versioning the model revision, input format, and normalization method.
This separation of offline corpus building and online search services allows the platform to maintain both high retrieval quality and fast response times for researchers.