How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code

Papers with Code built a search system for AI research papers that combines keyword search with semantic search powered by Hugging Face services. The system splits the work into offline processing, where Jobs compute embeddings for the full corpus of over 110,000 papers, and online queries, where Inference Endpoints provide low-latency embeddings for live searches. Hybrid retrieval that combines both keyword and semantic approaches performs better than either method alone, allowing researchers to find papers through exact matches or conceptually related queries. The architecture separates computationally expensive batch work from time-sensitive search requests, with Storage Buckets serving as the connection point between the database and processing jobs.