Creating Specialized RAG-Based Search Engines Using the Open Web Index

This paper presents a concept and supporting technology for building RAG-based specialized search engines using open-source frameworks and open web data. The Open Web Index (OWI) provides openly accessible web data, while the modular MOSAIC framework is designed to integrate topical OWI partitions obtained to create search applications tailored to specific use cases. MOSAIC-RAG extends this framework with features based on Large Language Models (LLM), such as summarization or re-ranking. Using this infrastructure, special-purpose and domain-specific search applications can easily be developed and experimented with. For demonstration purposes, we present three example applications in the topical domains of science, health, and arts.