Multi-Domain Neural Search Engine — Domain Routing, Hybrid Retrieval and RAG Grounding Evaluation
Overview
A search engine over six subject domains, each with the encoder that was pretrained on that literature, and a router that decides which one a query belongs to. That part began as MSc Information Retrieval coursework and was built as a team. Afterwards I added the part that interested me more: a generation layer, and a way to test whether the generated answer was actually using the passages the retriever had found.
Why one encoder is not enough
A single general-purpose embedding model treats a clinical trial abstract, a court judgment and an earnings filing as the same kind of text. They are not. Each domain has its own vocabulary, its own sense of which words carry the meaning, and domain-tuned tokenisers built on SciBERT, BioBERT, FinBERT and ClinicalBERT are used to chunk each corpus on its own vocabulary. So the engine holds six indexes, one per domain, over MS MARCO, SciFact, FiQA, TREC-COVID, NFCorpus and SCIDOCS, encoded with BAAI/bge-base-en-v1.5 and msmarco-bert for the general domain. That moves the difficulty to the front of the pipeline: something has to decide where a query goes before it can be answered. A fine-tuned DistilBERT router does that, and three routing strategies were compared against an oracle upper bound, so the cost of routing errors is measured rather than assumed.
Retrieval
Lexical and dense retrieval fail in different places. BM25 misses a paraphrase; a dense encoder misses an exact identifier. Both run, and their result lists are combined with reciprocal rank fusion, which merges by rank rather than by score and therefore does not require the two systems' scores to be on a comparable scale. FAISS-HNSW carries the dense side. ColBERTv2 reranks the top of the fused list with late interaction, matching query and passage token by token instead of comparing two single vectors. Every one of those stages is a claim that it earns its cost, so the system is evaluated as a five-step ablation ladder rather than as a finished pipeline: each component is switched off and the result recorded. I also annotated a cross-domain query set by hand, because the existing BEIR benchmarks are single-domain and cannot measure whether routing across domains works at all.
The question the retrieval metrics cannot answer
Put a generator on top and the system produces answers. nDCG and recall say the retrieval is good, and the answers read well. Neither of those facts tells you that the answer came from the retrieved passages. A large language model knows a great deal already, and an answer that is correct from memory looks exactly like an answer that is correct from evidence. This is the same problem as the one in my dissertation, where an agent that always turned the same way scored 50% on mean success while having learned nothing about the instruction cue. Aggregate accuracy could not see it. An intervention could. So the grounding layer perturbs the context and watches what happens to the answer, across six conditions. Remove the passages entirely and the answer should change; if it does not, retrieval was decorative. Substitute the passages retrieved for a different query and the answer should change; if it does not, the generator is ignoring what it was given. Alter the key fact in the passages and the answer should follow the evidence rather than contradicting it. Alongside those sit controls that separate a grounding failure from a different problem, such as sensitivity to passage order.
Scoring grounding
Three measures are reported per condition and per domain: a conditional grounding score, sentence-level faithfulness, and citation support, the last checking that the passage a sentence cites actually contains what the sentence claims. The scores are not averaged across intervention families. A system that holds up under one perturbation and collapses under another is not grounded, and a mean would hide precisely that, in the same way mean success hid the one-sided policy in the dissertation.
Telling a real difference from a coincidence
Twenty-one metrics, six domains and several conditions produce a large number of comparisons, and at that volume some of them will look significant by chance. Ignoring that is how a portfolio project ends up reporting an improvement that does not exist. Proportions are compared with a two-proportion z-test, paired per-query differences with a paired Welch's t-test, and the whole family of comparisons is corrected with Benjamini-Hochberg. Effect sizes are reported as Cohen's h and d, because a difference can be statistically detectable and still too small to care about. Proportions carry Wilson score intervals, which behave sensibly near zero and one where the normal approximation does not, and any comparison with fewer than 30 samples is flagged as such instead of being quietly reported as a result. SciFact claims are checked against the gold qrels. Long evaluation runs checkpoint and resume, so a crash does not silently truncate a run into a smaller and more flattering sample.
What was mine
The retrieval engine was coursework for the MSc Information Retrieval module and a team project. The generation layer, the six-condition grounding evaluation, and the statistical testing infrastructure were added afterwards, on my own, and are the part of this work I would want to be asked about.