Advanced RAG Techniques: Hybrid Search, Re-ranking, and Query Expansion

UpdatedSeptember 24, 2025

Introduction

Retrieval-Augmented Generation (RAG) systems have transformed how AI models access and utilize external knowledge by integrating document retrieval with generative capabilities. To maximize the quality of retrieved documents, advanced techniques such as hybrid search, re-ranking with cross-encoders, and query expansion play pivotal roles. This article explores these strategies in depth, targeting machine learning engineers and AI researchers aiming to enhance RAG systems’ precision and relevance.

Hybrid Search: Combining BM25 and Dense Retrieval

What is Hybrid Search?

Hybrid search synergizes sparse keyword-based retrieval (e.g., BM25) with dense semantic vector search (e.g., FAISS or similar vector indexes). BM25 excels at exact keyword matching by scoring documents based on term frequency and document length, while dense retrieval captures semantic relationships using embeddings generated by models like Sentence Transformers.

How Does Hybrid Search Work?

Typically, hybrid search executes BM25 and dense vector search in parallel or sequentially. BM25 first filters documents by lexical relevance, narrowing the candidate set. Then, dense retrieval refines this set by semantic similarity, ensuring the final results are contextually aligned with the query.

An example fusion method is Reciprocal Rank Fusion (RRF), which combines rankings from both searches by summing the reciprocal ranks of documents, balancing lexical precision and semantic depth.

| BM25 Rank | Dense Rank | RRF Score (1/rank BM25 + 1/rank Dense) |
|—————|—————-|——————————————–|
| A | B | 1/1 + 1/3 = 1.33 |
| B | C | 1/2 + 1/1 = 1.5 |
| C | A | 1/3 + 1/2 = 0.83 |

In this example, Document B ranks highest after fusion, illustrating how hybrid search surfaces documents relevant both lexically and semantically2.

Architecture Diagram of Hybrid Search

text

User Query ├─> BM25 Search (Sparse Retrieval) │ └─> Top-K Candidate Documents (Sparse) │ ├─> Dense Vector Search (Semantic Retrieval) │ └─> Top-K Candidate Documents (Dense) │ └─> Fusion Algorithm (e.g., Reciprocal Rank Fusion) └─> Final Ranked List

Pros and Cons of Hybrid Search

Hybrid search is especially effective in domains requiring both exact term matching and contextual understanding, such as legal research or healthcare1 2 5.

Re-ranking with Cross-Encoders

What is Re-ranking?

Re-ranking is a secondary retrieval step where an initial candidate set of documents is reordered based on deeper semantic analysis. Unlike bi-encoder models used in dense retrieval—which encode queries and documents independently—cross-encoders jointly encode query-document pairs, allowing for fine-grained interaction and better relevance estimation.

How Cross-Encoder Re-ranking Works

1. Initial Retrieval: A hybrid or dense retrieval system returns a shortlist of candidate documents.

2. Cross-Encoder Scoring: Each query-document pair is fed into a cross-encoder (e.g., BERT-based model) that outputs a relevance score considering contextual interactions.

3. Re-ranking: Candidates are reordered based on these scores to improve precision.

Example

Suppose a query “treatment for hypertension” retrieves 100 documents via hybrid search. The cross-encoder re-ranker scores each document with detailed semantic understanding, promoting those that explicitly discuss treatment protocols over loosely related content.

Architecture Diagram of Re-ranking

User Query
└─> Initial Retrieval (Hybrid Search)
└─> Candidate Documents (Top-K)
└─> Cross-Encoder Re-ranking
└─> Final Ranked Documents

Pros and Cons of Re-ranking

Re-ranking is a critical step in high-stakes applications where precision is paramount, such as regulatory compliance or scientific literature search8.

Query Expansion

What is Query Expansion?

Query expansion enhances the original user query by adding semantically related terms or phrases to improve recall and retrieval quality. This can be done via:

– Synonym expansion: Adding synonyms or related terms.

– Pseudo-relevance feedback: Using top retrieved documents to extract expansion terms.

– Neural query expansion: Leveraging language models to generate related queries.

Example of Query Expansion

Original query: “heart attack treatment”

Expanded query: “heart attack treatment OR myocardial infarction therapy OR cardiac arrest care”

This expansion helps retrieve documents that may use different terminology but are contextually relevant.

Implementation in RAG

In RAG, query expansion can be applied before the retrieval step to broaden the search space, allowing BM25 and dense retrieval to access more relevant documents.

Pros and Cons of Query Expansion

Query expansion is valuable in domains with rich synonymy or jargon variation, such as biomedical or technical fields8.

Integrating Advanced RAG Techniques: A Unified Workflow

1. Query Expansion: Enhance the user query with relevant terms.

2. Hybrid Retrieval: Use BM25 to filter and FAISS or similar for dense semantic search.

3. Fusion: Combine results using algorithms like Reciprocal Rank Fusion.

4. Re-ranking: Apply cross-encoder models to reorder top candidates.

5. Generation: Feed the refined documents into the generative model for answer synthesis.

This pipeline balances precision, recall, and semantic understanding, optimizing the retrieval quality for RAG systems.

Learn how Chatnexus.io integrates these advanced RAG workflows—hybrid search, re-ranking with cross-encoders, and query expansion—to deliver state-of-the-art document retrieval and generation solutions tailored for complex, real-world applications.

This comprehensive overview equips ML engineers and AI researchers with the knowledge to implement and optimize advanced retrieval techniques, pushing the boundaries of what RAG systems can achieve.

Optimizing Vector Search Performance for Large Document Collections

Scaling vector search systems to efficiently handle millions of documents requires a deep understanding of indexing algorithms, compression techniques, and tuning parameters that balance accuracy, latency, and resource usage. Developers building large-scale search infrastructures must optimize every aspect of the retrieval pipeline to meet enterprise demands for speed and precision. This article explores the strengths and tradeoffs of three leading vector search frameworks—HNSW, FAISS, and Milvus—while diving into index tuning, vector compression, and search-time tradeoffs, supported by benchmarks and practical insights.

Comparing HNSW, FAISS, and Milvus: Algorithms and Scalability

HNSW (Hierarchical Navigable Small World) is a graph-based approximate nearest neighbor (ANN) algorithm that constructs a multi-layer graph to enable efficient navigation through vector spaces. Each layer contains fewer nodes, allowing fast traversal from general to fine-grained neighborhoods. HNSW is known for its low latency and high recall, making it ideal for in-memory search on datasets with millions of vectors.

FAISS (Facebook AI Similarity Search) offers a versatile suite of ANN algorithms, including IVF (Inverted File), PQ (Product Quantization), and HNSW. FAISS supports GPU acceleration and is widely adopted for billion-scale vector search applications. Its modular design allows developers to tune tradeoffs between speed, memory, and accuracy.

Milvus is an open-source vector database designed for large-scale deployments. It integrates multiple ANN algorithms such as HNSW and IVF, supports distributed indexing and search, and provides hybrid search capabilities combining vector and scalar data. Milvus excels at horizontal scaling and enterprise-grade reliability.

Index Tuning: Parameters That Impact Performance

Optimizing vector search requires tuning algorithm-specific parameters to balance recall, latency, and resource consumption.

HNSW Parameters

– M (max connections per node): Controls graph connectivity. Higher M increases recall but also memory usage and search time.

– efConstruction: Influences index build quality. Larger values improve accuracy but increase indexing time.

– efSearch: Controls search breadth at query time. Higher values improve recall but increase latency.

Typical tuning for million-scale datasets involves setting M between 16 and 48, efConstruction around 200, and efSearch between 50 and 200. These settings balance high recall (\>90%) with sub-10ms latency in many scenarios.

FAISS Parameters

– IVF clusters (nlist): Number of partitions in the inverted file index. More clusters reduce search space but increase indexing overhead.

– nprobe: Number of clusters searched per query. Higher nprobe improves recall but increases latency.

– PQ subquantizers and bits: Determine compression granularity, affecting accuracy and memory footprint.

For billion-scale datasets, nlist is often set to around

(where

N is the number of vectors), and nprobe is tuned between 16 and 64. PQ compression is commonly used to reduce memory usage, with a refinement step rescoring top candidates using original vectors to recover accuracy lost during compression.

Milvus Parameters

Milvus exposes similar tuning options for underlying algorithms, with additional support for distributed index partitioning, replication, and load balancing. This enables scaling to billions of vectors with fault tolerance and high throughput.

Compression Techniques: Balancing Memory and Accuracy

Vector compression is essential to reduce memory and storage costs, enabling larger datasets to fit in RAM or on disk while maintaining search quality.

– Product Quantization (PQ): Splits vectors into subvectors and quantizes each into discrete codes, reducing storage by 4-8x with minimal accuracy loss.

– Binary Quantization (BQ): Converts vectors into compact binary codes for ultra-low memory footprint, trading off some precision.

Azure AI Search supports scalar and binary quantization, reducing storage costs by up to 92.5% without sacrificing search quality, as measured by metrics like Normalized Discounted Cumulative Gain (NDCG)2. These systems often employ oversampling and rescoring strategies, where compressed vectors are used for initial retrieval and original vectors refine the top results, mitigating compression artifacts.

Lossless compression of auxiliary data such as vector IDs and graph edges can reduce index size by up to 30% without impacting accuracy or search speed5. This is critical for billion-scale datasets where auxiliary data can dominate storage.

Search-Time Tradeoffs: Latency, Recall, and Resource Usage

Optimizing vector search involves navigating tradeoffs between latency, recall, and computational resources.

– Latency vs. Recall: Increasing search parameters like efSearch (HNSW) or nprobe (FAISS) improves recall but raises query latency. Developers must tune these based on application SLAs and user experience requirements.

– Compression vs. Speed: Compression reduces memory footprint but adds decompression overhead, slightly increasing latency. Rescoring with original vectors helps balance this tradeoff.

– In-memory vs. On-disk: In-memory indexes offer low latency but are limited by RAM size. Disk-based search with compressed vectors scales better but with higher latency.

For example, HNSW tuned with efSearch=100 can achieve recall \>90% with latency under 10ms on million-scale datasets, while FAISS IVF-PQ can handle billion-scale data with recall around 65-85% at tens of milliseconds latencyBenchmarks and Practical Insights

– Recall and Latency: HNSW is favored for low-latency, high-recall scenarios on million-scale datasets. FAISS IVF-PQ is preferred for very large datasets where memory constraints require compression and partitioning.

– Memory Footprint: PQ compression reduces vector storage by 4-8x, while lossless ID compression can reduce index size by 30% or more, enabling larger datasets in fixed memory budgets.

– Scaling: Milvus’s distributed architecture supports billions of vectors with configurable replication and sharding, balancing throughput and fault tolerance.

A recent study showed that applying lossless ID compression to a billion-scale IVF index reduced storage from 17.8 GB to 12.5 GB without affecting search time or recall 5. Similarly, Azure AI Search’s quantization techniques cut vector storage costs dramatically while maintaining high relevance scores2.

Best Practices for Developers

– Select the indexing algorithm based on dataset size, latency requirements, and hardware constraints.

– Tune parameters such as M, efSearch, nprobe, and cluster count iteratively for your workload.

– Apply vector compression to reduce memory footprint, but use rescoring with original vectors to maintain accuracy.

– Consider distributed solutions like Milvus for workloads exceeding single-machine capacity.

– Monitor latency and recall tradeoffs continuously and adjust parameters as data grows.

– Use lossless compression for auxiliary data to optimize storage without sacrificing performance.

Explore how Chatnexus.io leverages these advanced vector search optimization techniques—including efficient indexing, compression, and parameter tuning—to deliver scalable, high-performance search solutions tailored for enterprise workloads.