
Date: 2026-06-21
Suitable for readers: engineers, product leaders and technical leaders who are building an enterprise knowledge base Q&A, customer service assistant, internal search, research assistant or Agent retrieval layer.
Abstract
The previous article talked about knowledge base intake and chunking: the original document is processed into a retrieveable, referenceable, filterable, auditable unit of evidence.
This article addresses the next question: The evidence unit has been put into the warehouse, how to stabilize the correct evidence in front of the model?
Many RAG systems start by retrieving the top-k vector matches and passing them to a prompt. That works for a demo, but vector similarity can be unreliable for error codes, contract clauses, product models, table fields, and proper names. Keyword matching alone can be too rigid for conceptual questions, paraphrases, and summaries across documents. Even when retrieval finds useful candidates, without reranking the top 5 are often taken up by chunks that look similar but do not support the answer.
The retrieval layer of the production-grade RAG should not be understood as a vector library call, but should be understood as a sorted pipeline:
Query Understanding
-> Metadata Filter
-> Sparse Retrieval + Dense Retrieval
-> Hybrid Fusion
-> Candidate Pool
-> Reranker
-> Diversity and Policy Filters
-> Context Packing
-> Eval and Fix Loop
This paper gives a set of mixed retrieval and reranking methods that can be implemented, including what is the responsibility of Sparse and Dense, how to do RRF fusion, how big the candidate pool is, where is the RERAANKER, and how to deal with the low CONFIDENCE, how to evaluate by query type, and a minimal Python sort pipeline example.
Contents
- 1. Don't understand RAG retrieval as a vector top-k
- 2. What does a production retrieval link look like?
- 3. Sparse Retrieval: Keywords are not backward schemes
- 4. Dense Retrieval: Semantic Recall Solve Rewrite and Concept Match
- 5. Hybrid Fusion: Turn different Retriever into a candidate pool
- 6. Candidate Pool: You must give enough candidates before reranking
- 7. Reranker: Turn "like" into "support answers"
- 8. Query Rewrite and Query Decomposition
- 9. Diversity, Permission, Version and Freshness
- 10. Score Threshold and Low Confidence Policy
- 11. A Minimum Mixed Retrieval Example
- 12. How to Evaluate Whether Mixed Retrieval Is Really Better
- 13. Proposals for production architecture
- 14. Common Myths
- 15. Landing road map
- 16. Pre-release self-check list
- 17. Summary
- References
1. Don't understand RAG retrieval as a vector top-k
Vector search is useful, but it is not a complete retrieval system.
Assuming the user asks:
订单命中 E7782 时是否还能自动退款?
There is a strong keyword signal in this question: E7782. If there is a paragraph in the knowledge base:
错误码 E7782 表示支付渠道返回风险拦截。命中该错误码后,退款申请必须进入人工审核,不得自动退款。
Pure semantic vector retrieval does not necessarily rank it first. Because the vector model may pay more attention to "automatic refunds" and "manual review" of these semantically similar paragraphs, rather than the error code itself. At this time, BM25, field matching, phrase matching and exact matching are very important.
Look at another question:
客户说到账时间太久,客服应该怎么解释跨境退款?
Here, users did not directly say that "cross-border payment orders need to be manually reviewed", "1-5 working days" and "original payment channels", but they are obviously asking these policies semantically. Pure keyword search may be missed, and vector search is more suitable.
The key to producing RAGs is not to choose between keywords and vectors, but to admit that they solve different problems.
| search method | Be good at | Not good at |
|---|---|---|
| Keywords / BM25 / Sparse | Error code, product model, contract number, proper noun, original phrase, field name | Synonymous rewriting, conceptual problems, cross-sentence semantics |
| vector/dense | Synonymous expression, intent matching, concept related, natural language problems | Precision symbols, short strings, rare nouns, latest terms |
| metadata filtering | Permissions, Tenant, Version, Language, Time, Document Type | Determine if the content really answers the question |
| reranker | Fine-grained correlation, whether evidence supports answers, sorting and correcting deviations | Handling full library recalls with higher cost and latency |
So the goal of the retrieval layer is not "take top-k", but: Construct a full, sufficient, interpretable, and auditable set of evidence candidates from the full knowledge base, and then put the evidence that best supports the answer into the context.
2. What does a production retrieval link look like?

A more reliable RAG retrieval link usually contains these stages.
| Stage | Target | Key issue |
|---|---|---|
| Query Understanding | Understand user intentions, entities, time, constraints | Are users asking about rules, facts, processes, comparisons, or summaries? |
| Metadata Filter | Tighten the accessible range first | Is the tenant, permission, language, version, status correct? |
| Sparse Retrieval | Recall exact word face matching | Can error codes, terms, titles, and field names hit? |
| Dense Retrieval | Recall semantic similar content | Can synonymous expressions and concepts be hit? |
| Hybrid Fusion | Merge different retriever results | How do the different score scales merge? How to repeat chunks? |
| Candidate Pool | Give Reranker enough candidates | Is the recall pool big enough but not out of control? |
| Reranker | Re-examination of whether candidates really answer questions | Is there a direct support relationship between the evidence and the question? |
| Diversity Filter | Avoid the same-origin repeated extrusion context | Do I need the source cap, MMR, and adjacent chunk completions? |
| Context Packing | Organize evidence to generative models | How to deal with sequence, reference, budget, and conflict evidence? |
| Eval Loop | Discover failure patterns and fix them | What kind of problems are recalled, sorted, and referenced? |
The two facts that are most easily overlooked here are the two facts.
First, metadata filtering is essential. In enterprise RAG, enforce access permissions and version constraints before retrieval. Once restricted or inapplicable evidence has reached the model, asking it to disregard that evidence is too late.
Second, Reranker is not a recaller. Reranker is better at making a good squash in dozens of candidates, not suitable for directly facing millions of corpus. If the correct evidence is not put into the candidate pool during the recall phase, Reranker cannot be saved no matter how strong it is.
3. Sparse Retrieval: Keywords are not backward schemes
Sparse retrieval usually refers to keyword search methods such as BM25, inverted index, phrase matching, field weighting, and term extension. It doesn't look as good as vector retrieval "smart", but it is the most reliable base in many production problems.
3.1 When must be retained keyword search
These queries strongly depend on word face matches:
- Error code:
E7782、HTTP 429、ORA-00001 - Product model:
XG-2400、Pro Max、SKU-9812 - Field name:
refund_status、effective_date - Contract terms:
第 6.2 条、force majeure - Regulation number:
GDPR Article 15、SOX 404 - Internal project code:
Aurora、Bridge-v2 - Name, place name, organization name, report title
The semantic space of these words may not be stable. An error code may have no natural language meaning, and a SKU may just be a string of characters. Vector retrieval is difficult to guarantee precise recall if there is not enough training context.
3.2 Keyword search not only use the text
The previous article emphasizes chunk metadata. Here it will directly affect the quality of keyword recall.
It is recommended to include these fields into the Sparse Index and set different weights:
| Field | Effect | Recommend |
|---|---|---|
source_title |
Document-level themes | high weight |
hierarchy_path |
Chapter Path | high weight |
text |
evidence text | standard weight |
keywords |
Manual or automatic extraction of terms | mid-high weight |
product_line |
product line | for filtering or weighting |
version |
Version | For filtering, it is not recommended to rely on sorting alone |
status |
active/deprecated | for filtering |
If you only put the chunk body into the index, many short chunks will lose the context. For example, the body only has:
不得自动退款。
But its header path is:
退款政策 > 风险拦截 > E7782 处理
The title path is more critical than the body when retrieving.
3.3 Engineering details of Sparse retrieval
It is recommended to support at least:
- Exact phrase matching: suitable for error codes, clause numbers, titles;
- Field weighting: the title and chapter path weight is greater than the body;
- Analyzer strategy: Chinese tokenization, English case, symbol reserved;
- Synonyms: business alias, old name, new name;
- Stop word control: do not throw away all high-frequency words such as "refunds" and "orders";
- Query Boost: Increase the sparse weight when the error code, SKU, and field names are identified.
Keyword retrieval is not a "traditional scheme", it is RAG's respect for the real world symbology.
4. Dense Retrieval: Semantic Recall Solve Rewrite and Concept Match
Dense Retrieval uses Embedding to map query and chunks to vector space, and then recall them according to similarity. Its value lies in the case where the user does not repeat the document verbatim.
For example, the knowledge base says:
跨境支付订单需进入人工审核。审核完成后,退款通常在 1-5 个工作日内退回原支付渠道。
Users may ask:
海外订单退款为什么不能马上到账?
There is no obvious error code or clause number at this time, and Dense Retrieval is easier to recall the correct evidence.
4.1 Dense Retrieve Dependency Chunk Quality
If the chunk is too long, embedding will mix multiple themes; if the chunk is too short, the semantics is incomplete. The quality of the vector recall is not only determined by the Embedding Model, but also by these factors:
- Whether the chunk is full of semantics;
- Whether the title path enters the index text;
- whether the table is converted into understandable text;
- Whether the old version is filtered;
- Whether the same document is repeatedly stored in the database;
- Whether the user query needs to be rewritten or dismantled.
This is why the ingestion and chunking are discussed first. Without a good unit of evidence, the search algorithm can only be sorted in noise.
4.2 The starting point of the parameter searched by Dense
You can start with the following configuration, and then use the evaluation set to adjust.
| Scenes | dense top-n | Explanation |
|---|---|---|
| Small FAQ / Fewer documents | 20-40 | Candidates are enough |
| Medium Enterprise Knowledge Base | 50-100 | Leave space for fusion and reranking |
| Multi-tenant / multi-product line | 30-80 per filter range | Filter first and then recall |
| Long Documentation Research Assistant | 80-150 | Need for stronger deduplication and diversity control |
Don't start top_k=5 The model is fed directly as a search result. Top-k is the final number of contexts and should not be equal to the number of recalls. The recall phase should be wider, and the refined and contextual stages should be tightened.
5. Hybrid Fusion: Turn different Retriever into a candidate pool
The basic idea of Hybrid Search is: run sparse and dense on the same query at the same time, and then merge the two result lists into a candidate pool.
The difficulty is that the BM25 score and vector similarity are not the same scale. BM25 may be 12.7, Cosine Similarity may be 0.83. It is usually meaningless to add directly.
5.1 Recommended to start with RRF
A practical starting point is the Reciprocal Rank Fusion, or RRF for short. It does not directly compare the original score, but compares the ranking position.
Intuition is simple:
- At the same time, the top-ranked chunks in sparse and Dense should be higher;
- Chunks that are only ranked very high in one channel should also be reserved;
- The lower the ranking, the smaller the contribution.
Common form:
score(doc) = sum( weight_i / (k + rank_i(doc)) )
Among rank_i is the ranking in some retriever, k It is a smoothing constant, commonly used around 60. Weights can be adjusted by query type.
5.2 Weights should not be fixed
In the same system, different Query should have different policies.
| Query type | sparse weight | DENSE weights | Reason |
|---|---|---|---|
| Error code / SKU / clause number | High | In | Precise Symbols First |
| FAQ Natural Language Questions | In | High | User expression is often rewritten |
| title or document name search | High | Low medium | The title is strong |
| Concept Explanation / Principle Summary | In | High | Semantic correlation is more important |
| Multi-condition policy issues | Medium high | Medium high | Both need |
The OpenAI Retrieval document also provides mixed Retrieval related Ranking configurations: you can ranking_options.hybrid_search Adjust the weight of the embedding semantic match and the sparse text match, and pass score_threshold Controls the minimum score for the result returned. The specific product parameters will change, and the design should be regarded as a retrieval strategy that can be evaluated and rolled back, rather than a one-time write to death.
5.3 Before and after the fusion, we must remove the weight
The same evidence may appear in multiple forms:
- adjacent chunks of the same source;
- Different versions of the same document;
- FAQ and policy body duplication;
- Multilingual translation content is repeated;
- The summary page and the original page are repeated.
After fusion, you must press chunk_id、content_hash、source_id + locator Wait for the field to be heavy. Otherwise, the first 10 seem to be a lot, but they are all duplicate versions of the same paragraph.
6. Candidate Pool: You must give enough candidates before reranking
The premise of Reranker is that the correct evidence is already in the candidate pool. The candidate pool is too small and is the root cause of many RAG sort failures.
A safe starting point:
sparse top 80
dense top 80
metadata filter before retrieval
RRF fusion to top 80
dedupe and source cap to top 60
rerank top 60
context pack top 6-12 chunks
Numbers are not standard answers, but embody a principle: The recall is wide, the reranking is narrow, and the context is narrower.
6.1 Why can't just rerank top 10
If the initial platoon of Sparse and Dense is not perfect, the correct evidence may be ranked 18th, 33rd or 47th. Only rerank top 10 will make Reranker not see the correct answer.
At least 30-80 Reranks are recommended when delays are allowed. For high-value, low-frequency, and long answer questions, you can rerank more than 100; for customer service real-time scenarios, you can use cache, light reranker or layered rerank to control the delay.
6.2 Candidate pools to reserve interpreted fields
Each candidate must not only have text, but also keep sorting evidence:
{
"chunk_id": "refund_policy#sec_4#chunk_02",
"source_id": "refund_policy_2026_03",
"sparse_rank": 3,
"dense_rank": 18,
"fusion_score": 0.0271,
"rerank_score": 0.84,
"matched_terms": ["E7782", "自动退款"],
"metadata": {
"version": "2026-03",
"status": "active",
"access_scope": ["customer_service"]
}
}
These fields are the lifeline for debugging and evaluation. When there is a wrong answer on the line, if there is only the final prompt in the log, it is difficult to know whether it is not recalled or missed, or that there is no evidence for the generative model.
7. Reranker: Turn "like" into "support answers"
The initial recall judged "whether this chunk and query are similar". Reranker should go one step further: judge "whether this chunk can directly support answering this query".
For example, the user asks:
命中 E7782 后能否自动退款?
Candidate A:
自动退款适用于低风险订单,系统会在审核通过后按原支付渠道退款。
Candidate B:
错误码 E7782 表示支付渠道返回风险拦截。命中该错误码后,退款申请必须进入人工审核,不得自动退款。
Both A and Query contain "automatic refunds", which may be very similar. But B is the evidence that truly supports the answer. The value of Reranker is here.
7.1 Common Reranker Types
| Types of | Thinking | Advantage | Cost |
|---|---|---|---|
| Cross-encoder reranker | Enter Query and Chunk at the same time, and output the correlation score | Strong precision, suitable for fine discharge | Every candidate runs a model |
| LLM reranker | Let LLM determine whether candidates support answers | interpretable, can handle complex conditions | High cost and latency, to prevent format drift |
| Late interaction | For example, the Colbert-style token-level interaction | Recall and fine-grained matching | System implementation is more complex |
| Rules Reranker | Weighted with rules such as entity, time, version, title, etc. | Inexpensive, stable and controllable | Can't cover complex semantics |
Common combinations in engineering are:
RRF fusion -> rule boosts -> neural reranker -> final policy filters
The rules are not to replace the model, but to dispose of the deterministic constraints first. For example:
- Active version is preferred;
- The user's product line is preferred;
- The title is precisely hit weighted;
- The old version is downgraded or filtered;
- Up to 2-3 items are reserved for the same source;
- Form evidence must be heading with row and column.
7.2 The output of Reranker should not only save one score
If using LLM or interpretable reranker, it is recommended to output structured fields:
{
"chunk_id": "refund_policy#sec_4#chunk_02",
"relevance": 0.92,
"support": "direct",
"reason": "The chunk explicitly states that E7782 requires manual review and automatic refund is not allowed.",
"missing": []
}
Among support Can be divided into:
direct: answer directly;partial: answer part;background: only the background;conflicting: conflict with other evidence;irrelevant: It doesn't matter.
The final entry into the context should be the priority direct and high quality partial, not all "look related" materials.
8. Query Rewrite and Query Decomposition
User Query is often not the most suitable form for retrieval. Query rewrite can be done before retrieval, but the boundaries are controlled.
8.1 What is Query Rewrite suitable for
Suitable for:
- The verbalization problem is rewritten into standard terms;
- Omit contextual completion;
- Chinese and English terminology mapping;
- spelling errors;
- Press the long problem into a search keyword;
- Generate synonymous query to do multiple recalls.
For example:
用户问题:海外订单退款为什么慢?
检索 query 1:跨境支付订单 退款 人工审核 到账时间
检索 query 2:海外订单 退款 1-5 个工作日 原支付渠道
检索 query 3:跨境退款 审核完成后多久到账
The OpenAI Retrieval documentation is provided rewrite_query A class of capabilities that is used to rewrite the query before retrieving. Regardless of the use of hosting ability or self-built ability, you must record the rewrite results in the log, otherwise it is difficult to explain why some evidence was recalled.
8.2 Query Decomposition is suitable for complex problems
Complex questions often contain multiple sub-questions:
企业客户取消年付套餐时,退款金额怎么计算,是否会影响已开票金额?
This may be split into:
- The annual payment package cancellation rules;
- The amount of refund is calculated;
- invoices and financial processing;
- Whether corporate customers have special terms.
Each subquery is recalled individually, re-integrated and reranked, usually more stable than the original problem retrieval.
But decomposition has a risk: it introduces sub-problems of model speculation. It is recommended to record the dismantling result with the original query, and ask the evidence to still support the original problem during the Reranker stage, rather than only a rewritten problem.
9. Diversity, Permission, Version and Freshness
Sorting is not just about dependencies. The RAG's evidence context also satisfies engineering constraints.
9.1 Permissions must be filtered before retrieval
If the user does not have access to a document, it is best to filter out the retrieval layer, rather than let the model ignore it after recalling it.
Essential fields:
tenant_idaccess_scopeowner_teamclassificationstatuseffective_atexpires_at
The hosting Retrieval capability also usually supports metadata filtering. For example, OpenAI File Search and Retrieval documents both emphasize that filtering can be based on File or attribute metadata. The self-built system should also design the metadata filter to retrieve the first-class parameters of the API.
9.2 Version and status to be hard filtered
If the document has active/deprecated/draft status, only Active is retrieved by default. Don't leave the "new version higher score" to the sorting model naturally learn.
Common strategies:
| Status | Default processing |
|---|---|
| active | Recallable |
| deprecated | Default filtering; only recalled when the user asked the historical version explicitly |
| draft | Recall only to internal authorized users |
| archived | Default filter |
9.3 Diversity control avoids the context of the same source
If the top 10 is all from the adjacent chunks of the same long document, the model may not see another key source. Can be set:
- Apiece
source_idUp to 2-3 chunks are reserved; - Consolidation or completion of adjacent chunks of the same chapter;
- Use the MMR class policy to take into account the correlation and difference;
- At least one counter-example is retained for conflict evidence;
- Preserve the full rank context for the form evidence.
Diversity is not for "equal distribution", but for the context to cover the different evidence roles needed for the answer.
10. Score Threshold and Low Confidence Policy
Many teams only care about "what has been recalled", not "when should not answer".
In the actual product, the low Confidence problem must have a strategy:
- No candidates exceed the threshold: abstention or follow-up;
- Candidate scores are close but conflict with each other: the conflict and request confirmation;
- There is only background evidence, there is no direct evidence: give a conservative answer;
- You need user identity, time or product line to judge:
- Hit the old version and the new version is missing: the prompt cannot confirm the latest rule.
Thresholds don't just use one global number. Different query types can have different thresholds.
| Query type | Threshold Policy |
|---|---|
| Error code / clause number | Requires precision hits or strong support |
| Conceptual explanation | Multiple PARTIAL evidence can be accepted |
| Compliance / Legal | The threshold is higher, and it is not strong to answer in conflict |
| Customer service | Allows reference to generate expressions after a policy is allowed |
| data fact | Must hit clear source and time |
A mature RAG system does not always answer, but knows when to answer, when to ask, and when abstention.
11. A Minimum Mixed Retrieval Example
Below is a minimal sorting framework that does not depend on specific databases. in the real system, sparse_search can be from Elasticsearch/OpenSearch/Vespa/PostgreSQL FTS, dense_search Can be retrieved from vector library or hosting, rerank You can come from Cross-Encoder, LLM, or self-built models.
from dataclasses import dataclass, field
from typing import Dict, Iterable, List, Optional
@dataclass
class Candidate:
chunk_id: str
source_id: str
text: str
metadata: Dict[str, str]
sparse_rank: Optional[int] = None
dense_rank: Optional[int] = None
fusion_score: float = 0.0
rerank_score: float = 0.0
reasons: List[str] = field(default_factory=list)
def rrf_fuse(
result_lists: Dict[str, List[Candidate]],
weights: Dict[str, float],
k: int = 60,
) -> List[Candidate]:
merged: Dict[str, Candidate] = {}
for channel, results in result_lists.items():
weight = weights.get(channel, 1.0)
for rank, item in enumerate(results, start=1):
if item.chunk_id not in merged:
merged[item.chunk_id] = item
candidate = merged[item.chunk_id]
candidate.fusion_score += weight / (k + rank)
if channel == "sparse":
candidate.sparse_rank = rank
elif channel == "dense":
candidate.dense_rank = rank
candidate.reasons.append(f"{channel}:rank={rank}")
return sorted(merged.values(), key=lambda x: x.fusion_score, reverse=True)
def apply_policy_filters(
candidates: Iterable[Candidate],
user_scope: str,
max_per_source: int = 3,
) -> List[Candidate]:
kept: List[Candidate] = []
source_counts: Dict[str, int] = {}
for item in candidates:
if item.metadata.get("status") != "active":
continue
if user_scope not in item.metadata.get("access_scope", "").split(","):
continue
if source_counts.get(item.source_id, 0) >= max_per_source:
continue
source_counts[item.source_id] = source_counts.get(item.source_id, 0) + 1
kept.append(item)
return kept
def rerank(query: str, candidates: List[Candidate]) -> List[Candidate]:
# Replace this stub with a cross-encoder, LLM reranker, or managed reranking API.
# Keep the interface stable so retrieval experiments are easy to compare.
query_terms = set(query.lower().split())
for item in candidates:
text_terms = set(item.text.lower().split())
overlap = len(query_terms & text_terms)
item.rerank_score = 0.7 * item.fusion_score + 0.3 * overlap
return sorted(candidates, key=lambda x: x.rerank_score, reverse=True)
def retrieve_for_rag(query: str, user_scope: str) -> List[Candidate]:
sparse = sparse_search(query, top_n=80)
dense = dense_search(query, top_n=80)
weights = choose_weights(query)
fused = rrf_fuse(
{"sparse": sparse, "dense": dense},
weights=weights,
k=60,
)
filtered = apply_policy_filters(fused[:100], user_scope=user_scope)
ranked = rerank(query, filtered[:60])
return ranked[:10]
This example deliberately preserves several interfaces:
choose_weights(query): Adjust the Sparse / Dense weights by query type;apply_policy_filters(...): Put permissions, versions, and source caps in the sorting link;rerank(...): Decouple the ability and recall ability;reasons: Record why candidates are recalled, which is easy to debug.
Before the real system goes online, you should also put Query, Rewrite Query, Filter, Sparse Rank, Dense Rank, Fusion Score, ReRank for each request. Score and final references are written to retrieval logs.
12. How to Evaluate Whether Mixed Retrieval Is Really Better
As mentioned above, the RAG evaluation is focused on the search and sorting itself.
Don't just look at "feel the answer is better". To split the search link apart and compare:
| Experiment group | Purpose |
|---|---|
| sparse only | keyword baseline |
| dense only | vector baseline |
| sparse + dense fusion | Verify that the mixed recall is elevated Recall |
| fusion + reranker | Verify that the refined discharge improves TOP-K |
| fusion + reranker + diversity | Verify that the context quality is improved |
| rewrite + fusion + reranker | Verify that Query Rewrite is profitable |
12.1 Indicator layering
| Metric | What to see | typical use |
|---|---|---|
| Recall@50 | Whether the correct evidence enters the candidate pool | Diagnostics recall issues |
| Recall@10 | Is the correct evidence ahead? | Diagnostics and fusion |
| MRR | Which is the first correct evidence? | Q&A Evidence Quality |
| nDCG@k | Sort of multiple pieces of evidence quality | multi-evidence answer |
| Citation Support Rate | Whether the final reference supports the answer | Connection retrieval and generation |
| Empty / Abstain Rate | System abstention Frequency | Low Confidence policy |
| Latency p95 | User experience | Online SLA |
| Cost per Query | Inference and Retrieval Costs | Scale Cost |
12.2 Must be divided by query type
The total average will mask the problem. At least cut by these types:
- Precise and true body: error code, SKU, clause number;
- Policy rules: can, whether it is allowed, how to calculate;
- Process class: what to do first, what to do later;
- Comparison class: what is the difference between A and B;
- Summary class: summarize a document or a topic;
- Multi-hop class: need a combination of multiple sources;
- Low Confidence class: The knowledge base has no answers or insufficient permissions.
A system may increase the overall Recall@10 by 3%, but the error code class drops by 20%. If you don't divide the buckets, you will mistakenly think that the optimization is successful.
12.3 Evaluation samples come from online failure
The initial Gold Set can be constructed manually, but the really valuable sample comes from online failure:
- The user asks "that's not what it means";
- Manual customer service rewrites the answer;
- The user clicked "No Help";
- There is no supported answer after the reference is opened by the user;
- Reranker's high score candidate is manually judged as irrelevant;
- The model answers "cannot confirm" but the knowledge base actually has the answer.
Add the failed samples to the evaluation set every week, and then compare the search strategy to form a real sorting closed loop.
13. Proposals for production architecture
A maintainable RAG retrieval layer can be split into these modules.
Client / Agent
-> Retrieval API
-> Query Analyzer
-> Metadata Policy Service
-> Sparse Index
-> Vector Index
-> Fusion Service
-> Reranker Service
-> Context Packer
-> Retrieval Logger
-> Generation API
-> Evaluation Pipeline
13.1 Retrieval API to be stable
Don't let business code directly spell multiple database calls. It is recommended to provide a stable interface:
{
"query": "命中 E7782 后能否自动退款?",
"user": {
"tenant_id": "global",
"access_scope": ["customer_service"]
},
"filters": {
"status": "active",
"language": "zh-CN"
},
"strategy": "hybrid_rerank_v3",
"max_context_chunks": 8
}
Returns:
{
"strategy": "hybrid_rerank_v3",
"rewritten_queries": [
"E7782 自动退款 人工审核",
"错误码 E7782 退款处理规则"
],
"contexts": [
{
"chunk_id": "refund_policy#sec_4#chunk_02",
"source_id": "refund_policy_2026_03",
"text": "错误码 E7782 表示支付渠道返回风险拦截。命中该错误码后,退款申请必须进入人工审核,不得自动退款。",
"score": 0.92,
"support": "direct",
"citation": {
"title": "退款政策 2026-03 版",
"section": "4.2",
"page": 4
}
}
]
}
In this way, the generation layer does not need to know whether the underlying layer is Elasticsearch, the vector library or the managed file search. The retrieval strategy can be iterated and the call contract remains stable.
13.2 Policies should be versioned
There should be a version number for each retrieval policy adjustment:
dense_only_v1hybrid_rrf_v1hybrid_rrf_rerank_v2hybrid_rrf_rerank_rewrite_v3
Both logs and reviews record the policy version. Otherwise, when the performance on the line becomes good or bad, you cannot locate which weight, filter, or chunk changes are caused.
13.3 Context packaging is part of the sorting
It's not just the top-k text that is put into prompt at the end. Context packing to handle:
- Evidence sequence: first direct evidence, then background evidence;
- Reference format: each piece of evidence is source, page, section;
- Neighborhood completion: with front and rear chunks if necessary;
- Evidence of conflict: clearly marked with different versions or different conditions;
- Token Budget: Long tables and long paragraphs should be compressed but cannot lose key fields;
- Deduplication: Repeated evidence do not waste contextual windows.
The endpoint of the retrieval sort is not a list, but a package of evidence that can be reliably used by the generated model.
14. Common Myths
14.1 Misunderstanding 1: As long as the Embedding model is stronger, no keywords are needed
No matter how strong embedding is, it should not undertake all precise matching tasks. Error codes, contract numbers, field names, and clause numbers are naturally suitable for Sparse retrieval.
14.2 Misunderstanding 2: Add the RAW BM25 score and the cosine score directly
Different Retriever score scales are different, and direct addition will make a side dominate and sort. First use the ranking-based fusion, or do verified score normalization.
14.3 Misunderstanding 3: Recall TOP 5 Directly to the model
Top 5 is suitable for the final context, not for recall. The recall should be wide, the reranking should be accurate, and the context should be refined.
14.4 Misunderstanding 4: Reranker has too little input
Reranker only looks at the top 10, and it is easy to miss the correct evidence of 20-50. Broaden the candidate pool first, and then let reranker work.
14.5 Misunderstanding 5: Permission and version are reminded in prompt
This creates a safety and consistency risk. Permissions, tenants, versions, and states should be filtered at the retrieval layer.
14.6 Myth 6: Only evaluate the final answer, not the search process
The final answer is wrong, the reason may be that there is no recall, troubleshooting, evidence conflict, and context packing errors, or the generative model may not comply with the evidence. Just looking at the answer will blur the repair direction.
14.7 Myth 7: All Query with the same set of weights
Error code queries and concept interpreted queries should not use the same set of sparse/dense weights. At least simply route by query type.
15. Landing road map
Week 1: Establish an observable baseline
- Access two recalls of Sparse and Dense;
- Each candidate record Sparse Rank, Dense Rank, Source, Version, Access Scope;
- Establish 100-300 artificial Gold Query;
- Run SPARSE ONLY, DENSE ONLY, HYBID 3 groups of evaluations.
Week 2: Add Fusion and Filter
- Implement RRF or hosting Hybrid Ranking;
- Join Metadata Filter;
- Add source cap and deduplication;
- Adjust Sparse / Dense weights by query type;
- Output Recall@50, Recall@10, MRR, nDCG.
Week 3: Join Reranker
- Select Cross-Encoder, LLM reranker, or Managed ReRank capabilities;
- Rerank 30-80 candidates;
- Record Rerank Score and Support types;
- Compare Hybrid with Hybrid + Rerank.
Week 4: Open up contexts and references
- Package the evidence after rerank into a structured context;
- Access reference and attribution verification;
- Design and follow up on low Confidence questions;
- Create an on-line failure sample reflow process.
Week 5: Optimize latency and costs
- Cache popular Query and Query Rewrite;
- Skip heavy reranker for simple query;
- Use layered rerank;
- Control the token budget for each source and each answer;
- Set different SLAs by business value.
16. Pre-release self-check list
At least check these questions before going online.
Recall
- Both Sparse and Dense can run separately;
- Error code, SKU, and clause number can be stably recalled by keywords;
- Natural language rewriting can be recalled by vectors;
- The correct evidence is stable in Recall@50;
- The chunk header path enters the index text.
Fuse
- There is no direct addition to different retriever raw scores;
- Use RRF or verified normalization;
- The query type affects the sparse/dense weights;
- After fusion, press chunk_id, source, and hash to remove the weight;
- Record which channels each candidate comes from.
rearrange
- Reranker sees at least 30-80 candidates;
- Rerank Score and support type write logs;
- Evaluate the difference between Hybrid and Hybrid + Rerank;
- The low score candidates will not enter the final context;
- Higher risk scenarios have stricter thresholds.
Permissions and Versions
- Tenant, Access Scope, Status, and Version are filtered at the search layer;
- The deprecated document is not recalled by default;
- When the user clearly asks the historical version, the version filter is released;
- After the document permission is changed, the index can be updated in time;
- Logs are not leaked without permission to access content.
context
- Each context has a citation;
- Neighboring chunk completions will not introduce irrelevant content;
- The same source will not squeeze the entire context;
- Conflict evidence can be marked;
- Generation models can see enough direct evidence.
Evaluation
- Divide the bucket by query type;
- At the same time, see Recall@K, MRR, nDCG, reference support rate;
- Record LATENCY P50/P95 and cost;
- The online failure sample can be reflowed to the evaluation set;
- The search strategy has a version number and can be rolled back.
17. Summary
RAG's retrieval layer is not a vector library top-k parameter, but a sorted closed loop.
Sparse Retrieval is responsible for precise symbols and lexical evidence, Dense Retrieval is responsible for semantic rewriting and concept matching, and Hybrid Fusion turns multiple channels into candidate pools, Reranker Then determine which evidence really supports the user's problem. Metadata Filter, version control, permissions, freshness, diversity and context packaging are also part of the sorting quality.
A practical principle is:
召回要宽,重排要准,上下文要精,评测要分桶。
If the system only adjusts the parameters on the "Vector Top-K", the problem will occur repeatedly: the recall cannot be recalled, the troubleshooting, the reference is not supported, the old version is mixed, and the low Confidence strong answer. After the retrieval layer is engineered, the RAG can move from the demo to a maintainable production system.
Next article can continue to write: How to do the low Confidence and abstention strategy of the RAG: when to answer, ask and transfer to labor.
References
- OpenAI Developers, Retrieval guide, checked on 2026-06-21.
- OpenAI Developers, File search guide, checked on 2026-06-21.
- Nandan Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021.
- Omar Khattab and Matei Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, 2020.
- Thibault Formal et al., SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking, 2021.
- Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin, Document Ranking with a Pretrained Sequence-to-Sequence Model, 2020.
- Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, 2009.