
Date: 2026-06-20
Suitable for readers: Enterprise knowledge base, document Q&A, customer service assistant, compliance quiz, research assistant or engineer, product leader and technical leader of an internal Agent.
Abstract
The citation and attribution of RAG is mentioned earlier: the key conclusions in the answer must be traced back to the evidence.
This article goes back to the lower level question: Where do these evidence units come from?
Many RAG projects run into trouble during ingestion, before model strength or the choice of vector database comes into play. PDF headers and footers get mixed into the body text, table cells lose their row and column associations, obsolete documents remain active, and access permissions never reach the metadata. Overlong chunks make retrieval imprecise, while very short chunks lose their context. Tuning the prompt, reranker, or citation format can do only so much when the underlying evidence is damaged.
So knowledge base ingestion is not "uploading files to a vector library". It is a production line: Inventory of sources, parsing documents, normalized texts, reserved structures, segmented evidence, metadata, deduplication and version management, writing indexes, running retrieval evaluation, continuous reflow failure samples.
This article gives a set of ingest and chunking methods, including the evidence unit JSONL contract, chunk policy selection, parameter recommendation, table and code processing, permission and version fields, minimum structured chunker examples, and a self-check list before going online.
Contents
- 1. Why do you determine the upper limit of RAG
- 2. The 9 stages of the ingestion line
- 3. What should the unit of evidence look like?
- 4. Chunking is not a fixed length cut text
- 5. How to choose common chunk strategies
- 6. How to set the Chunk size and overlap
- 7. How to deal with different document types
- 8. Metadata is half of the retrieval quality
- 9. Deduplication, version and incremental updates
- 10. A minimal structured chunker example
- 11. How to evaluate whether chunking works
- 12. Production landing architecture
- 13. Common Mistakes
- 14. Landing road map
- 15. Pre-release self-check list
- 16. Summary
- References
1. Why do you determine the upper limit of RAG
The second half of the RAG looks more "smart": retrieval, reranking, generation, reference check. But what really determines the upper limit of the system is often the first ingestion.
If the ingestion phase handles the document like this:
退款政策
页脚:内部资料,仅供参考
表 2
人工审核 跨境支付 大额退款
1-5 个工作日
页眉:退款政策
It is difficult to answer the model stably after:
跨境订单是否需要人工审核?退款多久到账?
Because the original relationship has been lost. The rows and columns of the table, title level, version, page number, permission, and update time have not entered the Evidence unit.
The poor quality of RAG intake will bring six types of problems.
| Problem | Which performed | root cause |
|---|---|---|
| Can't be recalled | The user is right, but the correct content is not in the top-k | Documents are not stored in the warehouse, broken, and the title is lost |
| Recall too pan | Top-k are all relevant but not precise | chunk is too long, noise, metadata is missing |
| The answer is out of context | Only one sentence is quoted, and exception conditions are missed | chunk is too short, context broken |
| Old version overrides new version | Answer with expired policy | Release field and deprecated status is missing |
| permission leak | Recall users do not have access to content | Access Scope is not in search filter |
| References cannot be checked | The user can't find the original text when the user clicks on the source | Locator, page, span, source hash is missing |
Therefore, the goal of the ingestion pipeline is not to "generate embedding", but to generate retrieve, cisable, filterable, auditable units of evidence.
2. The 9 stages of the ingestion line
A reliable knowledge base ingestion pipeline can be split into 9 stages.

| Stage | Target | Product |
|---|---|---|
| 1. Source Inventory | Inventory of knowledge sources | source registry |
| 2. Permission Mapping | Establish permissions and tenant boundaries | access policy |
| 3. Parsing | Extract content from PDF/HTML/Docx/tables | parsed blocks |
| 4. Normalization | Clean headers, footers, garbled characters, duplicate blanks | normalized text |
| 5. Structure Recovery | Keep titles, page numbers, tables, lists, code blocks | structured document |
| 6. Segmentation | Cut into retrievable unit of evidence | evidence chunks |
| 7. Metadata Enrichment | Add version, permissions, source, hash, locator | chunk metadata |
| 8. Indexing | Embedding, sparse index, vector library write | vector/search index |
| 9. Evaluation Loop | Check recall and reference quality with Gold Set | eval report |
Many teams skip the first 5 steps and drop the document directly into the default chunker. This works in demo, but you will soon encounter boundaries in the enterprise knowledge base:
- The two columns of the PDF are misread;
- Tables are flattened into meaningless texts;
- Header and footer repeated pollution embedding;
- Multiple versions of the same policy are hit at the same time;
- Separation of document title and paragraph content;
- Permissions can only alert the model after answering, not filter before retrieval.
The more serious the intake phase, the easier the subsequent search and attribution is.
3. What should the unit of evidence look like?
In the previous article, I mentioned citation and attribution Evidence Span. This article expands it into an indexable evidence_chunk.
It is recommended that each chunk contain at least these fields:
{
"chunk_id": "refund_policy_2026_03#sec_4#chunk_02",
"source_id": "refund_policy_2026_03",
"source_title": "退款政策 2026-03 版",
"source_type": "policy_doc",
"version": "2026-03",
"status": "active",
"updated_at": "2026-03-18",
"owner": "finance_ops",
"tenant_id": "global",
"access_scope": ["customer_service", "finance"],
"language": "zh-CN",
"hierarchy_path": ["退款政策", "4. 跨境退款", "4.2 人工审核"],
"locator": {
"page": 4,
"section": "4.2",
"start_char": 1832,
"end_char": 2150
},
"text": "跨境支付订单需进入人工审核。审核完成后,退款通常在 1-5 个工作日内退回原支付渠道。",
"context_prefix": "退款政策 > 跨境退款 > 人工审核",
"content_hash": "sha256:...",
"source_hash": "sha256:...",
"ingested_at": "2026-06-20T22:18:24+08:00"
}
Here are a few key points.
3.1 text Not all
Vector indexes may be mainly used text, but the system cannot only save text. No source_id、version、access_scope、locator, will not be able to do permission filtering, version control and reference highlighting in the future.
3.2 context_prefix Very important
Many chunks themselves have only one sentence:
需进入人工审核。
Its semantics are weak without the title path. You can add light context to the index text:
退款政策 > 跨境退款 > 人工审核
需进入人工审核。
This is not the original text for users to read, but helps to retrieve the understanding context. The presentation should still go back to the original SPAN.
3.3 content_hash and source_hash Used to do incremental updates
The knowledge base will change. Without Hash, it is difficult to judge:
- whether the document has really changed;
- Which chunks need to recalculate embedding;
- Whether the old chunk should be abolished;
- Which version did the online accident use?
4. Chunking is not a fixed length cut text
The most common mistake is to understand chunking as:
每 800 tokens 切一段,重叠 200 tokens。
Fixed length segmentation is valuable, especially for plain text and fast baselines. But it's not the whole production RAG.
A good chunk should meet five conditions:
| Condition | Explanation |
|---|---|
| semantic intact | A chunk can express a relatively complete fact or step |
| Recallable | Words or semantics in user questions can hit it |
| Can be quoted | Users can understand what it supports after clicking |
| can be filtered | Metadata with permissions, versions, sources and times |
| Combine | Multiple chunks can form complete answers without conflicting with each other |
If the chunk is too large, the search will be general:
整章退款政策都被召回,但答案只需要跨境退款一条规则。
If the chunk is too small, the evidence will be broken:
chunk A: 跨境支付订单需进入人工审核。
chunk B: 审核完成后按原支付渠道退款。
chunk C: 退款通常在 1-5 个工作日内完成。
The model may only see A, and miss B and C.
So the essence of chunking is evidence modeling, not string slices.
5. How to choose common chunk strategies
5.1 Fixed token chunk
Divide by token number and set overlap.
Suitable for:
- fast baseline;
- plain text in simple structure;
- a large number of FAQs;
- Internal searches with low reference granularity requirements.
The advantage is that it is simple and stable. The disadvantage is that it is easy to cut off the title, table, steps and exceptions.
5.2 Structured chunks
Divide by title, paragraph, list, table, code block.
Suitable for:
- policy document;
- product manual;
- contract;
- API documentation;
- research report;
- Standard operating process.
This is the most commonly used production strategy for enterprise RAG. It keeps the document structure:
H1: 退款政策
H2: 跨境退款
H3: 人工审核
Paragraph: 跨境支付订单需进入人工审核...
5.3 Semantics chunk
Divide by semantic boundaries, such as topic changes, paragraph similarity, and cluster clustering.
Suitable for:
- long report;
- interview records;
- minutes of the meeting;
- Text without a clear title.
Risk is more complex to achieve, and the boundaries are not necessarily stable. It is best to keep the original LOCATOR in the production system, to avoid the inability to return to the original text after semantic segmentation.
5.4 Parent-child chunk
Index small chunk, with Parent Context when displayed or generated.
Example:
- child chunk: a clause;
- parent chunk: the section where it is located;
- source: The entire contract.
Suitable for scenarios that require precise recall but fear of context breaking. Hit with Child when retrieving, and bring parent or adjacent chunks together when generating.
5.5 Hierarchical chunk
Construct the document into multiple layers:
文档摘要
章节摘要
段落 chunk
原文 span
RAPTOR is a method that organizes texts into tree-like abstracts and retrieval structures, which are suitable for long documents, multi-hop problems and summary problems. In the project, you can do a simplified version first: the document-level abstract, the section summary, and the original chunk three layers.
5.6 Late chunking
The usual approach is to split a document into chunks and embed each chunk separately. Late chunking first computes token representations over a longer context, then pools the representations within each chunk. Each chunk embedding can thus retain information from the surrounding document.
Late chunking is well suited to documents with many internal references when the embedding model handles long contexts well. Before replacing the baseline, evaluate the method and check its cost, model support, and reproducibility.
5.7 Graph-aware chunk
If there are entities, relationships, and community structures between documents, you can re-make graphs outside the chunk. GraphRAG is suitable for cross-document synthesis, organization knowledge graph and query-focused summarization.
But it does not replace ordinary chunking. The graph structure relies on the underlying evidence unit, and if the quality of the chunk itself is poor, the map is only organized with bad evidence.
6. How to set the Chunk size and overlap
The OpenAI Retrieval document gives a practical baseline: default max_chunk_size_tokens is 800, chunk_overlap_tokens is 400; can pass chunking_strategy adjustment. The current limit is max_chunk_size_tokens It needs to be between 100 and 4096, and the overlap cannot exceed half of the chunk size.
This default value is suitable for the first version, but don't take it as always correct.
6.1 Scope by task type first
| Scenes | Recommended chunk range | overlap | Explanation |
|---|---|---|---|
| FAQ / Short Q&A | 100-300 tokens | 0-50 | A question and answer is a unit of evidence |
| policy clause | 300-800 tokens | 50-200 | Reservation conditions, exceptions and steps |
| Contract terms | 400-1000 tokens | 100-250 | Terms and definitions may refer to each other |
| technical documentation | 300-900 tokens | 50-200 | Keep header paths and code blocks |
| long report | 600-1500 tokens | 150-400 | Combine Section Summary |
| Form | By row/region/semantic block | Use less token overlap | Retaining the row and column relationship is more important than the number of tokens |
| Code | By function/class/file structure | little context | Do not cut function signatures and implementations |
6.2 Overlap is not the bigger the better
The role of overlap is to prevent the boundary cut off semantics. But overlap is too large, it will cause:
- Too many repetitions;
- top-k is filled with the same paragraph;
- Embedding storage costs increase;
- Reference results are redundant;
- MRR and Recall seem to be improved, but the answer coverage doesn't really improve.
A practical practice is to limit the number of the same source in the top-k, or to de-weight the adjacent chunk.
6.3 Select parameters with eval
Don't choose chunk size by feeling. At least compare these combinations:
300 / 50
600 / 100
800 / 200
800 / 400
1200 / 300
Retrieve eval as mentioned earlier for each group run:
- Hit Rate@k;
- Recall@k;
- MRR;
- nDCG;
- Duplicate Rate;
- Citation Support;
- token cost;
- P95 delay.
The final choice is usually not the highest Recall configuration, but a compromise of Recall, precision, cost and reference readability.
7. How to deal with different document types
7.1 PDF
PDF is the hardest hit area for enterprise RAG. Frequently asked questions include:
- The two columns are in the wrong order of layout;
- header and footer repeat;
- Footnotes are mixed into the text;
- Form lost rows;
- Scanned OCR error;
- The page number and original LOCator are lost.
Suggestion:
- Save page numbers and characters/coordinates;
- Remove the duplicate header and footer;
- Extract the table separately;
- OCR results are marked Confidence ;
- Scanned low-quality pages enter the manual repair queue.
7.2 Word / Markdown / HTML
These formats are better structured, and are preferred to be divided into the title tree.
Keep:
- H1/H2/H3;
- list level;
- form;
- code block;
- link;
- update time;
- document path.
Do not put HTML navigation, footers, and sidebars together.
7.3 Forms and Spreadsheets
Don't simply convert the form to continuous text.
Wrong practice:
企业版 10% 专业版 5% 免费版 0%
Better Evidence Unit:
{
"chunk_id": "pricing_2026_q2#table_sla#row_enterprise",
"source_id": "pricing_2026_q2",
"table": "SLA Credit",
"row_key": "enterprise",
"columns": {
"plan": "企业版",
"sla_credit": "10%",
"notice_window": "30 天"
},
"text": "SLA Credit 表中,企业版的 SLA credit 为 10%,通知窗口为 30 天。"
}
The retrieval text can be a natural language-based row, and still return to the row-column coordinates during auditing.
7.4 Code documentation and source code
The code should not be roughly cut by the fixed token.
Suitable borders:
- file;
- class;
- function;
- method;
- docstring;
- comment block;
- test case.
Save:
- repo;
- branch/commit;
- file path;
- symbol name;
- start line/end line;
- language;
- imports;
- dependency relation.
It is best to jump to the specific line number for the reference to the code question and answer.
7.5 FAQ and customer service knowledge
FAQ is best for one question and one answer as a chunk.
But be aware of the synonymous issue:
{
"canonical_question": "退款多久到账?",
"aliases": ["钱什么时候退回来?", "退款几天到?"],
"answer": "退款通常在 1-5 个工作日内退回原支付渠道。",
"tags": ["refund", "payment"]
}
Aliases can enter the search text, but don't confuse it into the original text.
8. Metadata is half of the retrieval quality
As mentioned above, permissions and timeliness must be handled at the retrieval layer. Metadata is the basis for achieving this.
The OpenAI Retrieval document mentions that the Vector Store File can be used with attributes for attribute filtering. The documentation also shows that the Attributes dictionary has key and length limits. This reminds us: metadata cannot be piled up casually, it should be designed into few and critical fields.
Recommended minimum fields:
| Field | Use |
|---|---|
source_id |
Associate the original document |
chunk_id |
Stable Evidence Unit |
tenant_id |
tenant isolation |
access_scope |
Permission filtering |
updated_at |
Timeliness |
version |
Multi-Version Policy |
status |
active/deprecated/draft |
source_type |
policy/contract/faq/code/table |
language |
multilingual search |
owner |
Responsible person and audit |
content_hash |
Incremental update |
hierarchy_path |
structural context |
If the metadata key of the platform is limited, you can put the filter required fields in attributes, and put the display and audit fields to your own database.
A common architecture is:
Vector Store:
- chunk_id
- searchable text
- embedding
- minimal filter attributes
Metadata DB:
- source registry
- locator
- permissions
- version history
- source hash
- full quote
- audit info
Don't put all governance information into the vector library, and don't let the vector library become the only source of truth.
9. Deduplication, version and incremental updates
9.1 Dedusting
There are often duplicates in the knowledge base:
- The same PDF is uploaded multiple times;
- Word and PDF are the same document in different formats;
- FAQ copied the policy terms;
- The old and new versions are highly similar;
- chunk overlap creates a large number of duplicate fragments.
There are three layers of weight removal:
| level | Method |
|---|---|
| file level | source hash |
| paragraph level | normalized text hash |
| Semantic level | Embedding similarity or MinHash |
But don't delete all the repetitions. The old and new versions are similar but need to be reserved, you should keep the version relationship instead of removing the old version.
Version 9.2
The version field to answer:
- Is this draft, active or deprecated?
- Which version is currently in effect?
- What is the effective date?
- Which version is replaced?
- Inquire
as_ofWhich version should be hit on a date?
Example:
{
"source_id": "refund_policy_2026_03",
"version": "2026-03",
"status": "active",
"valid_from": "2026-03-18",
"valid_to": null,
"supersedes": "refund_policy_2025_10"
}
9.3 Incremental updates
Each knowledge base update should not be rebuilt in full, unless the scale is small.
Better process:
- Calculate the source hash;
- Unchanged document skipped;
- Change the document re-parsing;
- Compare chunk hash;
- Add chunk to write index;
- Delete or abolish old chunks;
- record change logs;
- Run back to Eval for the affected theme.
The OpenAI Retrieval document mentions that after removing the Vector Store file, the search results may still appear removed in a short period of time. Such final consistency details should be considered in production: if it is a high-risk knowledge base, there must be a short-term protection strategy after deletion or abolishment, such as filtering status=active, not just relying on physical deletion.
10. A minimal structured chunker example
The following example demonstrates how to cut the Markdown document into Evidence chunks as a title structure. It is not a full production implementation, but demonstrates key ideas:
- Keep the title path;
- Generate a stable ID for each chunk;
- limit the maximum length;
- Add metadata;
- output JSONL.
import hashlib
import json
import re
from datetime import datetime, timezone
from pathlib import Path
def sha256(text):
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def approx_tokens(text):
# 粗略估算,生产环境应使用目标模型 tokenizer。
chinese_chars = len(re.findall(r"[\u4e00-\u9fff]", text))
ascii_words = len(re.findall(r"[A-Za-z0-9_]+", text))
return chinese_chars + ascii_words
def split_markdown_blocks(markdown):
blocks = []
hierarchy = []
buffer = []
def flush():
if buffer:
blocks.append({
"hierarchy_path": hierarchy[:],
"text": "\n".join(buffer).strip()
})
buffer.clear()
for line in markdown.splitlines():
heading = re.match(r"^(#{1,6})\s+(.+)$", line)
if heading:
flush()
level = len(heading.group(1))
title = heading.group(2).strip()
hierarchy[:] = hierarchy[: level - 1] + [title]
continue
if line.strip():
buffer.append(line)
else:
flush()
flush()
return [b for b in blocks if b["text"]]
def pack_blocks(blocks, max_tokens=700):
chunks = []
current = []
current_path = []
current_tokens = 0
for block in blocks:
block_text = block["text"]
block_tokens = approx_tokens(block_text)
if current and current_tokens + block_tokens > max_tokens:
chunks.append({
"hierarchy_path": current_path,
"text": "\n\n".join(current)
})
current = []
current_tokens = 0
current_path = block["hierarchy_path"]
current.append(block_text)
current_tokens += block_tokens
if current:
chunks.append({
"hierarchy_path": current_path,
"text": "\n\n".join(current)
})
return chunks
def build_chunks(path, source_meta):
raw = Path(path).read_text(encoding="utf-8")
source_hash = sha256(raw)
blocks = split_markdown_blocks(raw)
packed = pack_blocks(blocks)
ingested_at = datetime.now(timezone.utc).isoformat()
for index, chunk in enumerate(packed, start=1):
context_prefix = " > ".join(chunk["hierarchy_path"])
text = chunk["text"].strip()
chunk_id = f"{source_meta['source_id']}#chunk_{index:04d}"
yield {
"chunk_id": chunk_id,
"source_id": source_meta["source_id"],
"source_title": source_meta["source_title"],
"source_type": source_meta["source_type"],
"version": source_meta["version"],
"status": source_meta["status"],
"updated_at": source_meta["updated_at"],
"owner": source_meta["owner"],
"tenant_id": source_meta["tenant_id"],
"access_scope": source_meta["access_scope"],
"language": source_meta.get("language", "zh-CN"),
"hierarchy_path": chunk["hierarchy_path"],
"context_prefix": context_prefix,
"text": text,
"index_text": f"{context_prefix}\n{text}" if context_prefix else text,
"content_hash": "sha256:" + sha256(context_prefix + "\n" + text),
"source_hash": "sha256:" + source_hash,
"ingested_at": ingested_at
}
if __name__ == "__main__":
meta = {
"source_id": "refund_policy_2026_03",
"source_title": "退款政策 2026-03 版",
"source_type": "policy_doc",
"version": "2026-03",
"status": "active",
"updated_at": "2026-03-18",
"owner": "finance_ops",
"tenant_id": "global",
"access_scope": ["customer_service", "finance"],
"language": "zh-CN"
}
with Path("evidence_chunks.jsonl").open("w", encoding="utf-8") as f:
for row in build_chunks("refund_policy.md", meta):
f.write(json.dumps(row, ensure_ascii=False) + "\n")
Production versions also require:
- true tokenizer;
- PDF/HTML/Docx parser;
- table structure processing;
- locator;
- ACL mapping;
- source registry;
- Embedding write;
- deleting and abolishing policies;
- eval automation.
But this example has embodied the core principle: chunk is the object of evidence with context and governance information, not a plain text fragment.
11. How to evaluate whether chunking works
Evaluation chunking can't just look at "how many pieces are cut out". To go back to business issues.
11.1 Offline Search Evaluation
Use the Gold Set of the previous article:
{
"query_id": "refund_001",
"query": "跨境订单退款多久到账?什么情况需要人工审核?",
"gold_chunk_ids": [
"refund_policy_2026_03#sec_4#chunk_02"
],
"required_facts": ["到账时间", "跨境支付需人工审核"]
}
Run the same query for different chunk strategies, compare:
- Hit Rate@k;
- Recall@k;
- MRR;
- nDCG;
- Duplicate Rate;
- Citation Support;
- token cost;
- Retrieve delay.
11.2 Reference readability evaluation
Chunk hits are not equal to suitable for reference. Also ask:
- After the user clicks on the Citation, can he directly understand it?
- Is quote too long?
- Does it contain irrelevant paragraphs?
- Is the necessary context missing?
- Can it be highlighted to the original location?
11.3 Failed Sample Attribution
Divide errors into several categories:
| Failed type | Explanation | possible repair |
|---|---|---|
| Missing source | The document is not in the library | source inventory |
| Parse failure | Parsing error or OCR error | Parser/OCR fix |
| Bad boundary | chunk cut semantics | structured segmentation |
| Too broad | chunk too big | Reduce the chunk or press the section to cut |
| Too narrow | chunk too small | parent-child or overlap |
| Lost table relation | Table rows are missing | table structure |
| Stale version | old document hit | version filter |
| Permission leak | recall | metadata filter |
If there is no failure attribution, the chunking experiment will become a metaphysical metaphysics.
12. Production landing architecture
It is recommended to split the ingestion system into several modules.
Source Connectors
-> Parser Workers
-> Normalizer
-> Structure Extractor
-> Chunk Builder
-> Metadata Enricher
-> Quality Gate
-> Vector Index + Keyword Index
-> Metadata Store
-> Retrieval Eval
12.1 Source Connectors
Responsible for pulling:
- folder;
- CMS;
- Notion/Confluence;
- Google Drive/SharePoint;
- database;
- Git repo;
- work order system;
- web page.
Connectors to record source ID, path, update time, owner, and permissions.
12.2 Parser Workers
Different formats use different Parser. Don't use a generic read_text solve all problems.
Parser output blocks:
{
"block_id": "b_0012",
"type": "paragraph",
"text": "...",
"page": 4,
"section": "4.2",
"bbox": [72, 120, 520, 188]
}
12.3 Quality Gate
Quality check before storage:
- ratio of empty text;
- OCR Confidence;
- Repeat the header and footer;
- table parsing failed;
- chunk is too short/too long;
- Missing permission fields;
- missing version field;
- Source hash exception;
- Non UTF-8 text.
Do not silently store documents that have failed quality. mark ingestion_failed, enter the repair queue.
12.4 Double Index
Don't just rely on vector indexes. Many corporate issues include precise terms, numbering, contract terms, error codes, and API names. Hybrid search is usually more stable.
The OpenAI Retrieval document also mentions that the weights of embedding and text in the Ranking Options can be adjusted. The performance of semantic and keyword should be observed separately in the project.
12.5 Metadata Store
The vector library is responsible for finding similar texts, and Metadata Store is responsible for governance and auditing. both pass chunk_id and source_id association.
13. Common Mistakes
13.1 Uploading a file is equal to building a knowledge base
Uploading is just the beginning. The production knowledge base needs to resolve the closed loop of quality, authority, version, reference positioning and evaluation.
13.2 The default chunk parameters are always enough
Default parameters are suitable for starting, not all documents. FAQ, contracts, forms, codes, and long-reported segmentation strategies are different.
13.3 Save only chunk text
Chunks without metadata cannot be filtered, audited, or reliably referenced.
13.4 After the form is transferred to the text, it is finished
The most important thing about the table is the relationship between the row and column. After converting to prose, the model may mix the fields of the professional and enterprise versions together.
13.5 The bigger the overlap, the better
OverLap will increase redundancy. It solves the boundary problem and also creates Duplicate problems. Must be combined with Duplicate Rate and Citation Support.
13.6 Do not do version abolishment
Deleting old documents does not equal production security. There must be active/deprecated state, valid_from/valid_to and filtering when querying.
13.7 Do not retain LOCATOR
If there is no page number, section, line number or span, the reference UI can only jump to the entire document, and it is difficult for users to check.
14. Landing road map
Week 1: Build Source Registry
- Inventory of knowledge sources;
- Define the source ID;
- Save OWNER, permissions, version, update time;
- Mark which sources can be stored.
Week 2: Do Structured Analysis
- Parse PDF/Docx/HTML/Markdown/tables separately;
- Remove header footer and navigation noise;
- Keep title trees, page numbers, tables, code blocks.
Week 3: Generate Evidence Chunks
- Definition
evidence_chunks.jsonlschema; - Select the chunk policy by document type;
- Add context_prefix;
- Generate chunk hash and source hash.
Week 4: Write Index and Metadata Store
- Embedding writes the vector index;
- Exact fields are written to keyword index;
- Permissions, versions, and locator to write metadata store;
- Create deletion and abolition processes.
Week 5: Running Chunking Eval
- Label Gold Chunk with 50-100 business questions;
- Compare different chunk sizes, overlap and policies;
- output failure attribution;
- Curing a version of the on-line threshold.
15. Pre-release self-check list
At least ask these questions before going online:
- Is there a source registry instead of scattered file paths?
- Does each source have owner, update time, version and permissions?
- Does PDF handle header and footer, page number, two bars, OCR and table?
- Does the table retain row-column relationships and cell locator?
- Is each chunk stable?
chunk_idandsource_id? - Is each chunk has
hierarchy_path、context_prefixand the original locator? - Is there
status=active/deprecated/draftand version filtering? - Whether to record
content_hashandsource_hash? - Is there an incremental update and abolishing strategy?
- Did you run over Retrieval eval for chunk parameters?
- Have you seen Duplicate Rate, Citation Support, Token cost and delay?
- Can it jump back to the original position from Citation?
- Is there an Ingestion Failed queue instead of silently incoming bad documents?
If these questions cannot be answered, the knowledge base is only a "file collection", not a production-level RAG knowledge base.
16. Summary
The ingestion and chunking of RAG, the core is not to cut the text into multiple segments, but to turn the original material into an evidential unit that can be retrieved, referenced, filtered, and auditable.
The minimum available link is:
Source Registry -> Parser -> Normalizer -> Structure Recovery -> Evidence Chunk -> Metadata -> Index -> Eval -> Fix Loop
Well done ingestion system will make the subsequent search, reference and evaluation easier. The poorly done ingestion system will push all the problems to the Prompt and the model.
The next article recommends continuing to write: How to do mixed retrieval and reranking of RAG: sorting closed loops from keywords, vectors to reranker.
References
- OpenAI Developers: Retrieval
- OpenAI Developers: File search
- OpenAI Developers: Retrieval chunking section
- Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
- Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- Jina AI, Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models