Cover diagram: RAG Knowledge Base ingestion pipeline parses, cleans, segments, adds metadata, and writes retrieval indexes to original documents

Date: 2026-06-20
Suitable for readers: Enterprise knowledge base, document Q&A, customer service assistant, compliance quiz, research assistant or engineer, product leader and technical leader of an internal Agent.

Abstract

The citation and attribution of RAG is mentioned earlier: the key conclusions in the answer must be traced back to the evidence.

This article goes back to the lower level question: Where do these evidence units come from?

Many RAG projects run into trouble during ingestion, before model strength or the choice of vector database comes into play. PDF headers and footers get mixed into the body text, table cells lose their row and column associations, obsolete documents remain active, and access permissions never reach the metadata. Overlong chunks make retrieval imprecise, while very short chunks lose their context. Tuning the prompt, reranker, or citation format can do only so much when the underlying evidence is damaged.

So knowledge base ingestion is not "uploading files to a vector library". It is a production line: Inventory of sources, parsing documents, normalized texts, reserved structures, segmented evidence, metadata, deduplication and version management, writing indexes, running retrieval evaluation, continuous reflow failure samples.

This article gives a set of ingest and chunking methods, including the evidence unit JSONL contract, chunk policy selection, parameter recommendation, table and code processing, permission and version fields, minimum structured chunker examples, and a self-check list before going online.

Contents

1. Why do you determine the upper limit of RAG

The second half of the RAG looks more "smart": retrieval, reranking, generation, reference check. But what really determines the upper limit of the system is often the first ingestion.

If the ingestion phase handles the document like this:

退款政策
页脚:内部资料,仅供参考
表 2
人工审核 跨境支付 大额退款
1-5 个工作日
页眉:退款政策

It is difficult to answer the model stably after:

跨境订单是否需要人工审核?退款多久到账?

Because the original relationship has been lost. The rows and columns of the table, title level, version, page number, permission, and update time have not entered the Evidence unit.

The poor quality of RAG intake will bring six types of problems.

Problem Which performed root cause
Can't be recalled The user is right, but the correct content is not in the top-k Documents are not stored in the warehouse, broken, and the title is lost
Recall too pan Top-k are all relevant but not precise chunk is too long, noise, metadata is missing
The answer is out of context Only one sentence is quoted, and exception conditions are missed chunk is too short, context broken
Old version overrides new version Answer with expired policy Release field and deprecated status is missing
permission leak Recall users do not have access to content Access Scope is not in search filter
References cannot be checked The user can't find the original text when the user clicks on the source Locator, page, span, source hash is missing

Therefore, the goal of the ingestion pipeline is not to "generate embedding", but to generate retrieve, cisable, filterable, auditable units of evidence.

2. The 9 stages of the ingestion line

A reliable knowledge base ingestion pipeline can be split into 9 stages.

Frame diagram: Intake closed Loop from Source Inventory to Parse, Normalize, Segment, Metadata, Deduplicate, Index, Eval and Fix Loop

Stage Target Product
1. Source Inventory Inventory of knowledge sources source registry
2. Permission Mapping Establish permissions and tenant boundaries access policy
3. Parsing Extract content from PDF/HTML/Docx/tables parsed blocks
4. Normalization Clean headers, footers, garbled characters, duplicate blanks normalized text
5. Structure Recovery Keep titles, page numbers, tables, lists, code blocks structured document
6. Segmentation Cut into retrievable unit of evidence evidence chunks
7. Metadata Enrichment Add version, permissions, source, hash, locator chunk metadata
8. Indexing Embedding, sparse index, vector library write vector/search index
9. Evaluation Loop Check recall and reference quality with Gold Set eval report

Many teams skip the first 5 steps and drop the document directly into the default chunker. This works in demo, but you will soon encounter boundaries in the enterprise knowledge base:

  • The two columns of the PDF are misread;
  • Tables are flattened into meaningless texts;
  • Header and footer repeated pollution embedding;
  • Multiple versions of the same policy are hit at the same time;
  • Separation of document title and paragraph content;
  • Permissions can only alert the model after answering, not filter before retrieval.

The more serious the intake phase, the easier the subsequent search and attribution is.

3. What should the unit of evidence look like?

In the previous article, I mentioned citation and attribution Evidence Span. This article expands it into an indexable evidence_chunk.

It is recommended that each chunk contain at least these fields:

{
  "chunk_id": "refund_policy_2026_03#sec_4#chunk_02",
  "source_id": "refund_policy_2026_03",
  "source_title": "退款政策 2026-03 版",
  "source_type": "policy_doc",
  "version": "2026-03",
  "status": "active",
  "updated_at": "2026-03-18",
  "owner": "finance_ops",
  "tenant_id": "global",
  "access_scope": ["customer_service", "finance"],
  "language": "zh-CN",
  "hierarchy_path": ["退款政策", "4. 跨境退款", "4.2 人工审核"],
  "locator": {
    "page": 4,
    "section": "4.2",
    "start_char": 1832,
    "end_char": 2150
  },
  "text": "跨境支付订单需进入人工审核。审核完成后,退款通常在 1-5 个工作日内退回原支付渠道。",
  "context_prefix": "退款政策 > 跨境退款 > 人工审核",
  "content_hash": "sha256:...",
  "source_hash": "sha256:...",
  "ingested_at": "2026-06-20T22:18:24+08:00"
}

Here are a few key points.

3.1 text Not all

Vector indexes may be mainly used text, but the system cannot only save text. No source_id、version、access_scope、locator, will not be able to do permission filtering, version control and reference highlighting in the future.

3.2 context_prefix Very important

Many chunks themselves have only one sentence:

需进入人工审核。

Its semantics are weak without the title path. You can add light context to the index text:

退款政策 > 跨境退款 > 人工审核
需进入人工审核。

This is not the original text for users to read, but helps to retrieve the understanding context. The presentation should still go back to the original SPAN.

3.3 content_hash and source_hash Used to do incremental updates

The knowledge base will change. Without Hash, it is difficult to judge:

  • whether the document has really changed;
  • Which chunks need to recalculate embedding;
  • Whether the old chunk should be abolished;
  • Which version did the online accident use?

4. Chunking is not a fixed length cut text

The most common mistake is to understand chunking as:

每 800 tokens 切一段,重叠 200 tokens。

Fixed length segmentation is valuable, especially for plain text and fast baselines. But it's not the whole production RAG.

A good chunk should meet five conditions:

Condition Explanation
semantic intact A chunk can express a relatively complete fact or step
Recallable Words or semantics in user questions can hit it
Can be quoted Users can understand what it supports after clicking
can be filtered Metadata with permissions, versions, sources and times
Combine Multiple chunks can form complete answers without conflicting with each other

If the chunk is too large, the search will be general:

整章退款政策都被召回,但答案只需要跨境退款一条规则。

If the chunk is too small, the evidence will be broken:

chunk A: 跨境支付订单需进入人工审核。
chunk B: 审核完成后按原支付渠道退款。
chunk C: 退款通常在 1-5 个工作日内完成。

The model may only see A, and miss B and C.

So the essence of chunking is evidence modeling, not string slices.

5. How to choose common chunk strategies

5.1 Fixed token chunk

Divide by token number and set overlap.

Suitable for:

  • fast baseline;
  • plain text in simple structure;
  • a large number of FAQs;
  • Internal searches with low reference granularity requirements.

The advantage is that it is simple and stable. The disadvantage is that it is easy to cut off the title, table, steps and exceptions.

5.2 Structured chunks

Divide by title, paragraph, list, table, code block.

Suitable for:

  • policy document;
  • product manual;
  • contract;
  • API documentation;
  • research report;
  • Standard operating process.

This is the most commonly used production strategy for enterprise RAG. It keeps the document structure:

H1: 退款政策
H2: 跨境退款
H3: 人工审核
Paragraph: 跨境支付订单需进入人工审核...

5.3 Semantics chunk

Divide by semantic boundaries, such as topic changes, paragraph similarity, and cluster clustering.

Suitable for:

  • long report;
  • interview records;
  • minutes of the meeting;
  • Text without a clear title.

Risk is more complex to achieve, and the boundaries are not necessarily stable. It is best to keep the original LOCATOR in the production system, to avoid the inability to return to the original text after semantic segmentation.

5.4 Parent-child chunk

Index small chunk, with Parent Context when displayed or generated.

Example:

  • child chunk: a clause;
  • parent chunk: the section where it is located;
  • source: The entire contract.

Suitable for scenarios that require precise recall but fear of context breaking. Hit with Child when retrieving, and bring parent or adjacent chunks together when generating.

5.5 Hierarchical chunk

Construct the document into multiple layers:

文档摘要
章节摘要
段落 chunk
原文 span

RAPTOR is a method that organizes texts into tree-like abstracts and retrieval structures, which are suitable for long documents, multi-hop problems and summary problems. In the project, you can do a simplified version first: the document-level abstract, the section summary, and the original chunk three layers.

5.6 Late chunking

The usual approach is to split a document into chunks and embed each chunk separately. Late chunking first computes token representations over a longer context, then pools the representations within each chunk. Each chunk embedding can thus retain information from the surrounding document.

Late chunking is well suited to documents with many internal references when the embedding model handles long contexts well. Before replacing the baseline, evaluate the method and check its cost, model support, and reproducibility.

5.7 Graph-aware chunk

If there are entities, relationships, and community structures between documents, you can re-make graphs outside the chunk. GraphRAG is suitable for cross-document synthesis, organization knowledge graph and query-focused summarization.

But it does not replace ordinary chunking. The graph structure relies on the underlying evidence unit, and if the quality of the chunk itself is poor, the map is only organized with bad evidence.

6. How to set the Chunk size and overlap

The OpenAI Retrieval document gives a practical baseline: default max_chunk_size_tokens is 800, chunk_overlap_tokens is 400; can pass chunking_strategy adjustment. The current limit is max_chunk_size_tokens It needs to be between 100 and 4096, and the overlap cannot exceed half of the chunk size.

This default value is suitable for the first version, but don't take it as always correct.

6.1 Scope by task type first

Scenes Recommended chunk range overlap Explanation
FAQ / Short Q&A 100-300 tokens 0-50 A question and answer is a unit of evidence
policy clause 300-800 tokens 50-200 Reservation conditions, exceptions and steps
Contract terms 400-1000 tokens 100-250 Terms and definitions may refer to each other
technical documentation 300-900 tokens 50-200 Keep header paths and code blocks
long report 600-1500 tokens 150-400 Combine Section Summary
Form By row/region/semantic block Use less token overlap Retaining the row and column relationship is more important than the number of tokens
Code By function/class/file structure little context Do not cut function signatures and implementations

6.2 Overlap is not the bigger the better

The role of overlap is to prevent the boundary cut off semantics. But overlap is too large, it will cause:

  • Too many repetitions;
  • top-k is filled with the same paragraph;
  • Embedding storage costs increase;
  • Reference results are redundant;
  • MRR and Recall seem to be improved, but the answer coverage doesn't really improve.

A practical practice is to limit the number of the same source in the top-k, or to de-weight the adjacent chunk.

6.3 Select parameters with eval

Don't choose chunk size by feeling. At least compare these combinations:

300 / 50
600 / 100
800 / 200
800 / 400
1200 / 300

Retrieve eval as mentioned earlier for each group run:

  • Hit Rate@k;
  • Recall@k;
  • MRR;
  • nDCG;
  • Duplicate Rate;
  • Citation Support;
  • token cost;
  • P95 delay.

The final choice is usually not the highest Recall configuration, but a compromise of Recall, precision, cost and reference readability.

7. How to deal with different document types

7.1 PDF

PDF is the hardest hit area for enterprise RAG. Frequently asked questions include:

  • The two columns are in the wrong order of layout;
  • header and footer repeat;
  • Footnotes are mixed into the text;
  • Form lost rows;
  • Scanned OCR error;
  • The page number and original LOCator are lost.

Suggestion:

  • Save page numbers and characters/coordinates;
  • Remove the duplicate header and footer;
  • Extract the table separately;
  • OCR results are marked Confidence ;
  • Scanned low-quality pages enter the manual repair queue.

7.2 Word / Markdown / HTML

These formats are better structured, and are preferred to be divided into the title tree.

Keep:

  • H1/H2/H3;
  • list level;
  • form;
  • code block;
  • link;
  • update time;
  • document path.

Do not put HTML navigation, footers, and sidebars together.

7.3 Forms and Spreadsheets

Don't simply convert the form to continuous text.

Wrong practice:

企业版 10% 专业版 5% 免费版 0%

Better Evidence Unit:

{
  "chunk_id": "pricing_2026_q2#table_sla#row_enterprise",
  "source_id": "pricing_2026_q2",
  "table": "SLA Credit",
  "row_key": "enterprise",
  "columns": {
    "plan": "企业版",
    "sla_credit": "10%",
    "notice_window": "30 天"
  },
  "text": "SLA Credit 表中,企业版的 SLA credit 为 10%,通知窗口为 30 天。"
}

The retrieval text can be a natural language-based row, and still return to the row-column coordinates during auditing.

7.4 Code documentation and source code

The code should not be roughly cut by the fixed token.

Suitable borders:

  • file;
  • class;
  • function;
  • method;
  • docstring;
  • comment block;
  • test case.

Save:

  • repo;
  • branch/commit;
  • file path;
  • symbol name;
  • start line/end line;
  • language;
  • imports;
  • dependency relation.

It is best to jump to the specific line number for the reference to the code question and answer.

7.5 FAQ and customer service knowledge

FAQ is best for one question and one answer as a chunk.

But be aware of the synonymous issue:

{
  "canonical_question": "退款多久到账?",
  "aliases": ["钱什么时候退回来?", "退款几天到?"],
  "answer": "退款通常在 1-5 个工作日内退回原支付渠道。",
  "tags": ["refund", "payment"]
}

Aliases can enter the search text, but don't confuse it into the original text.

8. Metadata is half of the retrieval quality

As mentioned above, permissions and timeliness must be handled at the retrieval layer. Metadata is the basis for achieving this.

The OpenAI Retrieval document mentions that the Vector Store File can be used with attributes for attribute filtering. The documentation also shows that the Attributes dictionary has key and length limits. This reminds us: metadata cannot be piled up casually, it should be designed into few and critical fields.

Recommended minimum fields:

Field Use
source_id Associate the original document
chunk_id Stable Evidence Unit
tenant_id tenant isolation
access_scope Permission filtering
updated_at Timeliness
version Multi-Version Policy
status active/deprecated/draft
source_type policy/contract/faq/code/table
language multilingual search
owner Responsible person and audit
content_hash Incremental update
hierarchy_path structural context

If the metadata key of the platform is limited, you can put the filter required fields in attributes, and put the display and audit fields to your own database.

A common architecture is:

Vector Store:
- chunk_id
- searchable text
- embedding
- minimal filter attributes

Metadata DB:
- source registry
- locator
- permissions
- version history
- source hash
- full quote
- audit info

Don't put all governance information into the vector library, and don't let the vector library become the only source of truth.

9. Deduplication, version and incremental updates

9.1 Dedusting

There are often duplicates in the knowledge base:

  • The same PDF is uploaded multiple times;
  • Word and PDF are the same document in different formats;
  • FAQ copied the policy terms;
  • The old and new versions are highly similar;
  • chunk overlap creates a large number of duplicate fragments.

There are three layers of weight removal:

level Method
file level source hash
paragraph level normalized text hash
Semantic level Embedding similarity or MinHash

But don't delete all the repetitions. The old and new versions are similar but need to be reserved, you should keep the version relationship instead of removing the old version.

Version 9.2

The version field to answer:

  • Is this draft, active or deprecated?
  • Which version is currently in effect?
  • What is the effective date?
  • Which version is replaced?
  • Inquire as_of Which version should be hit on a date?

Example:

{
  "source_id": "refund_policy_2026_03",
  "version": "2026-03",
  "status": "active",
  "valid_from": "2026-03-18",
  "valid_to": null,
  "supersedes": "refund_policy_2025_10"
}

9.3 Incremental updates

Each knowledge base update should not be rebuilt in full, unless the scale is small.

Better process:

  1. Calculate the source hash;
  2. Unchanged document skipped;
  3. Change the document re-parsing;
  4. Compare chunk hash;
  5. Add chunk to write index;
  6. Delete or abolish old chunks;
  7. record change logs;
  8. Run back to Eval for the affected theme.

The OpenAI Retrieval document mentions that after removing the Vector Store file, the search results may still appear removed in a short period of time. Such final consistency details should be considered in production: if it is a high-risk knowledge base, there must be a short-term protection strategy after deletion or abolishment, such as filtering status=active, not just relying on physical deletion.

10. A minimal structured chunker example

The following example demonstrates how to cut the Markdown document into Evidence chunks as a title structure. It is not a full production implementation, but demonstrates key ideas:

  • Keep the title path;
  • Generate a stable ID for each chunk;
  • limit the maximum length;
  • Add metadata;
  • output JSONL.
import hashlib
import json
import re
from datetime import datetime, timezone
from pathlib import Path


def sha256(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def approx_tokens(text):
    # 粗略估算,生产环境应使用目标模型 tokenizer。
    chinese_chars = len(re.findall(r"[\u4e00-\u9fff]", text))
    ascii_words = len(re.findall(r"[A-Za-z0-9_]+", text))
    return chinese_chars + ascii_words


def split_markdown_blocks(markdown):
    blocks = []
    hierarchy = []
    buffer = []

    def flush():
        if buffer:
            blocks.append({
                "hierarchy_path": hierarchy[:],
                "text": "\n".join(buffer).strip()
            })
            buffer.clear()

    for line in markdown.splitlines():
        heading = re.match(r"^(#{1,6})\s+(.+)$", line)
        if heading:
            flush()
            level = len(heading.group(1))
            title = heading.group(2).strip()
            hierarchy[:] = hierarchy[: level - 1] + [title]
            continue

        if line.strip():
            buffer.append(line)
        else:
            flush()

    flush()
    return [b for b in blocks if b["text"]]


def pack_blocks(blocks, max_tokens=700):
    chunks = []
    current = []
    current_path = []
    current_tokens = 0

    for block in blocks:
        block_text = block["text"]
        block_tokens = approx_tokens(block_text)

        if current and current_tokens + block_tokens > max_tokens:
            chunks.append({
                "hierarchy_path": current_path,
                "text": "\n\n".join(current)
            })
            current = []
            current_tokens = 0

        current_path = block["hierarchy_path"]
        current.append(block_text)
        current_tokens += block_tokens

    if current:
        chunks.append({
            "hierarchy_path": current_path,
            "text": "\n\n".join(current)
        })

    return chunks


def build_chunks(path, source_meta):
    raw = Path(path).read_text(encoding="utf-8")
    source_hash = sha256(raw)
    blocks = split_markdown_blocks(raw)
    packed = pack_blocks(blocks)
    ingested_at = datetime.now(timezone.utc).isoformat()

    for index, chunk in enumerate(packed, start=1):
        context_prefix = " > ".join(chunk["hierarchy_path"])
        text = chunk["text"].strip()
        chunk_id = f"{source_meta['source_id']}#chunk_{index:04d}"

        yield {
            "chunk_id": chunk_id,
            "source_id": source_meta["source_id"],
            "source_title": source_meta["source_title"],
            "source_type": source_meta["source_type"],
            "version": source_meta["version"],
            "status": source_meta["status"],
            "updated_at": source_meta["updated_at"],
            "owner": source_meta["owner"],
            "tenant_id": source_meta["tenant_id"],
            "access_scope": source_meta["access_scope"],
            "language": source_meta.get("language", "zh-CN"),
            "hierarchy_path": chunk["hierarchy_path"],
            "context_prefix": context_prefix,
            "text": text,
            "index_text": f"{context_prefix}\n{text}" if context_prefix else text,
            "content_hash": "sha256:" + sha256(context_prefix + "\n" + text),
            "source_hash": "sha256:" + source_hash,
            "ingested_at": ingested_at
        }


if __name__ == "__main__":
    meta = {
        "source_id": "refund_policy_2026_03",
        "source_title": "退款政策 2026-03 版",
        "source_type": "policy_doc",
        "version": "2026-03",
        "status": "active",
        "updated_at": "2026-03-18",
        "owner": "finance_ops",
        "tenant_id": "global",
        "access_scope": ["customer_service", "finance"],
        "language": "zh-CN"
    }

    with Path("evidence_chunks.jsonl").open("w", encoding="utf-8") as f:
        for row in build_chunks("refund_policy.md", meta):
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

Production versions also require:

  • true tokenizer;
  • PDF/HTML/Docx parser;
  • table structure processing;
  • locator;
  • ACL mapping;
  • source registry;
  • Embedding write;
  • deleting and abolishing policies;
  • eval automation.

But this example has embodied the core principle: chunk is the object of evidence with context and governance information, not a plain text fragment.

11. How to evaluate whether chunking works

Evaluation chunking can't just look at "how many pieces are cut out". To go back to business issues.

11.1 Offline Search Evaluation

Use the Gold Set of the previous article:

{
  "query_id": "refund_001",
  "query": "跨境订单退款多久到账?什么情况需要人工审核?",
  "gold_chunk_ids": [
    "refund_policy_2026_03#sec_4#chunk_02"
  ],
  "required_facts": ["到账时间", "跨境支付需人工审核"]
}

Run the same query for different chunk strategies, compare:

  • Hit Rate@k;
  • Recall@k;
  • MRR;
  • nDCG;
  • Duplicate Rate;
  • Citation Support;
  • token cost;
  • Retrieve delay.

11.2 Reference readability evaluation

Chunk hits are not equal to suitable for reference. Also ask:

  • After the user clicks on the Citation, can he directly understand it?
  • Is quote too long?
  • Does it contain irrelevant paragraphs?
  • Is the necessary context missing?
  • Can it be highlighted to the original location?

11.3 Failed Sample Attribution

Divide errors into several categories:

Failed type Explanation possible repair
Missing source The document is not in the library source inventory
Parse failure Parsing error or OCR error Parser/OCR fix
Bad boundary chunk cut semantics structured segmentation
Too broad chunk too big Reduce the chunk or press the section to cut
Too narrow chunk too small parent-child or overlap
Lost table relation Table rows are missing table structure
Stale version old document hit version filter
Permission leak recall metadata filter

If there is no failure attribution, the chunking experiment will become a metaphysical metaphysics.

12. Production landing architecture

It is recommended to split the ingestion system into several modules.

Source Connectors
  -> Parser Workers
  -> Normalizer
  -> Structure Extractor
  -> Chunk Builder
  -> Metadata Enricher
  -> Quality Gate
  -> Vector Index + Keyword Index
  -> Metadata Store
  -> Retrieval Eval

12.1 Source Connectors

Responsible for pulling:

  • folder;
  • CMS;
  • Notion/Confluence;
  • Google Drive/SharePoint;
  • database;
  • Git repo;
  • work order system;
  • web page.

Connectors to record source ID, path, update time, owner, and permissions.

12.2 Parser Workers

Different formats use different Parser. Don't use a generic read_text solve all problems.

Parser output blocks:

{
  "block_id": "b_0012",
  "type": "paragraph",
  "text": "...",
  "page": 4,
  "section": "4.2",
  "bbox": [72, 120, 520, 188]
}

12.3 Quality Gate

Quality check before storage:

  • ratio of empty text;
  • OCR Confidence;
  • Repeat the header and footer;
  • table parsing failed;
  • chunk is too short/too long;
  • Missing permission fields;
  • missing version field;
  • Source hash exception;
  • Non UTF-8 text.

Do not silently store documents that have failed quality. mark ingestion_failed, enter the repair queue.

12.4 Double Index

Don't just rely on vector indexes. Many corporate issues include precise terms, numbering, contract terms, error codes, and API names. Hybrid search is usually more stable.

The OpenAI Retrieval document also mentions that the weights of embedding and text in the Ranking Options can be adjusted. The performance of semantic and keyword should be observed separately in the project.

12.5 Metadata Store

The vector library is responsible for finding similar texts, and Metadata Store is responsible for governance and auditing. both pass chunk_id and source_id association.

13. Common Mistakes

13.1 Uploading a file is equal to building a knowledge base

Uploading is just the beginning. The production knowledge base needs to resolve the closed loop of quality, authority, version, reference positioning and evaluation.

13.2 The default chunk parameters are always enough

Default parameters are suitable for starting, not all documents. FAQ, contracts, forms, codes, and long-reported segmentation strategies are different.

13.3 Save only chunk text

Chunks without metadata cannot be filtered, audited, or reliably referenced.

13.4 After the form is transferred to the text, it is finished

The most important thing about the table is the relationship between the row and column. After converting to prose, the model may mix the fields of the professional and enterprise versions together.

13.5 The bigger the overlap, the better

OverLap will increase redundancy. It solves the boundary problem and also creates Duplicate problems. Must be combined with Duplicate Rate and Citation Support.

13.6 Do not do version abolishment

Deleting old documents does not equal production security. There must be active/deprecated state, valid_from/valid_to and filtering when querying.

13.7 Do not retain LOCATOR

If there is no page number, section, line number or span, the reference UI can only jump to the entire document, and it is difficult for users to check.

14. Landing road map

Week 1: Build Source Registry

  • Inventory of knowledge sources;
  • Define the source ID;
  • Save OWNER, permissions, version, update time;
  • Mark which sources can be stored.

Week 2: Do Structured Analysis

  • Parse PDF/Docx/HTML/Markdown/tables separately;
  • Remove header footer and navigation noise;
  • Keep title trees, page numbers, tables, code blocks.

Week 3: Generate Evidence Chunks

  • Definition evidence_chunks.jsonl schema;
  • Select the chunk policy by document type;
  • Add context_prefix;
  • Generate chunk hash and source hash.

Week 4: Write Index and Metadata Store

  • Embedding writes the vector index;
  • Exact fields are written to keyword index;
  • Permissions, versions, and locator to write metadata store;
  • Create deletion and abolition processes.

Week 5: Running Chunking Eval

  • Label Gold Chunk with 50-100 business questions;
  • Compare different chunk sizes, overlap and policies;
  • output failure attribution;
  • Curing a version of the on-line threshold.

15. Pre-release self-check list

At least ask these questions before going online:

  • Is there a source registry instead of scattered file paths?
  • Does each source have owner, update time, version and permissions?
  • Does PDF handle header and footer, page number, two bars, OCR and table?
  • Does the table retain row-column relationships and cell locator?
  • Is each chunk stable? chunk_id and source_id?
  • Is each chunk has hierarchy_path、context_prefix and the original locator?
  • Is there status=active/deprecated/draft and version filtering?
  • Whether to record content_hash and source_hash?
  • Is there an incremental update and abolishing strategy?
  • Did you run over Retrieval eval for chunk parameters?
  • Have you seen Duplicate Rate, Citation Support, Token cost and delay?
  • Can it jump back to the original position from Citation?
  • Is there an Ingestion Failed queue instead of silently incoming bad documents?

If these questions cannot be answered, the knowledge base is only a "file collection", not a production-level RAG knowledge base.

16. Summary

The ingestion and chunking of RAG, the core is not to cut the text into multiple segments, but to turn the original material into an evidential unit that can be retrieved, referenced, filtered, and auditable.

The minimum available link is:

Source Registry -> Parser -> Normalizer -> Structure Recovery -> Evidence Chunk -> Metadata -> Index -> Eval -> Fix Loop

Well done ingestion system will make the subsequent search, reference and evaluation easier. The poorly done ingestion system will push all the problems to the Prompt and the model.

The next article recommends continuing to write: How to do mixed retrieval and reranking of RAG: sorting closed loops from keywords, vectors to reranker.

References