
Date: 2026-06-19
Suitable for readers: engineers, product leaders and technical leaders who are working as an enterprise knowledge base, document quiz, customer service assistant, research assistant, compliance quiz or internal Agent.
Abstract
In the previous article, I talked about how to evaluate the retrieval quality of RAG.
This article continues to go down: How should the reference and attribution of RAG be done?
Many systems will put several "source links" at the end of the answer, but this does not mean that the attribution is reliable. A really useful reference answers three questions:
- Who said this?
- What evidence is it based on?
- Does this evidence really support this conclusion?
If only one document name is displayed, the user cannot know which sentence in the answer comes from; if only a chunk is displayed ID, the user cannot judge whether the source is trusted, expired or out of power; if the model only writes the reference number by itself, but does not do post-check verification, the reference can easily become a decoration.
This article gives a set of engineering design: split the RAG output into claim, citation, evidence span, source metadata and audit record; use structured output constraint model; claim-to-citation checker for posterior verification; in the UI, make the source into a clickable, highlightable, and auditable chain of evidence.
Contents
- 1. References are not decorations, but product contracts
- 2. Three concepts: source, evidence, assertion
- 3. How to choose the reference granularity
- 4. A set of recommended data structures
- 5. Let the model output a verifiable answer
- 6. How to do Claim-to-Citation
- 7. Complex Scenarios: Multi-source, Conflict, Table, Summary, and Permissions
- 8. How to display references in UI
- 9. Online logs and audit records
- 10. How to Evaluate Reference Quality
- 11. An example of a minimum validator
- 12. Common Mistakes
- 13. Landing road map
- 14. Pre-release self-check list
- 15. Summary
- References
1. References are not decorations, but product contracts
In a production system, the reference bears at least four responsibilities.
| Responsibility | Problem | failures |
|---|---|---|
| can verify | Can users click on the evidence and see the original text? | The answer cannot be checked |
| to be blamed | Where does the conclusion come from? | Unable to replay after error |
| Can be audited | Does the system record the evidence retrieved at that time? | Online accident cannot be restored |
| manageable | Is the source expired, more powerful, and low credible? | Compliance and Security Risks |
So the reference is not the "beautify answer" format, but the RAG system's commitment to the user: The key conclusion in the answer must be able to return to the evidence.
It can be seen in the OpenAI File Search document that when using a File to Search, the response will include file_search_call and with file_citation Annotation message; if you want to analyze the search results, you can also explicitly include file_search_call.results. This means that the reference should not just stay at the natural language layer, but should be part of the system output.
The Web Search document also emphasizes that Inline Citations should be clearly visible and clickable when displaying Web results to users or information in them. This principle also applies to enterprise RAGs: references should be seen, clicked, and verified by users.
2. Three concepts: source, evidence, assertion
The most common problem with reference design is to mix Source, Evidence and Claim together.
2.1 Source: Source
Source is the original data unit, for example:
- PDF file;
- Notion page;
- contract version;
- work order record;
- database row;
- web URL;
- Internal policy documentation.
It answers "what is this material".
Source metadata should at least contain:
{
"source_id": "refund_policy_2026_03",
"title": "退款政策 2026-03 版",
"source_type": "policy_doc",
"version": "2026-03",
"updated_at": "2026-03-18",
"owner": "finance_ops",
"access_scope": ["customer_service", "finance"],
"canonical_url": "kb://policies/refund/2026-03",
"content_hash": "sha256:..."
}
2.2 Evidence: Evidence
Evidence is the smallest text or structured fragment that can support a conclusion. It can be a chunk, a piece of original text, a table cell, a database record, or a document area.
It answers "Which paragraph material supports this statement".
{
"evidence_id": "refund_policy_2026_03#chunk_09",
"source_id": "refund_policy_2026_03",
"page": 4,
"section": "跨境退款",
"span": {
"start_char": 1832,
"end_char": 1938
},
"quote": "跨境支付订单需进入人工审核,审核完成后按原支付渠道退款。",
"retrieval_score": 0.82
}
2.3 Claim: Assertion
Claim is a key conclusion in the answer that can be judged true and false.
For example:
跨境支付订单需要进入人工审核。
It is not a whole paragraph of answer, nor is it a topic. It should be small enough to ask: "Can this sentence be supported by a certain piece of evidence?"
FActScore splits long text into Atomic Facts, and then judges whether each fact is supported by reliable knowledge sources. This idea is very inspiring for RAG: Reference checks should also try to make assertion-levels, not paragraph-levels.
3. How to choose the reference granularity
Different risk scenarios require different reference granularity.
| Granularity | Example | When to use it | Risk |
|---|---|---|---|
| documentation level | "Source: Refund Policy 2026-03" | Low-risk summary, internal knowledge browsing | Users want to flip the document by themselves |
| paragraph level | lead to a section or chunk | General knowledge quiz, customer service assistant | Chunk is still difficult to check when it is too long |
| Sentence/Segment Level | A sentence in the original text of highlighting | Policies, contracts, medical, financial | Higher cost |
| field level | Bring to table cells or database fields | Report, Contract Field, Order Status | Structured ingestion is required |
| assertion level | Each claim binds the citations | High-risk Q&A, audit scenarios | Requires post-checking |
My advice is:
- For low-risk internal knowledge bases, at least paragraph-level references.
- For policies, contracts, and customer service processes, a fragment-level reference to key conclusions.
- For legal, medical, financial, compliance, and external commitments, make assertion-level references, and keep audit records.
Don't pursue all sentences with references in the first place. A better way is to mark the "key assertions":
- number;
- date;
- conditions;
- exception;
- rights and obligations;
- operating steps;
- Conclusions that affect users.
These places must have verifiable evidence.
4. A set of recommended data structures
The reference system of RAG can be organized with four types of objects.
4.1 Evidence Pack
Before entering the model, organize the search results into the Evidence Pack. Note that the Evidence ID for the model must be stable, short and referenced.
{
"query_id": "refund_001",
"evidence_pack": [
{
"evidence_id": "E1",
"source_id": "refund_policy_2026_03",
"title": "退款政策 2026-03 版",
"version": "2026-03",
"updated_at": "2026-03-18",
"section": "跨境退款",
"quote": "跨境支付订单需进入人工审核,审核完成后按原支付渠道退款。",
"access_scope": ["customer_service", "finance"],
"source_hash": "sha256:abc..."
},
{
"evidence_id": "E2",
"source_id": "refund_policy_2026_03",
"title": "退款政策 2026-03 版",
"version": "2026-03",
"updated_at": "2026-03-18",
"section": "到账时间",
"quote": "退款通常在 1-5 个工作日内退回原支付渠道。",
"access_scope": ["customer_service", "finance"],
"source_hash": "sha256:abc..."
}
]
}
Don't just give the model a large paragraph of text. Short IDs, titles, versions, quotes, and source metadata are all important.
4.2 Answer Contract
Let the model output structured answers, not just Markdown.
{
"answer": "跨境支付订单需要进入人工审核。审核完成后,退款通常会在 1-5 个工作日内退回原支付渠道。",
"claims": [
{
"claim_id": "C1",
"text": "跨境支付订单需要进入人工审核。",
"citations": ["E1"]
},
{
"claim_id": "C2",
"text": "审核完成后,退款通常会在 1-5 个工作日内退回原支付渠道。",
"citations": ["E1", "E2"]
}
],
"insufficient_evidence": false
}
This structure brings three benefits:
- Claim can be checked one by one;
- The reference can be rendered as a user-clickable UI;
- Audit logs can be recorded.
4.3 Citation Object
When it is finally displayed to the user, Citation should not just E1. It should be parsed into a readable source.
{
"citation_id": "E1",
"label": "退款政策 2026-03 版,跨境退款",
"source_id": "refund_policy_2026_03",
"quote": "跨境支付订单需进入人工审核,审核完成后按原支付渠道退款。",
"locator": {
"page": 4,
"section": "跨境退款",
"start_char": 1832,
"end_char": 1938
},
"url": "kb://policies/refund/2026-03#page=4",
"updated_at": "2026-03-18"
}
4.4 Audit Record
When there is a problem online, you can't just look at the answers that the user sees. To be able to restore what the system saw at that time.
{
"trace_id": "rag_20260619_000431",
"user_id_hash": "u_...",
"query": "跨境订单退款多久到账?",
"retrieval": {
"queries": ["跨境订单退款多久到账", "跨境支付退款人工审核"],
"filters": {"department": "customer_service"},
"top_k": 8,
"result_ids": ["E1", "E2", "E3"]
},
"answer_contract": "...",
"citation_check": {
"unsupported_claims": [],
"missing_citations": [],
"stale_sources": []
},
"model": "production-answer-model",
"generated_at": "2026-06-19T17:57:23+08:00"
}
Pay attention to privacy and compliance: The log does not necessarily save the full user identity or sensitive text, but you must save enough hash, source id, version, reference and verification results to facilitate review.
5. Let the model output a verifiable answer
Models are good at writing natural language, but natural language is not suitable for direct auditing. An available generation instruction should clarify three things.
5.1 Only allow references to the IDs in the Evidence Pack
你只能引用 evidence_pack 中出现的 evidence_id。
不要编造 citation id。
如果没有足够证据,设置 insufficient_evidence=true,并说明缺什么证据。
This can reduce the source of the model to create out of thin air.
5.2 Each critical assertion must have CITATIONS
关键断言包括数字、日期、条件、例外、步骤、限制、权限、法律/财务/医疗影响。
每个关键断言必须放入 claims 数组,并绑定一个或多个 citations。
不要把多个无关事实塞进同一个 claim。
This reduces the question of "there is only one reference in the entire answer".
5.3 Downgrade when uncertain
如果证据之间冲突,必须指出冲突来源,不要自行选择一个结论。
如果证据不足,必须输出 insufficient_evidence=true。
如果答案需要外部最新信息,而 evidence_pack 没有,必须说明无法基于当前资料回答。
This reduces "referenced illusions of packaging".
6. How to do Claim-to-Citation
After the model output is referenced, the posterior verification is also required. A basic checker can be divided into four steps.
6.1 Parsing CLAIMS
Let's split the answer first into claim. The most stable is to let the model output directly claims array; if the history system can only output Markdown, it can post-process the key assertion.
6.2 Check whether the Citation ID exists
Check if each Citation is from the Evidence Pack. Any ID that does not exist is a hard error.
6.3 Check if the quote supports claim
There are three levels of support relationships:
| Method | When to use it | Limitations |
|---|---|---|
| String/field rules | Exact numbers, dates, order status | Unfriendly to rewriting |
| structured comparison | Forms, databases, contract fields | Structured when ingested |
| LLM judge / NLI | Open text, complex induction | Calibration and sampling manual review required |
Although the OpenAI Graders document is in the platform migration period, the ideas are still valuable: the evaluation can be combined in String Check, Text Similarity, Score Model, Python Grader and other methods. For RAG references, you can also combine rule checks and model checks.
6.4 Failure CLAIM to do policy processing
Don't just use the checker as a log. It should affect product behavior:
| check result | Recommended processing |
|---|---|
| Citation ID does not exist | Refuse to show this answer, retry or downgrade |
| claim no citation | Delete the claim or ask for a model rewrite |
| Citation does not support claim | Delete, rewrite or mark as insufficient evidence |
| source expired | Retrieve with the new version, or prompt version conflict |
| source is more than right | Block now and log security events |
In high-risk scenarios, the Citation Check failure should not be "sickly online".
7. Complex Scenarios: Multi-source, Conflict, Table, Summary, and Permissions
7.1 Multi-source support
Some conclusions require common support for multiple evidence.
{
"claim_id": "C3",
"text": "跨境退款需要人工审核,审核后仍按原支付渠道退回。",
"citations": ["E1", "E2"],
"support_type": "joint"
}
This CLAIM cannot just check a single Citation. To allow Joint Support: E1 supports "manual review", and E2 supports "original payment channel return".
7.2 Conflict evidence
If two sources conflict, don't let the model be silenced.
{
"claim_id": "C4",
"text": "关于跨境退款到账时间,当前资料存在版本冲突。",
"citations": ["E2", "E7"],
"conflict": true,
"resolution": "E2 是 2026-03 版,E7 是 2025-10 版;应优先使用 E2。"
}
Conflict is not a bad thing. The bad thing is that the system pretends not to conflict.
7.3 Table and Database Fields
Table references should not just refer to the entire table. It is best to save cells, row and column names, and filter criteria.
{
"evidence_id": "E12",
"source_id": "pricing_table_2026_q2",
"locator": {
"row_key": "enterprise",
"column": "sla_credit",
"sheet": "SLA"
},
"value": "10%",
"quote": "enterprise / sla_credit = 10%"
}
Otherwise, the user will open a large table and still do not know where the answer is.
7.4 Abstract References
Summary is most likely to have the problem of "every sentence seems to have a basis, but the original text cannot be found".
The recommended practice is:
- First establish a citation for the original evidence;
- Abstract sentences refer to the original evidence, not the intermediate products of the abstract;
- If you have to refer to the intermediate summary, also keep the source ids used when the summary is generated.
7.5 Permission and Tenant Isolation
The reference system cannot disclose the source title, file name, or quote that the user does not have permission to see.
Error example:
根据 VIP 客户折扣表,ACME 的折扣为 27%。
Even if the answer does not expand the document content, the filename itself may leak sensitive information.
The correct way is to retrieve the forced permission filtering before retrieving, and check the Citation again before displaying. access_scope. When permissions fail, the Citation Label should not be sent to the frontend.
8. How to display references in UI
A good reference UI does not stack footnotes at the end of the answer, but allows users to quickly check.
8.1 Inline citation
Reference tags are displayed next to the key sentence:
跨境支付订单需要进入人工审核 [1]。审核完成后,退款通常会在 1-5 个工作日内退回原支付渠道 [1][2]。
Mark to be able to click. After clicking, the source card is displayed.
8.2 Source drawer
Source cards show at least:
- document title;
- version;
- update time;
- section/page;
- original quote;
- Permission range or visibility statement;
- "Open original text" link;
- "Which sentence should the reference support".
8.3 The original text is highlighted
If the source is a PDF, web page, knowledge base page, click Citation to jump to the original location and highlight the reference clip. Users should not manually find answers in PDF 80.
8.4 Uncertainty display
If there is insufficient evidence or conflict, do not pretend to be determined with a green tick.
Three states can be used:
| Status | Explanation |
|---|---|
| supported | Reference fragment directly supports claim |
| partially support | References to support some facts, still missing another part |
| Insufficient evidence | There is not enough source to support |
8.5 References do not block reading
The purpose of citing UI is not to show off. The text should be smooth, and the evidence should be able to unfold. In particular, it is necessary to avoid interrupting reading of large footnotes in particular on the mobile terminal.
9. Online logs and audit records
After the RAG error, I am most afraid that there is only one user complaint:
它引用了错误政策。
But you don't know at the time:
- What is the user problem;
- What is the retrieval of the query that is rewritten;
- What is the top-k result;
- what evidence is in the context;
- What Claims are output by the model;
- Whether the Citation Checker is passed;
- what version of the knowledge base was at that time;
- The front end shows what sources finally show.
So the online trace records at least:
| Field | Use |
|---|---|
trace_id |
Associate user feedback, logs and reviews |
query |
original question |
rewritten_queries |
Review search |
filters |
Review permissions and versions |
retrieved_evidence_ids |
See Recall and Sort |
context_evidence_ids |
Look at the evidence of the final model |
answer_claims |
See assertion split |
citations |
See reference relationship |
citation_check_result |
See if you are showing sickness |
source_hashes |
Review Knowledge Base Version |
model_and_prompt_version |
regression test |
If the data is sensitive, save the hash and locator, and if necessary, re-fetch the evidence through the controlled background. Don't turn the system into unauditable for logging.
10. How to Evaluate Reference Quality
ALCE evaluates long-form answers with citations on dimensions including fluency, correctness, and citation quality. WebGPT collects references to help users assess factual accuracy. RARR explores adding attribution to existing model outputs and revising unsupported content. Together, these studies make a case for measuring citation quality separately, since an overall answer satisfaction score can conceal problems with the sources.
A set of practical indicators can be designed as follows:
| Metric | Meaning |
|---|---|
| Citation Coverage | The ratio of citation in the key claim |
| Citation Support Rate | The ratio of CITATION actually supports claim |
| Unsupported Claim Rate | Claim ratios that are exported without evidence |
| Missing Citation Rate | Claim ratio that should be referenced but not referenced |
| Citation Precision | The true proportions in the cited evidence |
| Citation Recall | Proportion of reference in Gold Evidence |
| Over-citation Rate | Proportion of a simple claim binding too many weak related sources |
| Stale Citation Rate | Proportion of citing expired sources |
| Permission-safe Citation Rate | Reference ratios without an overpowered source |
The evaluation set covers these samples:
- Direct support for documents;
- Joint multi-evidence support;
- Reference partial support;
- References are not supported;
- The answer is correct but the reference is wrong;
- The answer is wrong but the reference seems relevant;
- conflict between old and new versions;
- Insufficient permissions;
- no answer questions;
- table field problem;
- Summary question.
If you don't have a sample of "wrong answer but wrong reference" in your eval, the system can easily learn to use relevant documents as decoration.
11. An example of a minimum validator
The following example only does basic rule verification: whether the Citation ID exists, whether the claim has a reference, and whether the reference quote quotes the key phrase in the claim. In the real system, semantic entailment, field alignment, LLM Judge and manual sampling are also required.
import json
import re
from pathlib import Path
def normalize(text):
return re.sub(r"\s+", "", text.lower())
def keyword_overlap(claim, quote):
claim_norm = normalize(claim)
quote_norm = normalize(quote)
# 真实项目应使用领域词表或实体抽取,这里只保留一个最小示例。
tokens = re.findall(r"[\u4e00-\u9fff]{2,}|[a-zA-Z0-9%.-]+", claim)
important = [t for t in tokens if len(t) >= 2]
if not important:
return 0.0
hits = sum(1 for token in important if normalize(token) in quote_norm)
return hits / len(important)
def check_answer(answer_contract, evidence_pack, threshold=0.5):
evidence_by_id = {item["evidence_id"]: item for item in evidence_pack}
errors = []
warnings = []
for claim in answer_contract.get("claims", []):
claim_id = claim.get("claim_id", "<missing>")
claim_text = claim.get("text", "")
citations = claim.get("citations", [])
if not citations:
errors.append({
"claim_id": claim_id,
"type": "missing_citation",
"message": "关键断言没有引用"
})
continue
support_scores = []
for citation_id in citations:
evidence = evidence_by_id.get(citation_id)
if evidence is None:
errors.append({
"claim_id": claim_id,
"type": "unknown_citation",
"citation_id": citation_id
})
continue
score = keyword_overlap(claim_text, evidence.get("quote", ""))
support_scores.append(score)
if support_scores and max(support_scores) < threshold:
warnings.append({
"claim_id": claim_id,
"type": "weak_support",
"best_score": round(max(support_scores), 3),
"message": "引用可能不足以支持该断言,需要语义校验或重写"
})
return {
"passed": not errors,
"errors": errors,
"warnings": warnings
}
if __name__ == "__main__":
answer_contract = json.loads(Path("answer_contract.json").read_text(encoding="utf-8"))
evidence_pack = json.loads(Path("evidence_pack.json").read_text(encoding="utf-8"))
print(json.dumps(check_answer(answer_contract, evidence_pack), ensure_ascii=False, indent=2))
This checker is rough, but it illustrates an important principle: References must be checked by the program, and cannot fully believe the model itself says "I'm referring".
For high-risk scenarios, you can add another LLM Judge:
你是引用校验器。判断 Claim 是否被 Evidence 直接支持。
Claim:
{{claim_text}}
Evidence:
{{quote}}
只输出 JSON:
{
"supported": true | false | "partial",
"reason": "一句话说明",
"missing_fact": "如果 partial 或 false,说明缺什么"
}
LLM Judge is also evaluated. Don't make Judge a new uncontrollable black box.
12. Common Mistakes
12.1 Reference is equivalent to trust
References may be irrelevant, expired, overpowered, or taken out of context. The real thing to check is Support Relation.
12.2 Only the source is listed at the end of the answer
The list of sources at the end is suitable for further reading and is not suitable to prove each key conclusion. The key assertion requires inline citation.
12.3 Let the model compile the Citation ID by itself
The Citation ID must come from the Evidence Pack provided by the system. The model cannot invent the source.
12.4 Reference the entire PDF
A citation to an entire 80-page PDF is too broad. Identify at least the section, page, or chunk. In high-risk settings, point to the exact quotation or supporting span.
12.5 Ignore tables and structured data
Many business answers come from tables, database fields, or configuration items. Don't throw away the row and column information after converting them into a large piece of text.
12.6 Reference UI cannot be clicked
Non-clickable references cannot be checked. Users can ultimately choose to believe or not to believe the model.
12.7 Do not save the current source version
The knowledge base will be updated. The source that is clicked today is not necessarily the source that the model saw when it answered yesterday. Audit logs must be logged in version or hash.
13. Landing road map
Week 1: Standardize the Evidence ID
- Assign short IDs to each chunk that enters the context;
- Keep source ID, title, version, quote, locator;
- Only these IDs are allowed in Prompt.
Week 2: Change the Output Contract
- Let the model output
answer + claims + citations + insufficient_evidence; - granular constraints on key assertions;
- Insufficient evidence is allowed to fail, rather than forcibly answering.
Week 3: Add Reference Checker
- Check if the Citation ID exists;
- Check if each key claim has a reference;
- Check if the source is expired or overpowered;
- Rewrite, delete, or downgrade the weakly supported claim.
Week 4: Change the UI and logs
- Inline Citation is clickable;
- Source drawer displays quote and locator;
- original text highlighting;
- Trace saves Retrieval, Context, Claims, Citations, Checker Result.
Week 5: Do Reference Quality Eval
- Mark a batch of claims-to-evidence samples;
- Evaluation of Citation Coverage, Support Rate, Unsupported Claim Rate;
- Return user feedback and online failure samples.
14. Pre-release self-check list
At least check these questions before going online:
- Is the evidence of each entry context stable?
evidence_id? - Can the model only reference the ID in the Evidence Pack?
- Whether the key assertion is broken into
claims? - Is there a citation for each key claim?
- Can Citation resolve to Source, Quote, Locator and version?
- Is there a checker checking the Citation ID, missing references, weak support, expiration and overpower?
- Will the Citation Check fail, rewrite, or downgrade instead of showing it as usual?
- Can the front end click the reference and highlight the original text?
- Does the audit log hold source hash, model version, prompt version, and checker results?
- Does eval set contain "what is correct but the reference is wrong", "the answer is wrong but the reference seems to be relevant", "multi-source joint support", "no answer" and "insufficient permission" samples?
If these questions cannot be answered, the reference is still just the format, not the engineering ability.
15. Summary
RAG's reference and attribution is not to add a few footnotes to the answer. It is a complete link from data ingestion, retrieval, context structure, model output, post-check, front-end display to audit logs.
The minimum usable design can be summarized as:
Source -> Evidence Span -> Evidence Pack -> Claim -> Citation -> Support Check -> UI Highlight -> Audit Record
As long as this link is disconnected, the RAG will return to the state of "seemingly sourced, but cannot be verified".
The next article recommends continuing to write: How to do the knowledge base of RAG with Chunking: from the original document to the retrievable unit of evidence.
References
- OpenAI Developers: File search
- OpenAI Developers: Retrieval
- OpenAI Developers: Web search output and citations
- OpenAI Developers: Message annotations
- OpenAI Developers: Graders
- Gao et al., Enabling Large Language Models to Generate Text with Citations
- Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Nakano et al., WebGPT: Browser-assisted question-answering with human feedback
- Gao et al., RARR: Researching and Revising What Language Models Say, Using Language Models