In the previous article, we have disassembled the boundaries of Function Calling, Tool Use and Agent.

If the tool call solves the "how the large model interacts with the external ability", then the RAG solves another more high frequency problem:

Even a strong model can give wrong or outdated answers, or sound as if it only half understands the question, when it encounters private knowledge, recent information, specialized documents, or lengthy business materials.

It's not that the model suddenly becomes stupid.

Instead, the parametric model naturally has several boundaries:

  • Training data is not updated in real time
  • Model parameters are not enterprise private knowledge base
  • Cost, length and attention dilution issues even if long documents are stuffed into the context
  • Fine-tuning alone does not effectively cover the business knowledge of high-frequency changes

So many teams will end up reaching the RAG.

But when it comes to RAG, many discussions are easy to be too simplified, as if it is just "retrieving it first, and then stuffing the data into the model".

When it was actually online, the problem came immediately:

  • Why is it obviously done vector search, but the answer is still not allowed?
  • How big is the chunk size? Should overlap open?
  • How to choose the Embedding model? Is the larger the better the better?
  • Is TOP-K the bigger the more stable, or the bigger the more chaotic?
  • Is reranker worth adding?
  • After searching a lot of materials, how to spell the context without wasting the token?
  • How to match the generation stage Temperature, Max tokens, and system prompt?

In this article, the RAG will be explained clearly according to the engineering link.

I will focus on four things:

  1. What is RAG making up, don't understand it as a "plug-in knowledge base" as simple as
  2. How to design the retrieval link: slicing, embedding, recall, and rerank what is responsible for
  3. How to adjust the key parameters: chunk size, overlap, top-k, embedding dim, max context, etc.
  4. Production pitfalls: accurate retrieval but poor answers, more results but less coherent context, and higher costs for little gain

If you are doing knowledge base quiz, enterprise document assistant, customer service quiz, paper retrieval, code knowledge assistant, RAG is basically impossible to get around.


First, let's talk about the conclusion: RAG is not to let the model "remember more", but let the model "see the data" before answering

When many people come into contact with RAG for the first time, they will understand it as a method of "expanding knowledge for the model".

This statement is not entirely wrong, but not precise enough.

The more accurate nature of RAG is:

Don't force the model to put all the knowledge in the parameters, but first find relevant materials from the external knowledge base when answering, and then generate answers based on the material.

Therefore, it does not directly modify the model parameters, but adds a process of "check first and then answer" during the inference stage.

This is different from the idea of fine-tuning.

  • fine-tuning is to write a certain type of ability or style into the model parameters
  • RAG is the dynamic injection of external knowledge in the AT Inference time context

Therefore, the problems that the two are good at solving are also different.

1) RAG is more suitable for the task of "knowledge frequently changes, many contents, and traceability is required"

Typical scenarios include:

  • Enterprise internal rules, product documentation, interface documentation
  • Legal Terms, Medical Guides, Paper Materials
  • FAQ, customer service knowledge base, after-sales manual
  • Code repository documents, architecture descriptions, change records
  • Data, announcements and reports that are updated every day

The commonalities of these questions are:

  • Knowledge is updated frequently
  • The original document is longer
  • References to the original text need to be referenced
  • Not suitable for re-Fine-tuning model every time you change a little bit

The advantages of RAG here are very direct:

knowledge and model decoupling.

If you update the knowledge base, you don't necessarily need to change the model; if you change the model, the knowledge base does not need to rebuild the business logic.

2) RAG is not a universal patch, it complements "external knowledge access", not all ability defects

To avoid another common mistake:

It's not that the model can't answer well, it's all relying on RAG to make up.

RAG is good at solving:

  • Knowledge of the old and the new
  • Private data access issues
  • Long document targeting problem
  • Evidence traceability problem

But it doesn't naturally solve:

  • Weak logical reasoning
  • Unstable multi-hop planning
  • Tool execution error
  • The output format is unstable
  • The hallucination disappears completely

If the bottom mold is too weak, the prompt design is too poor, the problem itself needs to execute SQL or the calling API, RAG alone cannot save it.

In one sentence:

RAG is to add "data" to the model, not the "intelligence of the model".


2. A complete RAG system is usually not a search, but a four-segment link

Many demos will draw RAG into two steps:

  • retrieve
  • Generate

But in real projects, it should be split into four layers:

  1. Data storage layer: document cleaning, slicing, metadata extraction, vectorization, indexing
  2. recall layer: Retrieve candidate clips based on user issues
  3. rearrangement: Screen out the most relevant small batch from the candidate fragment
  4. production layer: Organize fragments into Prompt and let the model answer based on evidence

As long as one of the layers is rough, the final answer will be obvious.

1) The data storage layer determines whether "knowledge can be found"

The storage is not the end of the PDF thrown in.

At least deal with these issues:

  • Can the document structure retain the header level?
  • Tables, code blocks, formulas, picture descriptions are lost
  • Is there a lot of header, footer, directory, duplicate watermark in the text
  • Whether different documents have metadata such as source, time, department, version number, etc.

If the original text is very dirty, it will be difficult to save no matter how advanced the search is later.

2) The recall layer solves "first screen out possible related content from the massive materials"

The recall phases are usually sought after:

It is better to find more candidates first, and don't miss the key evidence at the beginning.

So here is more emphasis on recall, rather than the final sorting is absolutely accurate.

Common practices include:

  • Sparse search: BM25, keywords reverse row
  • Dense search: embedding vector similarity search
  • Hybrid search: keyword recall + vector recall union or weighting

In many businesses, hybrid retrieval is often more stable than a single vector search.

Because keyword search is particularly effective for precise terms, numbering, error codes, and product models; vector search is better at semantic approximation and expression rewriting.

3) The reranking layer solves "there are many candidates, but there are only a few paragraphs that should be looked at most"

The first 20 paragraphs and the first 50 paragraphs recalled are often not thrown directly into the model.

The reason is simple:

  • Token is too costly
  • Too much noise can dilute key information
  • Too many similar fragments can make the context redundant

So usually reranker is also required.

reranker will do a more detailed sort based on the correlation of "Query + document fragments".

The difference between it and the embedding search is:

  • Embedding is more like a rough recall, pursuing fast and coverage
  • Reranker is more like a fine row, and the pursuit of the top few is really related

In a high-quality Q&A system, RRANKER is often one of the most cost-effective gain modules.

4) Generation layer solution "How to turn evidence into an answer"

Even if the retrieval is completely correct, the generation layer may make the answer bad.

Common causes include:

  • Prompt does not require "answer based only on given material"
  • Reference source is not required
  • Confusion sequence of context
  • The model starts to play freely after seeing too many irrelevant materials
  • Temperature too high leads to weak evidence constraints

So the last mile of the RAG is not as simple as "feeding the search results to the model", but:

How to design a generation environment where the model prioritizes the evidence, rather than the language fluency.


3. Why is the basic skill of RAG: the chunk is not cut correctly, and the back is full of remedies

Many RAG systems do not work well, and the root cause is not that the model is not good, but that the chunk strategy is too rough.

Because the basic unit of the search is not the entire manual, but the fragment you cut.

There are two types of problems:

  • The information that should be together is broken and incomplete after the recall
  • The information that should not be together is mixed together, and the noise is too loud after the recall

1) chunk size is not the bigger the better, not the smaller the better

chunk too small problem:

  • Incomplete context
  • refer to lost
  • A conclusion and its conditions, tables, and remarks are disassembled
  • The model lacks background when reading evidence

chunk too big problem:

  • Increased noise
  • A fragment contains multiple themes
  • After the retrieval hit, it will bring in a lot of irrelevant text together
  • Embedding means being averaged, and the degree of discrimination is reduced

The common starting point configuration in the project is:

  • General Chinese documentation: 300~800 tokens
  • FAQ or short description: 150~400 tokens
  • Technical manuals, paper paragraphs: 400~1000 tokens
  • Code and interface documentation: more suitable for structured segmentation by functions, classes, and chapters, rather than pure fixed length tiles

This is not an absolute value, but can be used as a baseline for the first round of experiments.

2) The meaning of overlap is to avoid the key information to be cut off

Many information exists naturally across paragraphs.

For example:

  • Defined in the previous paragraph, constrained in the next paragraph
  • The title is in the first paragraph, the details are in the next paragraph
  • A table description and the body of the table are cut

OverLap is important at this time.

Common experience points:

  • 10% to 20% overlap ratio is a more common starting point
  • Too small to make up for the boundary information
  • Too large, it will cause the fragments of the library to be highly duplicated and increase the recall redundancy

Well-structured documents may need relatively little overlap between chunks. When you convert a PDF to text and the paragraph boundaries are unreliable, it usually pays to leave a little more overlap.

3) More important than fixed length is the structural perception slicing

High-quality RAG systems are usually not hard-cut by characters or tokens.

A better strategy is usually:

  • Cut by title level first
  • Then cut by paragraph boundary
  • Excessive long section and then do a secondary split
  • Tables, code blocks, and lists try to keep the overall
  • Include metadata such as parent title, chapter path, source link, etc. to each chunk

Because when users ask questions, they often need not just a few sentences, but:

Which chapter, which section, which document, which version, which is this passage belongs to.

This structural information is valuable for subsequent rearrangement and generation.


4. What exactly does Embedding do: it is not "understanding everything", but maps the semantics to a retrieval space

The role of embedding can be simply understood as:

Turn the text into a vector, and make the content with similar semantics closer in the vector space.

In this way, after the user problem is also turned into a vector, you can do a near-neighborhood search.

1) Embedding model selection, first look at the field matching, then look at the cost and delay

Many people will first ask: Is the greater the dimension, the stronger?

Not necessarily.

The more critical ones are usually:

  • Is this embedding model suitable for your language
  • Is it suitable for your domain terms
  • How long text fragments behave
  • Inference cost and throughput is not acceptable

For example, Chinese, code, legal, and medical documents often have different preferences for the Embedding model.

If the terms density in the business is very high, domain adaptation is usually more important than simply pursuing "large dimensions".

2) Embedding Dim will affect precision, storage and retrieval efficiency

The higher the embedding dimension, usually means:

  • Vector indexes take up more space
  • Retrieve memory and bandwidth pressure is greater
  • Higher cost of building and updating

Sometimes it may also lead to stronger expressiveness, but not necessarily linearly.

So don't take dimensions as the only indicator.

A more reasonable idea is:

  • Compare Recall@K, MRR, nDCG on the same review set
  • At the same time observe the time of database building, query delay, memory usage
  • Choose a more balanced group of Effects - Costs

3) Whether query and document use the same encoding method will also affect the effect

Some embedding schemes use different templates or prefixes for query and documents.

If the encoding method is not aligned, the search quality will be significantly reduced.

Therefore, it is very certain to be very sure when landing:

  • Whether the inventory text and query text use the same set of preprocessing
  • Added recommended Instruction/Prefix
  • Whether the Chinese and English punctuation, blank, and code format have been over-cleaned

RAG is often not "model not working", but preprocessing ruins the semantics.


5. TOP-K, mixed search, RERANKER: how to adjust the recall parameters

The most common misunderstanding of retrieving parameters is to blindly pursue "it's always right to take a little more".

Actually not.

1) Top-k is not the bigger the better

The top-k problem is too small:

  • The real evidence may not be in the candidate collection
  • Multi-hop problem lacks supplementary background

The problem with the top-k too big is:

  • more and more noise
  • Reranker Cost Rising
  • Context selection is more difficult when generating
  • The model is more likely to be biased by the related content

Common starting points in engineering:

  • Retrieve only without reranking: top-k = 3~8
  • Bring Reranker after retrieval: the recall can be put into 20~50, re-exhaust 3~8 Segment into generation

That is:

  • Rough recall can be slightly larger
  • Restrain the amount of evidence that was finally fed to the model

2) Hybrid search is usually more stable for real business

Especially in these scenarios:

  • The user will ask the product name, interface name, error code, document number
  • There are exact phrases in query
  • Business terminology is fixed

If only vector retrieval is done, the model may find something "semantic but not that one".

So a common security approach is:

  • Retrieve one batch of candidates with BM25 or keyword search
  • Retrieve another batch using embeddings
  • Take the union of the results, weight them, or combine their rankings

This usually holds up better to noise in real applications than relying on vector search alone.

3) RERANKER is especially suitable for solving "recalls seem to be all related, but the priority is wrong"

The failure of many systems is not completely not recalled, but the most critical paragraph is not ranked in the front.

For example, the user asks:

"Can I get a refund after the course is opened in the refund rules?"

The recall results may be related to the refund, but the real key is the clause that contains the clause "Non-refundable after activation / refundable under a certain condition".

At this time, Reranker can often significantly improve the first hit rate.

If you can only add one module first, I usually recommend priority validation:

Mixed Search + Reranker Can you significantly lift the first 3 hits?


6. How to spell the context, it affects the final answer more than "what you retrieved"

Many teams do retrieving Accuracy is not bad, but users still feel that the answer is average.

The reason is often the context packing.

1) Do not mechanically splice all recall results

Common mistakes are:

  • Fill in the full text of 10 paragraphs directly according to similarity from high to low
  • Don't go heavy
  • Not organized by source
  • Whether contradictory or not

The problem with this is:

  • prompt is very long
  • Repeat a lot of information
  • Fighting with different documents
  • The real key sentence is buried in it

More reasonable practices usually include:

  • Adjacent chunks merge to reduce fragmentation
  • Repeat content deduplication
  • Prioritize aggregation with document fragments
  • Put the high credibility source in the front
  • If there is a time dimension, the latest version is preferred

2) Generate Prompt to be clear "the answer must be based on evidence"

The most important thing in RAG prompt is not "please answer the following questions", but constraint boundaries.

For example, it should be explicitly required:

  • Answer only based on the information provided
  • If the information is insufficient, it is clear that you do not know or the information is insufficient
  • Prioritize the original key statement of the original text
  • Give the source title or number
  • Avoid using the model itself that has not verified knowledge to fill holes

Because once the constraints are unclear, the model will mix "retrieval evidence" with "self-scheduling".

3) Max Context is not more valuable if it is full

The context window is very large, and it does not mean that it should be filled as much as possible.

The longer the context, the longer it means:

  • Prefill is more costly
  • More delay
  • Decreased evidence density
  • model attention distraction

In many tasks, 3 to 6 high-density valid evidence, which is better than a dozen loose materials.

Therefore, the goal of Context Packing is not to "stuff the window", but to "increase the density of evidence".


7. How to match the generated parameters: RAG should give priority to stability in the task, and do not give priority to fancy

RAG is not creative writing.

It is more like restricted generation based on evidence.

Therefore, generating parameters are usually more conservative than ordinary dialogues.

1) Temperature is usually recommended to be low

Common experience:

  • Factual Q&A: 0.0~0.3
  • Evidence summary and report extraction: 0.1~0.4
  • If a more natural expression is needed, but still based on the data: 0.3~0.5

When the temperature is too high, the model is more willing to "supplement more like human words", but it also means that it is easier to get rid of the evidence.

2) Max tokens should match the answer task

Too short will cut off the answer.

If it is too long, the model will continue to expand after the core answer is finished, increasing the risk of hallucination.

Can be dismantled by task:

  • FAQ category: 200~500 tokens
  • Document Q&A explanation class: 400~1000 tokens
  • Long abstract or multi-point induction: 800~1500 tokens

The point is not fixed numbers, but don't let it go without limit generation.

3) System prompt is best dealt with "don't know" strategies

A sign of a high-quality RAG system, not it can answer anything, but:

When the data is insufficient, it can honestly admit that it is not enough.

So system prompt should generally be explicit:

  • Don't guess when no evidence is found
  • In conflict with evidence, point out the source of conflict
  • If the problem is beyond the scope of the data, directly explain the scope limit

These constraints can significantly reduce the answer that "looks very complete, but is actually unreliable".


Eight, the four most common problems after the RAG goes online

1) The search indicators are good, but the user's answer satisfaction is still low

This usually means that the problem is not in the recall, but is generated.

To check:

  • Does Prompt encourage the model to play freely?
  • Is the context too redundant?
  • Whether the fragments that are finally sent to the model are sequentially confused
  • Is there a missing source reference and boundary constraints?

2) Recall is always a little bit, but not entirely wrong

This usually means you need to see:

  • Is the chunk cut broken?
  • Is embedding not suitable for domains
  • Should keywords be searched
  • Do you need reranker?
  • Does query need to be rewritten or extended

Many times, the problem is not in the model, but the recall link is too thin.

3) Costs and delays are too high

RAG is prone to "check a lot, fight a lot, and arrange a lot" in each problem.

To optimize, first look at:

  • Is the top-k too large
  • Is Reranker enabled for all requests
  • Is it possible to do a light recall first, and then press Confidence to decide whether to enter the fine discharge
  • Is chunk too broken and the number of candidates skyrockets?
  • Is it possible to cache popular Query or popular document vectors

RAG is not the more modules the better, but the more reasonable the link, the better.

4) The answer is cited, but the answer is still biased

This is usually because "retrieval correlation" is not equal to "answer available".

For example, the recalled fragment does talk about the same topic, but it is not the paragraph needed to answer the question.

Therefore, the evaluation should not only see whether it is recalled to the relevant documents, but also to see:

  • Whether the key evidence is in the top few
  • Does the final answer really use key evidence?
  • Whether the source reference is correct for the answer to the answer

9. Evaluation of RAG, you can't just look at the subjective feeling of the large model and feel "like the right"

The RAG system is particularly prone to being deceived by the Demo hallucination.

Because many answers are read smoothly, but it does not mean that it is really based on the correct evidence.

A more stable evaluation idea is usually divided into three layers:

1) Search layer evaluation

See recall quality:

  • Recall@k
  • Precision@k
  • MRR
  • nDCG
  • Whether the key evidence enters the candidate collection

2) Rearrangement layer evaluation

See if the most critical evidence can be ranked in the front:

  • TOP-1 / TOP-3 hit rate
  • Reranker sorting changes before and after
  • Whether the first evidence can directly support the answer

3) Generation layer evaluation

See if the answer is really loyal to the information:

  • Whether to refer to the correct source
  • Whether there is a conclusion that is not in the data
  • Will abstension when the data is insufficient
  • Whether the key points in the user question are fully covered

If the business is important enough, it is best to make a small set of labels yourself.

Don't just rely on "asking a few examples to feel okay".


10. When should you use RAG and when should you not be strong?

Suitable for RAG cases:

  • Large amount of knowledge, frequent updates
  • Documentation is the main source of information
  • Need traceable evidence
  • Business Q&A is highly dependent on private data
  • Don't want frequent Fine-Tuning models

Not suitable for RAG alone:

  • The essence of the task is execution, not question and answer
  • Problems rely on real-time database operations or business actions
  • Requires complex workflow orchestration
  • Outputs require strict structured or call external systems

Such scenarios should generally be:

  • RAG is responsible for finding information
  • Tool Use is responsible for performing actions
  • Agent is responsible for multistep coordination

That is, RAG is not the end point of all systems, it is more like a knowledge access layer.


11. Suggestion for landing: If you are going to be an enterprise knowledge assistant now, I would recommend starting this way

Don't pile up the most complex architecture as soon as you come up.

A more stable first version of the route is usually:

  1. Clean the document first: Don't lose the title, paragraph, table, metadata
  2. First do structural perception slicing: Don't just cut the fixed length
  3. First on the mixed search: Keyword + vector, generally more stable than pure vector
  4. Add Reranker: Focus on the top 3 evidence hit rate
  5. Generate parameters conservatively: Low Temperature, strong source constraints
  6. Do small-scale manual evaluation sets: Don't just look at the demo feel
  7. Optimization with latency and cost: Don't over-complicate the link in the first place

If you take these seven steps solidly, the availability of the first version of the RAG is usually one level higher than "stree the PDF directly into the large model".


12. Final summary

RAG is important not because it makes the model suddenly have infinite knowledge.

Instead, it provides a more knowledge-based way of using knowledge in engineering reality:

  • The model is responsible for understanding the questions and organizing the answers
  • The retrieval system is responsible for finding relevant materials
  • The rearrangement system is responsible for placing key evidence ahead
  • Prompt and Generation Parameters are responsible for constraining the model to be true to evidence

So an easy-to-use RAG system is essentially not "models can endorse", but:

Before answering, the model really looked at the correct information first, and only answered within the boundary of the evidence.

If you're doing a knowledge base question and answer right now, I'd recommend giving priority to three things:

  • chunk cut right
  • Whether the first few evidences are arranged correctly
  • Is the generation stage really restrained by evidence?

These three things often affect the results more than you change to a more expensive bottom mold.

In the next article, I will continue to go down this series and talk about a very core problem in the Vision-Language Model:

How does the image "access" the language model? What exactly are the visual encoder, projection layer and alignment mechanisms doing?

If you read this article with the previous Transformer, pre-training, and RAG, you will find that:

The multimodal system does not grow out of thin air. It is essentially outside the backbone of the language model, and then connects to new perceptual representation and alignment links.