For graduate students, early-career researchers and research engineers; search and experiment date: 2026-09-16.

Concept cover of scientific research installations for two-hop evidence transmission

Abstract

Before connecting the knowledge graph to the Large Language Model, ask a small question: The answer entity has been retrieved, and the path to support the answer is also retained? This article reads the official implementation of G-Retriever, and then trains a graph neural network with ten parameters, and compares the complete structure, reverse edge and different evidence clipping methods. The actual results of the three seeds show that the answer node recall is very high, and the reading of evidence may still fail. The experiment only verifies the graph messaging and evidence interface, and does not run the real language model, nor does it reproduce the paper list.

Contents

  1. Reproduce value: distinguish nodes, paths and answers
  2. Core ideas and formulas
  3. A guide to the official implementation
  4. A minimal experiment and its results
  5. Evaluation protocol and independent audit
  6. Troubleshooting and limits of the evidence
  7. Questions for further research
  8. Summary and References

Reproduce value: distinguish nodes, paths and answers

The graph neural network, that is, GNN, passes and aggregates node information through edges; large language model, that is, LLM, processing text and its vector representation. When the two are combined, the graph can provide a relational structure, and the language model can explain the problem and generate the answer, but the connection of the interface does not mean that the reasoning is reliable. The node names are similar, the answer entity is selected, and the complete evidence path exists, which are three different conditions.

Set the problem to start from A, first follow the "cooperation" and then follow the "job" relationship to find the end point. Only give the names of the model armor and the end point, the middle character and the two directed edges may have been lost. At this time, the answer is correct, which may come from the parameter memory; the answer is wrong, and it cannot be immediately blamed on the ability of language reasoning. The value of reproduction is to disassemble these possibilities into checkable input and output.

Paper report: QA-GNN of Yasunaga et al. establishes a joint graph between the Q&A context and the knowledge graph, and is updated through the messaging, published in 2021 NAACL. It is the precursor of the joint reasoning of language model and graph, and is not equal to today's generative chat model. Paper page

G-Retriever is oriented to graphs with text attributes, combined with subgraph retrieval, graph coding and language generation. This article uses the third edition of the preprint revised in May 2024; the attribution given by the official repository is NeurIPS in the same year. The cross-version score is not quoted here, nor does it claim that the search can eliminate all hallucinations. Version page; official statement

Core ideas and formulas

Problem condition retrieval and graph prompts, textual evidence dual path

The method diagram shows the configuration of the frozen language model of the original method: the same subgraph forms a soft prompt and text evidence for the same subgraph, and the problem is also sent to the language model. Soft prompts are continuous vectors, not sentences written manually. The lock in the figure only means that the parameters of the language model are frozen, and the figure encoder and projector can still be learned; this picture is not a screenshot of the running of the toy experiment below.

The original method combines the correlation with the connected cost, and selects the subgraph with the Steiner problem with reward. In layman's terms, high correlation nodes are worth keeping, but there is a price to connect them, so search cannot be understood as selecting several entities independently. The graph then represents the word vector space that is projected into the language model, and the textualized node and edge provide another path to evidence. Full paper

In order to isolate the structural problem, this time, the two-layer relationship gate control message transmission is adopted:

h(0)=es,h(ℓ)=h(ℓ−1)∑r=01αqℓ,r(ℓ)Ar,α(ℓ)=softmax⁡r(W(ℓ)).h^{(0)}=e_s,\qquad h^{(\ell)}=h^{(\ell-1)}\sum_{r=0}^{1}\alpha^{(\ell)}_{q_\ell,r}A_r, \qquad \alpha^{(\ell)}=\operatorname{softmax}_{r}(W^{(\ell)}).

Each graph has twelve nodes, ArA_r is the shape [12,12][12,12] The direct adjacency matrix, the row represents the starting point and the column represents the end point; ese_s It is the source node single hot row vector. The question gives an orderly relationship q1,q2q_1,q_2, two WW all [2,2][2,2] trainable parameters. Each layer is mixed with two types of edges according to the problem relationship; h(2)h^{(2)} It is a twelve-dimensional weighted path count, and there is no degree normalization.

zv=exp⁡(a)hv(2)+b,L=1BN∑i,v[log⁡(1+exp⁡ziv)−yivziv].z_v=\exp(a)h^{(2)}_v+b,\qquad \mathcal L=\frac{1}{BN}\sum_{i,v}\left[\log(1+\exp z_{iv})-y_{iv}z_{iv}\right].

zvz_v is the unnormalized score of the node belonging to the set of answers, yy Indicates whether there is a two-hop path in accordance with the order, BB for figures, N=12N=12. scalar quantity a,ba,b Add eight gate parameters, a total of ten parameters. To achieve the use of stable logarithmic addition to calculate binary cross entropy; a question can have multiple answers, or there can be no answers, and cannot be replaced by single-label classification.

A guide to the official implementation

Read the data first, then read the model, and finally read the training loop to avoid only looking at the schema name.

The first stop is webqsp.py. Check the questions, answers, cache diagrams, and text descriptions in the sample how to correspond by index. After training, validation, and testing, relying on the saved index division after splicing, the repeater must check whether the cache generation sequence is consistent; the splicing itself is not leaking evidence.

The second stop is retrieval.py. Node similarity is converted into a ranking reward, and the edge also involves virtual node conversion; when returning, the original node number will be re-mapped to the local number. The structure and text description should be checked at the same time to prevent the two evidences from different entities. After modifying the retrieval parameters, the cache should also be regenerated.

The third stop gnn.py and graph_llm.py. The former includes several graph encoders, which need not all use edge attributes. The latter mean-pools the node representations, projects the result, and concatenates a single graph vector with the text embeddings. The projection output width is set to 4096. Check the embedding width before swapping language models. Descriptions can be truncated, so evidence present in the graph may still be missing from the text branch.

Final check train.py Gradient clearing, optimizer step and validation mode selection. The current code is cleared and stepped in each batch, not just the parameter name grad_steps It is considered that gradient accumulation is realized. The above is the static source code verification, and the official environment is not run; these links point to the main branch on the day of retrieval, and there is no lock submission, and the follow-up compatibility is for manual verification.

A minimal experiment and its results

From the article directory, run:

python3 -m pip install -r code/requirements.txt
python3 code/experiment.py
python3 code/audit_results.py

This time, Python 3.12.14, NumPy 2.3.5 and CPU are actually used. Each seed generates 256 training pictures, 64 verification pictures, and 128 test pictures; each of the two types of edges is independently sampled with a probability of 0.12, and the self-loop is removed, and the source node and problem relationship are randomly selected. The three seeds have a total of 1,344 different pictures, and the node number is only valid in this picture.

To use full batch Adam, the learning rate is 0.03, and the learning rate is 500 steps; the verification loss is only recorded, and the checkpoint is not selected and the threshold is not adjusted. The training adjacency tensor is [256,2,12,12], the output is [256,12]. The training loss of seeds 75 has dropped from 0.3503 to 0.0000000541, and the probability of the two layers of correct relationship gating is about 0.98. Complete data, weights, curves and question-by-question evidence are all kept in code/results/.

This actual verification: the complete matching rate of the set of answers on the complete map, all three seeds are one; after freezing the weight, all sides are inverted, keep the original answer unchanged, and the average exact matching rate has dropped to 0.1302. When all edges are removed, the model outputs an empty set, and there is still an exact matching rate of 0.3359, because there is indeed an empty answer in the test set. The latter two are input damage diagnostics, not retrained, and cannot be regarded as a fair comparison independent model.

Then the first three candidates are selected according to the score of the node, and the two evidence packets are compared. The "Induce Subgraph" only retains the edge between the source node and the candidate; the "Bridge Reserve" holds all the out edges of the source node and all the edges of the candidate, allowing the intermediate node to remain in the evidence. The latter doesn't read the answer tag, but it will carry more triplets. The triple is the three fields of "starting point, relationship, and ending point".

The reader is a handwritten deterministic two-hop traversal, and can only return the end of the candidate from the given triple. It is not LLM, nor does it simulate natural language understanding; this allows failure to locate the evidence interface.

Three Candidate Evidence Packages Non-Empty Question Recall Answer micro-average f1 set exact match rate Average triple number
induction subplot 0.9312 0.1466 0.3724 2.43
Briefing reserved 0.9312 0.9256 0.8646 12.10

The tables are the mean of the three seeds. The same candidate does not equal the same evidence; the gain of the bridge scheme is accompanied by a larger input budget and cannot be written as the same cost advantage. When the candidates are increased to twelve, both schemes are restored to the complete map, and the exact matching rate of the reader is one, which is a protocol upper limit check.

Evaluation protocol and independent audit

The micro-average F1 combines the real, false positives and false negative cases of all the problems, which is suitable for multi-answer collections. The exact matching rate requires that the prediction set is the same as the answer set, including the empty set; the candidate recall is only calculated for non-empty questions, and then the question is averaged one by one, so as not to record the empty question as a perfect recall by default. These denominators must be reported at the same time.

The fluctuation of the three seeds comes from the resampled map and initialization, not Confidence interval . It is not surprising that the complete graph high score is only two hops, and the model structure just covers the real generation rules; it does not learn the solid semantics, and does not prove that the long path generalization is. The plots of training and testing are completely different, and the same graph generation distribution is still shared.

The independent audit script does not import training scripts, and uses a scalar loop to reconstruct all truth values and 13,824 points, the maximum difference The difference is about one by ten to the negative 14th power; another 3,072 evidence records are reproduced, and the candidate ranking, side source, answer and indicators are checked. The finite difference gradient check of ten parameters passes, and after re-labeling nodes, source nodes and labels, the predicted scores are consistent. Sorting the same points are broken by number, so it is not claimed that the candidate sets are also replaced by the same points.

Troubleshooting and limits of the evidence

The zero question of the seed 75, the relationship between the node three and the two times is zero to the node eleven, and the complete path is "three to five to eleven". The first three candidates are eleven, six, and two, and the answer is already ranked first, but there is no intermediate node five; the induction subgraph reader therefore returns to the empty set. This case is saved in the audit results, which is more capable of locating the problem than the sentence "retrieval effect is not good".

The investigation sequence should be: whether the relationship direction and type are correct, whether the node number is aligned, whether the bridge node is cut, and whether the text is truncated, and finally check the generator. If it is changed to true LLM, it is necessary to save the pure evidence input, generate text and verifiable references; the saved record will be prompt with rating gold Separation, the entire evaluation record cannot be sent into the model.

The authors inference: High node recalls are not enough to ensure that evidence is available. This zero-false positive case comes from the design that the reader only traverses along the real edge, and cannot be extrapolated as a generative model without illusions. At present, semantic embedding, Stanner tree solution, figure soft prompt training, official dataset or GPU are not running; these complete reproducing links are all manual verification.

Questions for further research

The next step is to fix the textual evidence budget, compare the plain text, picture tips, and the combination of the two to determine where the new income comes from. Then the evidence is controlled and synonymous rewriting: the former tests the structural dependence, and the latter tests the language robustness, and cannot be mixed with one disturbance. Finally, increase the three-hop problem, relational combination extrapolation and real drawing noise, and report node recall, path integrity rate, answer score and reference Accuracy .

These experiments need to be re-divided the development set and freeze the model, budget and evaluation rules in advance. In particular, don't observe the test error first, and then choose the most favorable way to delete the causal interpretation; this bridge case is a mechanism diagnosis, not to prove that a real language model uses the same internal reasoning.

Summary

A minimal replication of graph-augmented reasoning should track the candidate entities, the paths connecting them, and the input the reader receives. We verified one specific failure mode. Even when the correct node is selected, the path can still be cut off. Recording what passes through these interfaces before connecting a real LLM gives later replications a way to locate the failure.

References

The following first-hand information is 2026-09-16 It is actually opened; the source code is the main branch of the day, and the submission is not fixed.

  1. Yasunaga、Ren、Bosselut、Liang、Leskovec,2021: QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering,NAACL.
  2. He、Tian、Sun、Chawla、Laurent、LeCun、Bresson、Hooi,2024: G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering, arXiv third edition; Full text.
  3. G-Retriever author repository: Entrance to Environment and Experiment、data interface、subgraph search.
  4. The same official implementation: Graphic Encoder、Diagram and language interface、Training and model selection.