Cover illustration

Date: 2026-08-20
For graduate students, early-career researchers and AI practitioners with an engineering background
Search date: 2026-08-20

Abstract

After reading ten papers, many graduate students get not a research proposal, but a longer Related Work: who did what, what model, and which Benchmark is better. The problem is that proposals are not the natural extension lines for literature review. It needs to compress the paper into a answerable question, a falsified assumption, a set of causal-pointing method selection, and a set of resources to bear the experimental plan.

This paper gives a workflow for AI research training: how to start from 10 papers, draw a map of the problem, construct a Gap Matrix, screen the research hypotheses, design the minimum experimental plan, and finally write it as 2-4 pages pre-proposal. This article referenced DARPA Heilmeier Catechism, NSF Proposal Preparation Guidance, ICLR 2026 Author Guidance, also combined with 2025-2026 about The latest information on LLM Research Ideation, AI Scientist, PaperBench, LiveIdeaBench, InciteResearch, and Proposal Writing. The point is not to let LLM "generate topic selection" for you, but to train you to read the paper as a research decision.

Contents

  1. Why is this topic important
  2. 10 papers are not a quantity requirement, but a boundary of evidence
  3. Core Process: From Paper Set to Proposal
  4. Representative paper and data road map
  5. Intensive reading of key papers: where can AI help, but not how to help
  6. Gap Matrix: Turning Reading Notes into Research Judgment
  7. Proposal draft structure
  8. Reproduction and pre-experimental recommendations
  9. Common misconceptions
  10. Research questions for graduate students
  11. Summary and References

Why is this topic important

The end point of the paper intensive reading training is not "I know what the work is in this direction", but "I can ask a small and clear question that can be tested and can be executed under the existing resources". This is the key step from the reader to the researcher.

In the AI direction, this step is especially difficult. First, the paper is updated too fast. If a proposal stays only on "improve a 2024 baseline", it may be swallowed by the 2025-2026 model or benchmark before the opening of the question. Second, many papers package contributions into system engineering: data, models, prompts, post-processing, training recipes and evaluation strategies, and it is difficult for readers to judge truly effective variables. Third, LLM has begun to participate in literature retrieval, idea generation, proposal writing and experimental execution, but the latest research also repeatedly reminds us that the model can give a seemingly reasonable Idea is prone to problems in feasibility, citation accuracy, diversity and real execution.

DARPA's Heilmeier Catechism is very suitable as the minimum threshold for proposals: what are you going to do? What to do today, where is the restriction? What is your new method and why is it possible to succeed? Who cares? What is risk, cost, time and mid-term/final acceptance? NSF's Proposal Guidance also compresses the core problem into a similar structure: what you want to do, what to do, how to do it, how to know success, and what value it has after success. For graduate students, these questions are more important than "whether the title is cool" and "whether the model is new".

10 papers are not a quantity requirement, but a boundary of evidence

"Forming Proposal from 10 papers" is not equal to random reading of 10 articles. The better composition is:

  • 2 foundation-laying paper: Define tasks, paradigms or core assumptions.
  • 3 recent strong baseline: representing the current mainstream practice, preferred technical reports for 2025-2026 or still being used in the current work.
  • 2 Counter-Example or Negative Results: Point out the problem of Benchmark, reproduction, generalization, or mechanism interpretation.
  • 2 adjacent branches: Provides transferable methods, but not simple stacks of the same set.
  • 1 review, systematic paper, or Benchmark paper: helps you understand the evaluation boundary.

The purpose of these 10 papers is not to cover all related work, but to establish a "sufficient judgment" boundary. After reading it, you should be able to answer four questions: What are the core variables in this direction? Which conclusions have been repeatedly verified? Which conclusions only hold in narrow settings? If I can only do a one-month pre-experiment, what is the most worth testing gap?

If the ten papers are all the SOTA paper of the same experimental routine, you can easily get a mediocre proposal: change the model, change the data, change the loss, and run the leaderboard again. A really training-value paper set should contain tension: one method claims to be effective, and the other re-examination research question it; one benchmark is popular, and another contamination or Leakage analysis points out the risk; an agent The framework automatically generates experiments, and another review shows that the Agent is still much lower than the human reproduction ability.

Core Process: From Paper Set to Proposal

Flowchart from 10 papers to Proposal

The first step is to do a Problem map. Don't rank by the timeline of the paper first, but by "where do they think the problem is". Each paper fills in three sentences: What is the failure mode to solve? What variable does it attribute the failure to? How does it think success should be measured? For example, in the direction of LLM Research Ideation, some papers define the problem as novelty, some are defined as feasibility, some are defined as groundedness, and some are defined as the execution success rate. On the surface they are all doing "generate research ideas", but the definition of the problem is not the same.

The second step is to do the claim extraction. Write the core claim of each paper as a verifiable sentence, not an abstract sentence. The wrong way is to "propose an automatic scientific research system". A better way of writing is "the system can complete idea generation, experimental iteration, writing and automatic review under the conditions of a given template and code base, but there are still problems of interpretation and reliability in the output paper". Once the claim is written as a verifiable sentence, the next step can be compared.

The third step is to do Gap Matrix. Rows are papers or methods, and columns are problem definitions, assumptions, data/tasks, model scale, experimental variables, metric, baseline, reproducing materials, failure cases, and external validity. The GAP you are looking for is not "no one has done X", but "the existing evidence cannot support an important judgment". For example: Idea Generation papers may prove that generating IDEAs are novel, but do not prove that they can be verified by low-cost experiments; Proposal writing papers may improve text quality, but still need to check references separately Validity; PaperBench shows that the AI agent is still difficult to reproduce complex AI papers, which limits the realistic credibility of the fully automatic proposal-to-paper system.

The fourth step is to generate 3 candidate hypotheses. Each hypothesis must contain directions, mechanisms, and falsic conditions. The weak hypothesis is "joining the search will improve the quality of research proposals". The strong assumption is that "in the task of generating AI Research Proposal from 10 papers, the documentary evidence is explicitly compressed into a Gap Matrix, which is better than the direct RAG generation proposal. and Reference Validity; if there is no significant difference in manual review or execution of pre-experimentation, the hypothesis is not true."

The fifth step is to design the minimum experimental plan. Proposal is not to fill all experiments, but to prove that you know how to get the first piece of evidence. Minimum experiments usually include a primary comparison, an ablation, a failure case study, and an artificial verification link. For AI paper training, in particular, it is necessary to write clearly SEED, data version, model version, prompt, reasoning budget, scoring rules and log storage methods.

The sixth step, write pre-proposal. Write 2-4 pages first, do not write 15 pages directly. The order is: problem, existing evidence, core Gap, hypothesis, method, experiment, risk, timeline. As long as these pages are not clear, the long proposal will only hide the problem.

Representative paper and data road map

Category representative data Inspiration for Proposal training
Proposal Judgment Framework DARPA Heilmeier Catechism;NSF Preparing Your Proposal Press the topic into goals, status quo, innovation, value, risk, cost, acceptance
Dinghui Writing Specification ICLR 2026 Author Guide;NeurIPS 2026 review guidance Focus on contribution types, reproducibility, anonymous code, ethics and experimental support
AI research ideation Si et al., ICLR 2025;Measuring the Gap Between Human and LLM Research Ideas, 2026 LLM Idea can be novel, but FEASIBILITY, DIVERSITY and HUMAN TASTE GAP need to be evaluated separately
automatic research system The AI Scientist / AI Scientist-v2;PaperBench Automatic generation and reproducing still subject to execution, code, experiment and review reliability
Idea benchmark LiveIdeaBench;InciteResearch / TF-Bench Scientific creative evaluation should be split into Originality, FEASIBILITY, FLEXIBILITY, Clarity and other dimensions
Proposal writing Assisting Research Proposal Writing with LLMs, 2025 The quality of the text and the authenticity of the reference should be evaluated separately, and iteration prompts cannot replace manual verification

Intensive reading of key papers: where can AI help, but not how to help

The ICLR 2025 study of Si, Yang and Hashimoto is the entrance to this topic. It recruited more than 100 NLP researchers to participate in Idea writing and blind review, and found that LLM generation ideas can surpass human experts in novelty evaluation, but the feasibility is slightly weaker, and exposes LLM self-evaluation The problem of failure and generation of insufficient diversity. The direct inspiration for Proposal training is: don't take "novels" as the sole goal. A graduate proposal can easily become a beautiful but unverifiable fantasy if there is only Novelty, there is no standard for executable experiments and failures.

2026 Measuring the Gap Between Human and LLM Research Ideas go further. It not only evaluates a single idea, but compares the distribution of Human papers with LLM IDEA differences. The paper pointed out that LLM Idea is often more concentrated on Bridge-like Opportunity and Synthesis method, and the distribution of Human Research Ideas is more scattered. This conclusion is important for those who use LLM-assisted topics: the model may continue to give you advice on "combining A and B", but the real Research opportunities may also come from abnormal phenomena, measurement gaps, negative results, mechanism interpretation or constraints.

The Ai Scientist and AI Scientist-v2 demonstrate The possibilities of The Agentic Research Pipeline: generate hypotheses, write code, run experiments, analyze results, and write papers. The GitHub README of AI Scientist-v2 also clearly reminds that The system will execute The code written by LLM, which needs to run in a controlled Sandbox, and v2 is not necessarily better than templated v1. The output is better, especially in The task of having a strong template. This is a hard constraint on The proposal design: automation can help explore, but proposals still need to write clearly which steps are verified by people, which experiments can be repeated, and which failures trigger Pivot.

PaperBench offers a reality check. Released by OpenAI in 2025, it asks agents to reproduce 20 ICML 2024 Spotlight or Oral papers from scratch. They are scored with hierarchical rubrics developed with the paper authors. In the published results, even the best agent tested at the time did not surpass the human baseline. The rubrics offer a lesson beyond the scores. A research plan should break “reproduce the paper” into tasks that can be assessed separately, including data preparation, model implementation, training objectives, core experiments, ablations, figures and tables, and checks on the conclusions.

LiveIdeaBench and InciteResearch are suitable to remind us: Proposal is not a simple convergent reasoning. LiveIdeaBench Measure Scientific IDEA with dimensions such as Originality, Feature, Fluency, FlexiBility, and Clarity Generation shows that scientific creativity is not equivalent to general intelligence scores. InciteResearch emphasizes that many research starting points are not a clear problem, but "hidden friction": you feel that something is wrong in a certain direction, but you can't say the problem. This is exactly the meaning of reading 10 papers: instead of generating answers immediately, it is a research question that turns fuzzy dissatisfaction into a reviewable research question.

Finally, Assisting Research Proposal Writing with LLMs discusses Proposal Writing directly. It evaluates the content quality separately from the Reference Validity, and uses iteration Prompting to improve the quality of Writing and reduce reference errors. This setting is very practical: when you use the LLM to assist Proposal draft, you should keep at least two tables, one to evaluate the argument structure, and one to check whether the reference is true and whether the corresponding sentence is supported.

Gap Matrix: Turning Reading Notes into Research Judgment

Below is a Gap Matrix template that can be reused directly.

Field What to fill in judgment criterion
Problem The specific failure mode of the paper resolution Is it more specific than "the large model is not good enough"
Assumption The prerequisite for the method to establish dependence Can it be broken by experiments?
Intervention Variables that really change methods Confused with engineering details
Evidence Evidence for Claim Whether it contains baseline, ablation, statistical or manual verification
Boundary Evidence-applicable boundaries Whether to describe data, models, tasks and budget constraints
Missing Questions still unanswered Is it important and experimental
Feasibility Can you do pre-experiment in 2-4 weeks Whether data, computing power, code, evaluation can be obtained
Proposal Hook Formable research hypotheses Is there a mechanism and falsified condition

After completing the matrix, don't rush to choose the "maximum" GAP. Postgraduate training is more suitable for GAP to choose "medium-important, low to medium cost, clear evidence chain, and can get pre-experiment results in one month". The goal of a small proposal is not to defeat the entire field, but to build a starting point of evidence that can be seriously discussed by a tutor, classmate or reviewer.

Proposal draft structure

It is recommended to write a page version first:

  1. Title: Must contain tasks, variables, and goals, not just write a general direction.
  2. Problem: Use 3-5 sentences to explain which specific scenario the existing method fails in.
  3. Literature Evidence: List 10 papers together to support this problem.
  4. Core GAP: Explain what is lacking in the existing evidence, not just "no one has been studied".
  5. Hypothesis: written as a falsic sentence.
  6. Method: Describe which variable you change, and why this might solve the GAP.
  7. Experimental plan: column dataset, metric, baseline, ablation, manual verification and log.
  8. Risk and Alternative Plan: Write clearly how to shrink the problem or pivot after failure.

When expanding into a 2-4 page version, you can increase the Related Work Map, Pilot Experiment, Timeline and resource budget. Referring to the idea of NSF, Proposal should answer "what you want to do, why, how to do it, how to know success". Referring to the idea of DARPA, it is also necessary to answer "who cares, what is the risk, and what is the phased test". For AI research training, the phased test can be very specific: the baseline is reproduced within two weeks; the main comparison is completed in the third week; the ablation and error are completed in the fourth week Analysis; if the main indicator does not improve, then turn to explaining the cause of failure, rather than continuing to pile up skills.

Reproduction and pre-experimental recommendations

First, first reproduce a trusted baseline, rather than implementing your new method first. Baseline reproduction fails, usually explains the environment, data, metric, or understanding problems. The logs at this stage should contain commit, dependency version, seed, model checkpoint, data hash, run commands and exception records.

Second, identify the smallest intervention that tests the proposal’s central hypothesis. Suppose the hypothesis is that a gap matrix improves proposal feasibility. Keep the model, prompt, retrieval collection, and review criteria fixed. Using the same 10 papers, model, and output format, compare a proposal generated directly with RAG against one generated by constructing a gap matrix first. Have people score both on specificity, feasibility, support from the cited sources, and testability.

Third, keep negative results. The real value of many proposals is not "the method is immediately effective", but the boundaries of the existing claim are found. Meetings such as NeurIPS 2026 have been given to Negative Results, Concept & Feasibility, Evaluation Analysis in the contribution types Leave room; as long as the experimental design is rigorous, the negative result can be a strong topic.

Fourth, don't take LLM Judge as final evidence. You can use LLM for initial screening, structural inspection and reference reminders, but the Reference Validity, experimental executability and key claims must be manually verified. The reason why PaperBench has training value is precisely because it splits the reproduction task into a hierarchical Rubric, rather than just asking the model "whether this reproduction is successful".

Common misconceptions

Misunderstanding 1: Write proposals as Related Work. Related work explains the past, and Proposal must bet on a future verifiable judgment.

Myth 2: Understand the GAP as "no one has done it". Many things that no one has done are unimportant, impossible, or meaningless. A better GAP is "the existing evidence is not enough to support an important claim".

Misunderstanding 3: Only pursue novelty. Both Si et al. and subsequent IDEation studies suggest that Novelty is not the same thing as FEASIBILITY, DIVERSITY, and EXECUTION SUCCESS.

Myth 4: The experimental plan is too large. The first phase of Proposal should not rely on training the 70b model or building a new Benchmark. Prove that key variables are worth studying.

Misunderstanding 5: References are not verified. The smoother the Proposal text generated by LLM, the easier it is to hide the reference mismatch. References must correspond to claim one by one.

Myth 6: There is no failure standard. Proposal, which does not fail the standard, is not a research plan, but a wishlist.

Research questions for graduate students

  1. Gap-matrix-first Proposal Writing: Compare the impact of Structured Gap Matrix and direct RAG on Proposal feasibility, reference accuracy and testability.
  2. Research Idea Diversity Audit: Reproducing the idea of Measuring the Gap Between Human and LLM Research Ideas, and analyzing the LLM idea in a certain AI sub-field Whether it is excessively biased towards "method fusion".
  3. Proposal Reference Validity Benchmark: Build a small Chinese/English AI Proposal dataset to evaluate whether the LLM cited a real paper and whether the citation supports the corresponding judgment.
  4. PaperBench-style proposal Rubric: Split a proposal into a scoring task to study whether this Rubric improves the quality of the tutor's feedback.
  5. Negative-result-driven Proposal Mining: Automatically extract executable follow-up ideas from reproduction failures, benchmark traps, and ablation negative results.
  6. Human-AI Proposal Co-Writing Protocol: Design a process to determine which steps are assisted by LLM and which must be verified by human researchers.

Summary

From 10 papers to the core of proposal, it is not "the more you read, the better", but the literature has become a decision. You need to extract the problem definition, core hypothesis, method intervention, evidence intensity and unsolved GAP from the paper, and then press one of the GAPs into a falsified hypothesis and a minimum experimental plan.

A qualified graduate proposal should convince the reader at least three things: the question is true and important; your method is chosen with a mechanism; the experimental plan can give a clear answer to the limited resources. LLM can help retrieve, organize, generate candidate hypotheses and check structures, but it cannot take the final judgment for you. The real training is: can you explain why this is the problem, why this experiment, and what you will learn when you fail.

References

Search date: 2026-08-20. The following links are first-hand or near-one-hand materials that can be accessed at the time of writing; meeting policies, code repository, benchmark results, model version and arXiv version may change, please check again before official release.

  1. DARPA, The Heilmeier Catechism. https://www.darpa.mil/about/heilmeier-catechism
  2. NSF, Preparing Your Proposal. https://www.nsf.gov/funding/preparing-proposal
  3. ICLR 2026 Author Guide. https://iclr.cc/Conferences/2026/AuthorGuide
  4. Chenglei Si, Diyi Yang, Tatsunori Hashimoto, "Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers", ICLR 2025. https://mlanthology.org/iclr/2025/si2025iclr-llms/
  5. Ziyu Chen, Yilun Zhao, Arman Cohan, "Measuring the Gap Between Human and LLM Research Ideas", arXiv:2607.01233, 2026. https://arxiv.org/abs/2607.01233
  6. SakanaAI, AI-Scientist-v2 GitHub repository. https://github.com/SakanaAI/AI-Scientist-v2
  7. Sakana AI, The AI Scientist project page. https://sakana.ai/ai-scientist/
  8. OpenAI, PaperBench: Evaluating AI's Ability to Replicate AI Research, 2025. https://openai.com/index/paperbench/
  9. OpenAI frontier-evals, PaperBench README. https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md
  10. Jing Ren, Weiqi Wang, "Assisting Research Proposal Writing with Large Language Models: Evaluation and Refinement", arXiv:2509.09709, 2025. https://arxiv.org/abs/2509.09709
  11. Kai Ruan et al., "Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context", Nature Communications, 2026. https://www.nature.com/articles/s41467-026-70245-1
  12. Jie Yu, Song Qiu, "More Than Can Be Said: A Benchmark and Framework for Pre-Question Scientific Ideation", arXiv:2605.06345, 2026. https://arxiv.org/abs/2605.06345
  13. Rosni Vasu et al., "HARPA: A Testability-Driven, Literature-Grounded Framework for Research Ideation", OpenReview ICLR 2026 withdrawn submission. https://openreview.net/forum?id=zGr73fcSsB
  14. NeurIPS 2026 Evaluations and Datasets Reviewing Guidelines. https://neurips.cc/Conferences/2026/EvaluationsDatasetsReviewerGuidelines