From a minimal reproduction to a testable question
Follow retrieval, fine-tuning and circuit experiments from baselines to reproducible reports.
Reading order
- From a toy example to SciFact: building an auditable retrieval baseline
- Are embeddings better than BM25? Comparing SciFact results one query at a time
- Why does hybrid retrieval work? How reciprocal rank fusion changes frozen SciFact rankings
- Where is the reranker’s ceiling? Separating retrieval failures from ranking failures
- Titles, abstracts and truncation: can preprocessing reorder retrieval methods?
- Does a frozen retrieval configuration transfer from SciFact to NFCorpus?
- Before fine-tuning: capability and format baselines on 64 SST-2 examples
- Which tokens contribute to SFT loss? Auditing templates and label masks on eight examples
- More examples or more tokens? Comparing data scale at a fixed budget
- The cost of mixing data: learning a new task and forgetting an old one
- Does it work with a new template? Format adaptation versus task generalization
- Is a fine-tuning gain reliable? Multiple seeds and paired reporting with 32 training examples
- Reading an evaluation harness task configuration on 64 SST-2 examples
- When few-shot examples hurt: testing example selection and order
- Scoring a letter or its text: option order and probability scoring on 64 examples
- How often does answer extraction fail? Auditing 64 real outputs against human judgments
- Two close checkpoints: paired intervals on 64 development examples
- Ten configurations, one improvement: selection bias on development and confirmation sets
- From toy networks to GPT-2: an indirect-object identification baseline on 64 examples
- What does activation patching restore? Localizing attention heads in 16 GPT-2 pairs
- Do different corruptions identify the same components? Two counterfactuals on 16 GPT-2 examples
- Does removing a few heads identify a circuit? Ablation and compensation on 16 GPT-2 examples
- Does circuit evidence survive new names and sentence structures? A 256-example GPT-2 test
- How trustworthy is a circuit diagram? A local test on 256 GPT-2 examples
- From failures to research questions: avoiding post-hoc stories on 100 SciFact queries
- Testing an intuition with a counterexample: evidence-shift probes on four SciFact queries
- Is the gain just more data or compute? Equal-budget tests on 100 SciFact queries
- Which module helped? Ablations and interactions on four SciFact probes
- Beyond a local gain: intervals and failure boundaries on 100 SciFact development queries
- From experiment bundle to research report: a frozen evaluation on 300 SciFact queries