New York Site views —Page views —
← All series

From a minimal reproduction to a testable question

Follow retrieval, fine-tuning and circuit experiments from baselines to reproducible reports.

Reading order

  1. From a toy example to SciFact: building an auditable retrieval baseline
  2. Are embeddings better than BM25? Comparing SciFact results one query at a time
  3. Why does hybrid retrieval work? How reciprocal rank fusion changes frozen SciFact rankings
  4. Where is the reranker’s ceiling? Separating retrieval failures from ranking failures
  5. Titles, abstracts and truncation: can preprocessing reorder retrieval methods?
  6. Does a frozen retrieval configuration transfer from SciFact to NFCorpus?
  7. Before fine-tuning: capability and format baselines on 64 SST-2 examples
  8. Which tokens contribute to SFT loss? Auditing templates and label masks on eight examples
  9. More examples or more tokens? Comparing data scale at a fixed budget
  10. The cost of mixing data: learning a new task and forgetting an old one
  11. Does it work with a new template? Format adaptation versus task generalization
  12. Is a fine-tuning gain reliable? Multiple seeds and paired reporting with 32 training examples
  13. Reading an evaluation harness task configuration on 64 SST-2 examples
  14. When few-shot examples hurt: testing example selection and order
  15. Scoring a letter or its text: option order and probability scoring on 64 examples
  16. How often does answer extraction fail? Auditing 64 real outputs against human judgments
  17. Two close checkpoints: paired intervals on 64 development examples
  18. Ten configurations, one improvement: selection bias on development and confirmation sets
  19. From toy networks to GPT-2: an indirect-object identification baseline on 64 examples
  20. What does activation patching restore? Localizing attention heads in 16 GPT-2 pairs
  21. Do different corruptions identify the same components? Two counterfactuals on 16 GPT-2 examples
  22. Does removing a few heads identify a circuit? Ablation and compensation on 16 GPT-2 examples
  23. Does circuit evidence survive new names and sentence structures? A 256-example GPT-2 test
  24. How trustworthy is a circuit diagram? A local test on 256 GPT-2 examples
  25. From failures to research questions: avoiding post-hoc stories on 100 SciFact queries
  26. Testing an intuition with a counterexample: evidence-shift probes on four SciFact queries
  27. Is the gain just more data or compute? Equal-budget tests on 100 SciFact queries
  28. Which module helped? Ablations and interactions on four SciFact probes
  29. Beyond a local gain: intervals and failure boundaries on 100 SciFact development queries
  30. From experiment bundle to research report: a frozen evaluation on 300 SciFact queries