New York Site views —Page views —
← All series

Reproducing open AI experiments

Small implementations connect equations to model behavior, from encoders to calibration.

Reading order

  1. Reproducing a Transformer encoder: from equations to minimal PyTorch
  2. Attention heatmaps and the limits of interpretation
  3. LoRA and QLoRA: low-rank updates, quantization and memory budgets
  4. DPO: data format, loss and reward-free training
  5. A RAG baseline with BM25, embeddings, reranking and citation evaluation
  6. Dense retrieval with DPR and Contriever dual encoders
  7. Reranking with a cross-encoder and hard negatives
  8. Self-consistency: sampling, voting and the cost–accuracy trade-off
  9. Tool-calling agents: ReAct, tool errors and verifiable execution
  10. Language models as judges: bias, rubrics and consistency
  11. Position sensitivity in long contexts: reproducing Lost in the Middle
  12. Defending against prompt injection by isolating untrusted context
  13. Deduplication and contamination checks: from n-grams to semantic candidates
  14. Filtering synthetic data: quality, diversity and informative hard examples
  15. Distillation: from soft teacher probabilities to response supervision
  16. Instruction tuning a small model: data mixtures, templates and overfitting
  17. INT8 and INT4 inference: why size, perplexity and speed change differently
  18. KV caching and speculative decoding: verify outputs before measuring speed
  19. Visualizing MoE routing: what load-balancing loss misses about expert health
  20. CLIP alignment: check the positive pairs before trusting retrieval scores
  21. Evaluating a VLM on charts: audit the scorer before diagnosing reading or arithmetic
  22. Explaining an embodied agent’s successes, detours and timeouts with state logs
  23. World models: why accurate one-step predictions can drift during rollout
  24. GNNs and language models: finding the answer node versus explaining the reasoning
  25. Activation patching: does restoring an answer identify the mechanism?
  26. What a linear probe can read versus what a model uses
  27. Does lower ECE make an abstention threshold reliable?
  28. Does an unsuccessful adversarial test establish model robustness?
  29. A results-reporting template that traces every number to an experiment log
  30. Building a research code library that can replay earlier experiments