Reproducing open AI experiments
Small implementations connect equations to model behavior, from encoders to calibration.
Reading order
- Reproducing a Transformer encoder: from equations to minimal PyTorch
- Attention heatmaps and the limits of interpretation
- LoRA and QLoRA: low-rank updates, quantization and memory budgets
- DPO: data format, loss and reward-free training
- A RAG baseline with BM25, embeddings, reranking and citation evaluation
- Dense retrieval with DPR and Contriever dual encoders
- Reranking with a cross-encoder and hard negatives
- Self-consistency: sampling, voting and the cost–accuracy trade-off
- Tool-calling agents: ReAct, tool errors and verifiable execution
- Language models as judges: bias, rubrics and consistency
- Position sensitivity in long contexts: reproducing Lost in the Middle
- Defending against prompt injection by isolating untrusted context
- Deduplication and contamination checks: from n-grams to semantic candidates
- Filtering synthetic data: quality, diversity and informative hard examples
- Distillation: from soft teacher probabilities to response supervision
- Instruction tuning a small model: data mixtures, templates and overfitting
- INT8 and INT4 inference: why size, perplexity and speed change differently
- KV caching and speculative decoding: verify outputs before measuring speed
- Visualizing MoE routing: what load-balancing loss misses about expert health
- CLIP alignment: check the positive pairs before trusting retrieval scores
- Evaluating a VLM on charts: audit the scorer before diagnosing reading or arithmetic
- Explaining an embodied agent’s successes, detours and timeouts with state logs
- World models: why accurate one-step predictions can drift during rollout
- GNNs and language models: finding the answer node versus explaining the reasoning
- Activation patching: does restoring an answer identify the mechanism?
- What a linear probe can read versus what a model uses
- Does lower ECE make an abstention threshold reliable?
- Does an unsuccessful adversarial test establish model robustness?
- A results-reporting template that traces every number to an experiment log
- Building a research code library that can replay earlier experiments