Appendix D Further Reading

D.1 Canonical Papers by Topic

Each cluster below maps to one or more chapters in this book. Clusters 1-2 support chapter 3 and chapter 13-chapter 14. Clusters 3-5 map to chapter 6-chapter 7 and chapter 8-chapter 10. Clusters 6-9 support chapter 15-chapter 19. Cluster 10 maps to chapter 20-chapter 22. Clusters 11-12 support chapter 26-chapter 28 and chapter 23-chapter 25. Format: Title - Authors (last names) - one-line description.

1. Transformer Architecture & Attention

Attention Is All You Need - Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin - Introduces the Transformer: encoder-decoder architecture built entirely from multi-head self-attention and feedforward layers, replacing recurrence.

RoFormer: Enhanced Transformer with Rotary Position Embedding - Su, Lu, Pan, Murtadha, Wen, Liu - Proposes Rotary Position Embedding (RoPE), encoding position by rotating query/key vectors so that dot-product attention captures relative distance naturally.

Fast Transformer Decoding: One Write-Head Is All You Need - Shazeer - Introduces Multi-Query Attention (MQA), sharing a single key-value head across all query heads to reduce KV-cache memory and decoding latency.

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints - Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebron, Sanghai - Proposes Grouped-Query Attention (GQA) as an interpolation between MHA and MQA, providing near-MQA speed with quality close to full multi-head attention.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness - Dao, Fu, Ermon, Rudra, Re - Rewrites attention computation with tiling to avoid materialising the full $N \times N$ attention matrix, cutting HBM traffic and achieving $2$ - $4 \times$ speed-up.

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning - Dao - Refines FlashAttention with improved thread-block scheduling and reduced non-matmul FLOPs to nearly double throughput on A100/H100 GPUs.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model - DeepSeek-AI - Introduces Multi-Head Latent Attention (MLA), compressing KV projections into low-rank latent vectors to reduce KV-cache by up to $93 . 3 \%$ without quality loss.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation - Press, Smith, Lewis - Presents ALiBi, adding a linear bias to attention logits based on token distance so models trained on short sequences generalise to longer ones at inference.

GLU Variants Improve Transformer - Noam Shazeer - Studies Gated Linear Unit variants including SwiGLU (swish-gated FFN) and shows they consistently outperform standard ReLU feed-forward layers.

2. Scaling Laws

Scaling Laws for Neural Language Models - Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei - Establishes empirical power-law relationships between model parameters, dataset size, compute budget, and validation loss for autoregressive language models.

Training Compute-Optimal Large Language Models - Hoffmann, Borgeaud, Mensch, Buchatskiy, Cai, Millican, van den Driessche, Clark, Casas, Guy, Hendricks, Jones, Menick, Hernandez, Jakub, Hubert, Vinyals, Osindero, Kavukcuoglu, Simonyan, Rae - The “Chinchilla” paper, showing that previous models were overparameterised and under-trained; optimal compute budget splits equally between model size and token count.

Scaling LLM Test-Time Compute Optimally - Snell, Lee, Xu, Kumar, Garg, Sharan, Abbeel, Levine, Ghosh - Demonstrates that allocating more computation at inference time (e.g. best-of- $N$ sampling, process-rewardguided search) can match gains from scaling parameters.

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer - Yang, Hu, Babuschkin, Sidor, Liu, Farhi, Ryder, Pachocki, Chen, Gao - Introduces Maximal Update Parameterisation ( $\mu$ P), enabling hyperparameter tuning on small proxy models that transfer to large models without retuning.

3. Mixture of Experts

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer - Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton, Dean - Introduces the conditional-computation MoE layer with a learned routing gate, scaling model capacity to 137B parameters without proportional compute cost.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity - Fedus, Zoph, Shazeer - Simplifies MoE routing to top-1 (single-expert) selection and demonstrates stable trillion-parameter training with strong downstream performance.

Mixtral of Experts - Jiang, Sablayrolles, Roux, Mensch, Savary, Bamford, Chaplot, de las Casas, Hanna, Bressand, Lengyel, Bour, Lample, Lavaud, Saulnier, Lachaux, Stock, Gall, Acharya, Gottesman, Ourry, Rastogi, Roelofs, Sanseviero, de Vries, Launay - Presents Mixtral 8 $\times$ 7B (and 8 $\times$ 22B), a sparse MoE model using top-2 routing that matches or exceeds dense models many times its active parameter count.

DeepSeekMoE: Towards Ultimate Expert Specialisation in Mixture-of-Experts Language Models - Dai, Deng, Zhao, Xu, Gao, Chen, Li, Zeng, Yu, Wu, Xie, Li, Huang, Luo, Liang, Chen, Liu, Dai - Proposes fine-grained expert segmentation with shared plus routed experts, achieving higher specialisation and efficiency than standard top- $k$ routing.

4. Distributed Training & Parallelism

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism - Shoeybi, Patwary, Puri, LeGresley, Casper, Catanzaro - Introduces intra-layer (tensor) model parallelism for Transformers, enabling efficient multi-GPU training of models too large for a single device.

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM - Narayanan, Shoeybi, Casper, LeGresley, Patwary, Korthikanti, Vainbrand, Kashinkunti, Bernauer, Houston, Tulloch, Shi, Singh, Catanzaro - Combines tensor, pipeline, and data parallelism (“3D parallelism”) with interleaved pipeline scheduling to train 1T-parameter models efficiently. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism - Huang, Cheng, Bapna, Firat, Chen, Chen, Lee, Gong, Shen, Wu, Le - Proposes pipeline parallelism with micro-batching and gradient accumulation to train models across multiple accelerators in a synchronous fashion.

PipeDream: Generalized Pipeline Parallelism for DNN Training - Narayanan, Harlap, Phanishayee, Se-shadri, Devanur, Ganger, Gibbons, Zaharia - Introduces asynchronous 1F1B (one-forward-one-backward) pipeline scheduling to overlap communication and computation, reducing idle pipeline bubbles.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models - Rajbhandari, Rasley, Ruwase, He - Proposes Zero Redundancy Optimizer (ZeRO), partitioning optimizer state, gradients, and parameters across data-parallel ranks to remove memory redundancy without increasing communication volume significantly.

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel - Zhao, Gu, Varma, Luo, Huang, Xu, Wright, Shojanazeri, Ott, Sheng, Bhatt, Shen, Abid, Truong, Bhosale, Mihailescu, Patel - Describes Py-Torch’s production FSDP implementation, detailing sharding strategies, mixed precision, and the API choices made for usability at scale.

5. GPU Kernels & Hardware

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness - Dao, Fu, Ermon, Rudra, Re - (See Cluster 1.) The foundational IO-aware kernel paper; listed again here as the exemplar of GPU-level memory hierarchy optimisation for DL.

Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations - Tillet, Kung, Cox - Introduces the Triton DSL that lets researchers write high-performance GPU kernels in Python without hand-coding CUDA, with tiling handled by the compiler.

NVIDIA H100 Tensor Core GPU Architecture - NVIDIA - Official architecture whitepaper covering the Hopper SM, fourth-generation Tensor Cores, NVLink 4.0, and the new Transformer Engine with FP8 support.

CUTLASS: Fast Linear Algebra in CUDA C++ - Kerr, Nicholls, Tanner, Naumov, Young, Coles, Mer-cat, Noel, Haluza, Kerr (NVIDIA) - Describes CUTLASS, NVIDIA’s open-source template library for high-performance GEMM and convolution operations that underlies cuBLAS and many DL frameworks.

6. Inference Systems

Orca: A Distributed Serving System for Transformer-Based Generative Models - Yu, Jeong, Kim, Kim, Jeong, Cho, Shin - Proposes iteration-level (continuous) batching so that finished sequences are immediately evicted and new requests join the batch without waiting, greatly improving GPU utilisation.

Efficient Memory Management for Large Language Model Serving with PagedAttention - Kwon, Li, Zhuang, Sheng, Luo, Foster, Flammarion, Huang, Stoica, Zhang - Introduces PagedAttention, storing KV-cache in non-contiguous memory pages analogous to OS virtual memory, eliminating fragmentation and enabling flexible sharing for vLLM.

Accelerating Large Language Model Decoding with Speculative Sampling - Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper - Shows that a small draft model can propose $k$ tokens in parallel, which the large target model verifies in a single forward pass, yielding lossless $2$ - $3 \times$ decoding speed-up.

Fast Inference from Transformers via Speculative Decoding - Leviathan, Kalman, Matias - Independent concurrent work establishing the same speculative-decoding framework with formal acceptance-rejection proofs preserving the target distribution exactly.

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads - Cai, Li, Geng, Peng, Lee, Chen, Dao - Adds multiple lightweight “Medusa heads” to a frozen LLM to predict several future tokens simultaneously, enabling tree-based speculative decoding without a separate draft model.

Efficiently Programming Large Language Models using SGLang - Zheng, Yin, Xie, Huang, Sun, Yu, Cao, Kozyrakis, Stoica, Gonzalez, Barrett, Sheng - Presents SGLang, a structured generation language that fuses RadixAttention (prefix-tree KV-cache reuse) with a runtime for complex, multi-call LLM programs.

7. KV-Cache Reduction

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints - Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebron, Sanghai - (See Cluster 1.) Reduces KV-cache by a factor equal to the group size by sharing key/value heads across groups of query heads.

Efficient Streaming Language Models with Attention Sinks - Xiao, Tian, Chen, Han, Lewis - Observes that a small number of “attention sink” tokens attract disproportionate attention, enabling infinite-length generation by maintaining only sinks plus a sliding-window cache (StreamingLLM).

H $\text{2}$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models - Zhang, Sheng, Jiang, Chen, Li, Guo, Zhu, Zheng, Gonzalez, Stoica, Re - Identifies “heavy-hitter” tokens that accumulate the most attention mass and proposes a greedy eviction policy that preserves them to maintain quality under aggressive KV-cache budgets.

SnapKV: LLM Knows What You Are Looking for Before Generation - Li, Zhang, He, Guo, Qian, Zhang, Ma, Shan, Yang, Yin - Clusters per-head attention patterns from a short observation window at prefill time to select and compress the KV-cache before long generation begins.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model - DeepSeek-AI - (See Cluster 1.) MLA compresses the joint KV projection into a single low-rank latent vector, reducing pertoken KV-cache size by $93 . 3 \%$ relative to standard MHA.

8. Beyond-Transformer Architectures

Mamba: Linear-Time Sequence Modeling with Selective State Spaces - Gu, Dao - Introduces a selective state-space model (SSM) with input-dependent gating that matches Transformer quality on language modelling while scaling linearly in sequence length.

RWKV: Reinventing RNNs for the Transformer Era - Peng, Alcaide, Anthony, Albalak, Arcadinho, Bi-derman, Cao, Cheng, Chung, Derczynski, et al. - Proposes RWKV, reformulating attention as a linear recurrence that admits both parallel training (like Transformers) and constant-memory inference (like RNNs).

Jamba: A Hybrid Transformer-Mamba Language Model - Lieber, Lenz, Bata, Cohen, Osin, Dalmedigos, Safahi, Meirom, Belinkov, Shalev-Shwartz, Shoham - Interleaves Transformer (attention) and Mamba (SSM) layers with MoE feed-forwards, combining the long-context efficiency of SSMs with the in-context learning strength of attention.

9. Compression & Quantization

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers - Frantar, Ashkboos, Hoefler, Alistarh - Uses second-order (Hessian) information via the Optimal Brain Quantisation framework to quantise large LLMs to 3-4-bit weights with minimal perplexity degradation.

AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration - Lin, Tang, Tang, Yang, Dang, Han - Identifies salient weight channels by observing activation magnitudes and protects them from quantisation error, enabling efficient 4-bit inference without retraining.

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models - Xiao, Lin, Seznec, Wu, Demouth, Han - Migrates quantisation difficulty from activations to weights by per-channel scaling, making both amenable to INT8 without accuracy loss.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale - Dettmers, Lewis, Belkada, Zettlemoyer - Shows that emergent large-magnitude outlier features in activations cause INT8 degradation, and proposes mixed-precision decomposition to quantise $\sim$ 99.9% of weights to INT8 while keeping outliers in FP16.

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter - Sanh, Debut, Chaumond, Wolf - Demonstrates knowledge distillation for Transformers, training a 40%-smaller student BERT that retains 97% of teacher performance on GLUE via soft-label and hidden-state matching.

LoRA: Low-Rank Adaptation of Large Language Models - Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen - (See Cluster 11.) Also a compression technique: decomposing weight updates into low-rank products reduces trainable parameters by $10 , 000 \times$ compared with full fine-tuning.

10. Data Pipelines

CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data - Wenzek, Lachaux, Con-neau, Chaudhary, Guzman, Joulin, Grave - Presents the CCNet pipeline: language identification, deduplication, and quality filtering of Common Crawl at scale; foundational for many subsequent pre-training corpora.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling - Gao, Biderman, Black, Golding, Hoppe, Foster, Phang, He, Thite, Nabeshima, Presser, Leahy - Introduces a curated, diverse 800 GB dataset from 22 high-quality sources, providing an open training corpus and evaluation of data diversity effects.

The RefinedWeb Dataset for Falcon LLMs: Outperforming Curated Corpora with Web Data Only - Penedo, Malartic, Hesslow, Cojocaru, Cappelli, Alobeidli, Pannier, Almazrouei, Launay - Shows that massively deduplicated and filtered Common Crawl data (RefinedWeb) can outperform manually curated mixtures, challenging assumptions about the necessity of diverse high-quality sources.

Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research - Soldaini, Kinney, Bhagia, Schwenk, Atkinson, Magnusson, Authur, Chandu, Dror, Fang, et al. - Releases a fully open 3T-token corpus from AI2 with detailed provenance and filtering documentation, enabling reproducible pre-training research.

Deduplicating Training Data Makes Language Models Better - Lee, Ippolito, Carlini, Chang, Uszkoreit, Carlini, Litwin - Demonstrates that near-duplicate removal (MinHash LSH) from pre-training corpora reduces memorisation and improves downstream task generalisation.

11. Alignment & Post-Training

Training Language Models to Follow Instructions with Human Feedback - Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike, Lowe - The InstructGPT paper: SFT on human demonstrations followed by PPO against a reward model trained on human preference comparisons, establishing the canonical RLHF pipeline.

Constitutional AI: Harmlessness from AI Feedback - Bai, Jones, Ndousse, Askell, Chen, DasSarma, Drain, Fort, Ganguli, Henighan, Joseph, Kadavath, Kernion, Conerly, El-Showk, Elhage, Hernandez, Hume, Kaplan, Lovitt, Nanda, Olsson, Amodei, Brown, Clark, McCandlish, Olah, Mann, Amodei, Clark - Trains a harmless assistant using a written “constitution” of principles: supervised learning from self-critique and RLAIF, removing dependence on human preference labels for harmlessness.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model - Rafailov, Sharma, Mitchell, Ermon, Manning, Finn - Derives a closed-form training objective from the RLHF reward-maximisation problem, enabling preference-based alignment without an explicit reward model or PPO.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning - DeepSeek-AI - Trains reasoning models with Group Relative Policy Optimisation (GRPO) and verifiable outcome rewards, achieving OpenAI-o1-level performance while open-sourcing weights and methodology.

LoRA: Low-Rank Adaptation of Large Language Models - Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen - Freezes pre-trained weights and injects trainable rank-decomposition matrices into attention projections, reducing fine-tuning memory and compute by orders of magnitude.

QLoRA: Efficient Finetuning of Quantized LLMs - Dettmers, Pagnoni, Holtzman, Zettlemoyer - Combines 4-bit NormalFloat quantisation (NF4) of the frozen base model with LoRA adapters and paged optimisers, enabling full-parameter-equivalent fine-tuning on a single consumer GPU.

12. Evaluation

Measuring Massive Multitask Language Understanding - Hendrycks, Burns, Basart, Zou, Mazeika, Song, Steinhardt - Introduces MMLU: 57 subjects spanning STEM, humanities, and social science, providing a broad measure of world knowledge and academic reasoning.

Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models - Sri-vastava, Rastogi, Rao, Shoeb, Abid, Fisch, Brown, Santoro, Gupta, Garriga-Alonso, et al. - Presents BIG-Bench: 204 tasks collaboratively designed to probe model capabilities beyond standard benchmarks, with analysis of scaling behaviour.

HellaSwag: Can a Machine Really Finish Your Sentence? - Zellers, Holtzman, Bisk, Farhadi, Choi - Con-structs adversarially filtered commonsense NLI completions that are trivial for humans but hard for models, exposing the gap between surface statistics and genuine understanding.

Evaluating Large Language Models Trained on Code - Chen, Tworek, Jun, Yuan, de Oliveira Pinto, Ka-plan, Edwards, Burda, Joseph, Brockman, Ray, Puri, Krueger, Petrov, Khlaaf, Sastry, Mishkin, Chan, Gray, Ryder, Pavlov, Power, Kaiser, Bavarian, Winter, Tillet, Such, Cummings, Plappert, Chantzis, Barnes, HerbertVoss, Guss, Nichol, Paino, Tezak, Tang, Babuschkin, Balaji, Jain, Saunders, Hesse, Carr, Leike, Achiam, Misra, Morikawa, Radford, Knight, Brundage, Murati, Mayer, Welinder, McGrew, Amodei, McCandlish, Sutskever, Zaremba - Introduces HumanEval: 164 hand-written Python programming problems with unit tests for pass@ $k$ evaluation, the standard functional-correctness benchmark for code LLMs.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, Zhang, Gonzalez, Stoica - Introduces MT-Bench (multi-turn instruction-following) and Chatbot Arena (human pairwise voting), establishing LLM-as-a-Judge as a scalable evaluation methodology aligned with human preference.

Holistic Evaluation of Language Models - Liang, Bommasani, Lee, Tsipras, Soylu, Yasunaga, Zhang, Faber, Wu, Bharadwaj, et al. - Presents HELM: a holistic evaluation framework that measures accuracy, calibration, robustness, fairness, bias, and efficiency across dozens of scenarios and model families.

D.2 Blogs, Talks, and Engineering Reports

The entries below cover flagship model technical reports and practitioner blogs that put flesh on the theory developed across Parts I-XI. Model reports (GPT-4, Llama 3, Gemini, DeepSeek) are the best place to see how the concepts in chapter 3-chapter 14 manifest at production scale. Infrastructure and alignment blogs complement chapter 11, chapter 15, chapter 19, and chapter 26-chapter 28. Format: Title - Publisher/Company - one-line description.

13. Engineering Blogs and Technical Reports

Flagship Model Technical Reports

GPT-4 Technical Report - OpenAI - Describes GPT-4 capabilities, evaluations (including bar-exam, MMLU, vision), and safety mitigations; notable for its detailed model-card approach while withholding architectural specifics.

The Llama 3 Herd of Models - Meta AI - Covers the full Llama 3 family (8B-405B), detailing pre-training data curation, supervised fine-tuning, RLHF, and safety alignment with publicly released weights.

Llama 2: Open Foundation and Fine-Tuned Chat Models - Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale, et al. (Meta AI) - Introduces Llama 2 and Llama 2-Chat, documenting the SFT-then-RLHF training recipe, ghost attention, and iterative red-teaming for safety.

Gemini: A Family of Highly Capable Multimodal Models - Google DeepMind - Presents the Gemini 1.0 family (Ultra, Pro, Nano), covering multimodal architecture, training at scale, and benchmark results across text, code, image, audio, and video. Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context - Google DeepMind - Details Gemini 1.5 Pro’s million-token context window via a sparse MoE architecture and efficient attention, enabling long-document and long-video understanding.

PaLM 2 Technical Report - Google - Documents the PaLM 2 model family with improved compute-optimal training (Chinchilla scaling), multilingual capability, and reasoning improvements over PaLM.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning - DeepSeek-AI - Open technical report detailing GRPO-based reinforcement learning on verifiable reasoning tasks, cold-start strategies, and distillation to smaller models; open-weights release included.

DeepSeek-V3 Technical Report - DeepSeek-AI - Describes DeepSeek-V3, a 671B MoE model trained at low cost using FP8 mixed precision, auxiliary-loss-free load balancing, and multi-token prediction pretraining objectives.

Claude’s Character - Anthropic - Anthropic’s model card and character overview explaining the values, persona, and constitutional principles embedded in the Claude model family.

Inference Systems Engineering Blogs

vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention - UC Berkeley / vLLM Blog - Introductory blog post accompanying the vLLM open-source release, explaining PagedAttention and demonstrating $24 \times$ throughput gains over HuggingFace Transformers baseline serving.

Continuous Batching: How to Serve LLMs 23 $\times$ Faster Than HuggingFace - Anyscale Engineering Blog - Engineering walkthrough of how iteration-level (continuous) batching works in practice and why it dominates static batching for variable-length generation workloads.

How to Make LLM Inference Faster with Speculative Decoding - Hugging Face Blog - Practitioner-focused explanation of speculative decoding variants (draft-model, Medusa, EAGLE) with HuggingFace transformers integration examples.

Distributed Training & Infrastructure

Megatron-LM: Continued Training and Fine-tuning for GPT and BERT - NVIDIA NeMo / Megatron Engineering Blog - NVIDIA’s engineering series covering tensor, sequence, and pipeline parallelism recipes for Megatron-LM, including practical tuning guides for A100/H100 clusters.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (DeepSpeed blog) - Microsoft DeepSpeed Blog - Accessible introduction to ZeRO stages 1-3 and ZeRO-Infinity, with empirical memory and throughput numbers for large-model training.

Scaling Distributed Machine Learning with the Parameter Server - Li, Andersen, Park, Smola, Ahmed, Josifovski, Long, Shekita, Su - Classic systems paper introducing the parameter-server architecture for distributed gradient aggregation; foundational reading before data parallelism and ZeRO.

How We Trained Llama 2 at Scale - Meta AI Engineering Blog - Meta’s post-mortem of Llama 2 training infrastructure: FSDP, custom CUDA kernels, fault tolerance, and lessons from training 70B models on clusters of thousands of GPUs.

Quantization & Efficiency Engineering

Making LLMs Even More Accessible with bitsandbytes, 4-bit Quantization and QLoRA - Hugging Face Blog - Walkthrough of the QLoRA recipe integrated into the PEFT library, demonstrating fine-tuning of 65B models on a single 48 GB GPU with NF4 quantisation.

Quantization Fundamentals with Hugging Face - Hugging Face Blog - Survey of quantisation approaches (GPTQ, AWQ, SmoothQuant, bitsandbytes) as implemented in the Hugging Face ecosystem, comparing quality and throughput tradeoffs.

Data Curation & Pre-Training

What’s in My Big Data? - Allen Institute for AI (AI2) Blog - Engineering post accompanying the Dolma corpus release, detailing data sourcing decisions, quality filtering heuristics, and the open-source dolma toolkit.

DataComp-LM: In Search of the Next Generation of Training Sets for Language Models - Li, Fang, Smyrnis, Ivgi, Jordan, Gadre, Bansal, Guha, Keh, Arora, et al. - Introduces DCLM: a large-scale competition and dataset for studying data curation strategies, with controlled experiments comparing filtering pipelines for LLM pre-training.

Alignment, Safety, and Interpretability

Reward Modeling for Mitigating Sycophancy in RLHF - Anthropic Research Blog - Discusses failure modes of reward models trained on human preferences, specifically sycophancy, and approaches for building more robust reward signals.

Toy Models of Superposition - Anthropic Interpretability Team - Mechanistic interpretability study showing that neural networks can represent more features than neurons by using superposition, with implications for understanding polysemanticity at scale.

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet - Anthropic Interpretability Team - Applies sparse autoencoders to decompose Claude 3 Sonnet residual-stream activations into millions of monosemantic features, demonstrating that large-scale interpretability is tractable.

OpenAI’s Approach to AI Safety - OpenAI - Overview of OpenAI’s safety research agenda including RLHF, red-teaming, interpretability, and preparedness framework for frontier models.

Evaluation & Benchmarking

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference - LMSYS / Hugging Face - Ongoing open leaderboard using blind pairwise human votes to rank LLMs via Elo ratings; the de-facto community standard for chat-quality comparisons.

Open LLM Leaderboard - Hugging Face - Automated leaderboard evaluating open-weight models on a standardised suite (MMLU, HellaSwag, ARC, TruthfulQA, WinoGrande, GSM8K) enabling reproducible open-model comparisons.