llm/curriculum
Sources

Where the facts come from.

Papers, specs, official docs and the model repos themselves. Lessons link to the exact source next to each claim. This page collects them.

Number formats

Mixed Precision Training (Micikevicius et al., 2017)FP16 training with FP32 master weights and loss scaling.arxiv.orgFP8 Formats for Deep Learning (Micikevicius et al., 2022)E4M3 and E5M2.arxiv.orgOCP Microscaling (MX) Formats Specification v1.0MXFP4, MXFP8, E8M0 shared scales.www.opencompute.orgMicroscaling Data Formats for Deep Learning (Rouhani et al., 2023)The research behind MX.arxiv.orgNVIDIA: Introducing NVFP416-value blocks, FP8 block scale, FP32 tensor scale.developer.nvidia.com

Quantization

GPTQ (Frantar et al., 2022)Second-order, error-compensating weight quantization.arxiv.orgAWQ (Lin et al., 2023)Activation-aware scaling of salient channels.arxiv.orgLLM.int8() (Dettmers et al., 2022)Outlier features in large models.arxiv.orgQuIP# (Tseng et al., 2024)Hadamard incoherence processing and lattice codebooks.arxiv.orgQTIP (Tseng et al., 2024)Trellis-coded quantization, the basis of EXL3.arxiv.orgExLlamaV3 (turboderp)EXL3 format, converter, inference.github.comExLlamaV2 (turboderp)EXL2 format with per-layer mixed bitrates.github.comllama.cpp k-quants PR #1684Q2_K…Q6_K block layouts and _S/_M/_L mixes.github.comllama.cpp imatrix toolImportance matrix from calibration text.github.comllama.cpp perplexity / KL divergence--kl-divergence-base and --kl-divergence.github.comMLX quantization docsAffine group quantization with scales and biases.ml-explore.github.iollm-compressor (vLLM project)Quant recipes: GPTQ, AWQ, FP8, NVFP4 for vLLM.github.com

File formats

safetensorsHeader layout and safety model.github.comGGUF specSingle-file format for llama.cpp.github.comHugging Face chat templatesHow chat_template turns messages into tokens.huggingface.co

Models read in this curriculum

Qwen/Qwen3-0.6BDense, GQA, tied embeddings.huggingface.coQwen/Qwen3-30B-A3BMoE, 128 experts, 8 active.huggingface.coQwen/Qwen3-Next-80B-A3B-InstructHybrid linear attention, 512 experts.huggingface.coQwen/Qwen3.5-9BHybrid Gated DeltaNet + full attention.huggingface.coornith-ai/Ornith-1.5-9BQwen3.5-family, Otter's base model.huggingface.codeepseek-ai/DeepSeek-V3MLA, shared expert, FP8 block weights.huggingface.coDeepSeek-V3 Technical ReportArchitecture and FP8 training.arxiv.orgmoonshotai/Kimi-K2-InstructDeepSeek-V3-style MoE at ~1T parameters.huggingface.coopenai/gpt-oss-20bMXFP4 experts, alternating sliding/full attention.huggingface.coMuse Spark Open 816B EXL3 SAGE 1.13bpwCommunity EXL3 mixed-precision quant (ViC305).huggingface.co

Learning

3Blue1Brown: Neural networksVisual intuition for transformers.www.3blue1brown.comAndrej Karpathy: Neural Networks: Zero to HeroBuild GPT from scratch.karpathy.aiSebastian Raschka: The Big LLM Architecture ComparisonQwen, DeepSeek, Kimi, gpt-oss side by side.magazine.sebastianraschka.com