Every term, plainly.

Hover any dotted word in a lesson to see its entry. 63 terms so far.

Abliteration

Removing a behaviour (usually refusals) by finding the direction in activation space linked to it and projecting it out of the weights. No training needed. → Modifying models

Active parameters

Parameters actually used for one token. Qwen3-30B-A3B: ~30B total, ~3B active. Speed follows active, memory follows total. → Mixture of Experts

AWQ

Activation-aware Weight Quantization: finds the ~1% of weight channels hit by large activations and scales them up before rounding so they keep precision (Lin et al., 2023). → Quantization

BF16 (bfloat16)

16-bit float with 8 exponent bits and 7 mantissa bits. Same range as FP32, less precision. The default format for releasing open weights. → Number formats

BPE (byte-pair encoding)

The usual way to build a tokenizer vocabulary: start from bytes and keep merging the most frequent neighbouring pair into a new token until the vocabulary is big enough. → Tokens and tokenizers

bpw (bits per weight)

Average storage per parameter including scales. File size ≈ parameters x bpw / 8 bytes. 4-bit with a 16-bit scale per 128 weights = 4.125 bpw. → Number formats

Calibration data

A few hundred sample texts run through the model during quantization so the method knows which weights matter most (GPTQ, AWQ, imatrix, EXL3). → Quantization

Chat template

A small Jinja program shipped with the tokenizer that turns a list of chat messages into the exact token stream the model was trained on. → Tokens and tokenizers

config.json

The model's blueprint: number of layers, hidden size, heads, experts, vocab size, context length. Code reads it to build the empty model before loading weights. → Anatomy of a model repo

Context length

How many tokens the model can see at once (prompt + answer). max_position_embeddings in config.json. → Inference, KV cache and context

Data-free quantization

Quantization that needs no calibration data, like RTN, plain FP8 casting or MLX's default quantize. → Quantization

E2M1 (FP4)

A 4-bit float: 1 sign, 2 exponent, 1 mantissa bit. It can only hold 0, 0.5, 1, 1.5, 2, 3, 4, 6 and their negatives. → Number formats

Embedding

The lookup table that turns a token id into a vector of hidden_size numbers. Tensor name: embed_tokens. → Anatomy of a model repo

EXL3

ExLlamaV3's format, based on QTIP: weights are rotated with Hadamard transforms and encoded with a trellis code, which allows fractional bitrates like 2.25 or 1.13 bpw. → Quantization

Exponent

The bits of a float that say how big the number is (which power of two). More exponent bits = wider range. → Number formats

FP16 (half)

16-bit float with 5 exponent bits and 10 mantissa bits. More precise than BF16 but largest value is 65,504, so big activations can overflow. → Number formats

FP32

32-bit float: 1 sign, 8 exponent, 23 mantissa bits. Used for optimizer states and master weights in training. → Number formats

FP8

8-bit float. E4M3 (4 exponent, 3 mantissa, max 448) for weights/activations, E5M2 (max 57,344) for gradients. Needs scale factors. Native on H100/Blackwell. → Number formats

GGUF

llama.cpp's single-file format: metadata, tokenizer and (usually quantized) tensors in one file. Used by llama.cpp, Ollama, LM Studio. → Anatomy of a model repo

GPTQ

Quantizes a layer column by column and adjusts the not-yet-quantized weights to cancel the error, using second-order statistics from calibration data (Frantar et al., 2022). → Quantization

GQA

Grouped-Query Attention: several query heads share one key/value head, shrinking the KV cache. num_key_value_heads in config.json. → The transformer

Gradient

For every parameter, how much the loss would change if that parameter went up a little. Training moves each parameter a small step against its gradient. → Foundations

Group size

How many weights share one scale. Smaller groups follow the data more closely (better quality) but store more scales (more bits). → Quantization

GRPO

Group Relative Policy Optimization: sample a group of answers to one prompt, score them, and reinforce each by how it compares with the group average. If every answer scores the same, the group teaches nothing. → Post-training

Harness (agent scaffold)

The program around the model that runs the agent loop: gives it tools, builds the context, parses its replies and decides when to stop. OpenCode, Claude Code, Codex and Terminus-2 are harnesses. The same model can score very differently in different harnesses. → Post-training

hidden_size

How many numbers represent each token while it flows through the model (the width of the residual stream). 1024 in Qwen3-0.6B, 7168 in DeepSeek-V3. → Anatomy of a model repo

imatrix

llama.cpp's importance matrix: activation statistics from calibration text used to weight the rounding error so important weights are kept more precise. → Quantization

k-quants

llama.cpp GGUF quant types like Q4_K_M: super-blocks of 256 weights split into sub-blocks with their own quantized scales. _S/_M/_L mixes keep some tensors at higher bits. → Quantization

KL divergence (KLD)

How different two probability distributions are. 0 = identical. Used to compare a quant's next-token probabilities with the original's over all ~150k tokens. → Quantization

KV cache

The saved keys and values of every previous token in every attention layer, so the model doesn't recompute them. It grows with context length. → Inference, KV cache and context

lm_head

The final matrix that turns the last hidden vector into one score (logit) per vocabulary token. Some models share (tie) it with the embedding table. → Anatomy of a model repo

Logit

A raw score the model gives to one possible next token. Softmax turns all logits into probabilities that sum to 1. → Quantization

LoRA

Fine-tuning by freezing the model and training two small matrices per layer whose product is added to the original weight. → Post-training

Loss

One number saying how wrong the model is. For LLMs: cross-entropy, -log(probability the model gave the real next token). → Foundations

Mantissa

The bits of a float that hold the precise digits. More mantissa bits = finer steps between neighbouring numbers. → Number formats

Mixed precision

Using different number formats for different parts. In quantization: sensitive tensors (embeddings, attention, router) at more bits, bulk MLP/expert weights at fewer. In training: BF16 compute with FP32 master weights. → Quantization

Mixture of Experts (MoE)

Each MLP is split into many 'experts' and a router picks a few per token. Total parameters are large, active parameters per token are small. → Mixture of Experts

MLA

Multi-head Latent Attention (DeepSeek-V2/V3, Kimi K2): keys and values are compressed into a small latent vector, making the KV cache much smaller. → Architectures compared

MLX

Apple's array framework for Apple Silicon. MLX model repos store safetensors with grouped affine quantization (weights, scales, biases per group). → Anatomy of a model repo

MXFP4

OCP Microscaling FP4: 4-bit floats (E2M1) in blocks of 32 sharing one power-of-two (E8M0) scale. About 4.25 bits per weight. gpt-oss ships its experts in it. → Number formats

NVFP4

NVIDIA's FP4: E2M1 values in blocks of 16 with an FP8 (E4M3) scale per block plus one FP32 scale per tensor. About 4.5 bits per weight. Native on Blackwell GPUs. → Number formats

OpenEnv

A standard interface for RL environments (reset / step / state, typed actions and observations, packaged as Docker, publishable to the Hub). It doesn't train or define rewards. → Post-training

Outlier

A weight or activation much larger than its neighbours. One outlier forces a big scale on its whole group and crushes the small values to zero. → Quantization

Parameter (weight)

One learned number inside the model. A 9B model has about 9 billion of them, stored in tensors. → Anatomy of a model repo

Perplexity (PPL)

exp(average loss) on real text. Roughly: how many tokens the model is 'choosing between' on average. Lower is better, but it hides rare big mistakes. → Quantization

QLoRA

LoRA on top of a 4-bit (NF4) frozen base model, so big models fit in less memory. → Post-training

Quantization

Storing weights with fewer bits by rounding them to a small set of allowed values plus scale factors. Smaller and faster, slightly less accurate. → Quantization

Recipe

The exact reproducible settings used to make or run a model: a quant config, a merge YAML, a training config or a serving command. → Recipes

RL environment

The system a model acts on during RL. It owns the task, state, tools, observations, reward and when the episode ends. → Post-training

Router (gate)

A small layer that scores every expert for the current token and sends it to the top-k. Usually kept at high precision when quantizing. → Mixture of Experts

RTN (round to nearest)

The simplest quantizer: divide by a scale, round to the nearest allowed value. No calibration data. GPTQ/AWQ/EXL3 improve on it. → Quantization

safetensors

Hugging Face's file format for weights: a small JSON header listing every tensor (name, dtype, shape, byte range) followed by the raw bytes. It cannot run code, unlike old pickle .bin files. → Anatomy of a model repo

Scale

The number you multiply a quantized integer by to get the weight back. One scale can cover a whole row, a group of 32-128 weights, or a 2-D block. → Quantization

SFT

Supervised fine-tuning: training on example conversations so the model learns to answer in that style. → Post-training

Shard

Big models are split into several .safetensors files (shards). model.safetensors.index.json says which tensor lives in which shard. → Anatomy of a model repo

Softmax

Turns a list of scores into probabilities: exponentiate each score, divide by the sum. Bigger score, bigger share. → Quantization

Special token

A token with a fixed role that ordinary text never produces: begin of text, end of turn, start of the assistant's reply, tool-call markers. → Tokens and tokenizers

Tensor

A grid of numbers with a shape. [1024] is a list, [3072, 1024] is a table with 3072 rows and 1024 columns. Model files are named collections of tensors. → Anatomy of a model repo

Tied embeddings

tie_word_embeddings: true means the input embedding table and lm_head are the same matrix, saving vocab x hidden_size parameters. → Anatomy of a model repo

Top-1 agreement

Share of positions where the quantized model's most likely next token is the same as the original model's. → Quantization

Top-5 agreement

Share of positions where the quant's top token is among the original's 5 most likely tokens. Definitions vary between authors, so check theirs. → Quantization

Vocabulary

The full list of tokens a model knows. Qwen models use 151,936. The embedding table and the output layer each have one row per token. → Anatomy of a model repo

Zero point

An offset used by asymmetric quantization so the integer range can cover lopsided data, e.g. -0.1 to +0.3. → Quantization