Qwen3-8B is 399 tensors, 36 layers that look alike and a handful of numbers. This post takes the model apart down to the last matrix: the name of every tensor in the checkpoint, its exact shape, how many numbers it holds, which rows of a matrix belong to which attention head, how the shapes change on their way through one layer, and how rotary position encoding really rotates. Everything is derived from the model’s published configuration and checked against the model repository: adding up all tensors reproduces the size of the weight files to the byte.

How to use it: every section has a widget. Click, type, drag.

Contents

1. Eight numbers define the network
2. How to read a shape
3. The checkpoint: 399 tensors (interactive)
4. The blueprint: one forward pass with exact shapes
5. The matrices drawn to scale (interactive)
6. One layer, operation by operation (interactive)
7. Attention in exact dimensions: heads, rows and rotations (interactive)
8. Why these numbers: layers, heads and key/value heads (interactive builder)
9. Check it yourself
10. Limits of this post and sources

1. Eight numbers define the network

The whole architecture of Qwen3-8B is fixed by a few entries of its config.json (source, read on 2026-10-20). I give them short symbols, because every shape in this post is built from them:

From these follow the derived sizes that appear everywhere:

q_dim  = n_q  × d = 32 × 128 = 4,096     (width of the query vector, equals H here)
kv_dim = n_kv × d =  8 × 128 = 1,024     (width of the key vector and of the value vector)
g      = n_q / n_kv = 4                  (query heads that share one key/value head)
I      = 3 × H                           (the MLP is three times as wide as the stream)
V      = 1,187 × 128                     (a multiple of 128, chosen to suit GPU kernels)

A few more config entries describe what is absent, and that matters for counting tensors:

• attention_bias: false: the projections have no bias vectors, only weight matrices.
• tie_word_embeddings: false: the embedding table and the output head are two separate matrices.
• sliding_window: null: every token may look at all earlier tokens.
• torch_dtype: bfloat16: every number is stored in 2 bytes. max_position_embeddings is 40,960 and rms_norm_eps is 1e-6.

2. How to read a shape

A shape such as [B, T, 4096] lists the sizes of the dimensions of a tensor:

• B is the batch: how many sequences are processed side by side.
• T is the number of tokens processed in this pass. It is the length of the prompt when the prompt is read, and 1 when one new token is generated.
• L is the length of the keys and values the attention looks at: the tokens already in the cache plus the T new ones.

A linear layer multiplies every token vector with a weight matrix. PyTorch stores that matrix as [out_features, in_features] and computes

y = x @ W.T          with  x: [..., in]   W: [out, in]   y: [..., out]
y[r] = dot( W[r, :], x )          row r of W produces output number r

So rows are outputs and columns are inputs. q_proj.weight has the shape [4096, 4096], k_proj.weight has [1024, 4096] (1,024 outputs, each reading all 4,096 inputs), and down_proj.weight has [4096, 12288]. Reshapes (view) and transposes only change how the same numbers are indexed, they do not copy or compute anything. Remember the convention: whenever this post says “rows 512 to 639 of q_proj”, it means 128 output numbers.

3. The checkpoint: 399 tensors

The weights of the model live in five safetensors files, plus a small index (model.safetensors.index.json) that maps the name of every tensor to the file that contains it. I read that index from the repository. It lists 399 tensors, and its metadata says that together they occupy 16,381,470,720 bytes. The tensors fall into three groups:

• model.embed_tokens.weight: the embedding table (1 tensor).
• model.layers.{i}.… for i = 0 … 35: 11 tensors per layer, 396 in total.
• model.norm.weight and lm_head.weight: the final norm and the output head (2 tensors).

The 11 tensors of a layer are four attention matrices, three MLP matrices and four small RMSNorm scale vectors: two for the input of each block and two for the heads (q_norm and k_norm, 128 numbers each, applied to every query and key head).

Use the explorer to list the tensors of any layer, filter by name, and see the exact shapes, parameter counts, byte sizes and the shard file of each tensor. The sums at the bottom are computed from the shapes, not looked up.

Two things worth noticing in the table. First, there is no .bias tensor anywhere, which is what attention_bias: false means in practice. Second, the shard files do not split exactly at layer boundaries: a layer such as layer 7 or 17 is spread over two files, simply because the exporter filled one file up to its size limit and continued in the next.

4. The blueprint: one forward pass with exact shapes

This is the complete path of a batch of token IDs through the network. Boxes show the tensors that hold weights (with the exact shape of the stored matrix), arrows carry the shapes of the data that flows through. The belt on the left is the residual stream: the vector x of 4,096 numbers per token. Each of the two blocks of a layer reads x, computes a correction and adds it back (the circles with a plus).

Four details in the picture deserve a sentence each.

• Two residual adds per layer. The attention output and the MLP output are never written over x, they are added to it. The stream keeps its shape [B, T, 4096] all the way from the embedding to the final norm.
• Q, K and V are born with different widths. The query has 32 heads, the key and the value only 8. They meet in the attention step, where every key/value head is read by four query heads.
• The value does not pass through the norm and RoPE box. Only the query and the key are normalised per head and rotated. The value goes straight into the attention.
• The MLP is the wide part. For a moment the stream swells from 4,096 to 12,288 numbers per token (gate and up), is combined element by element, and is squeezed back by down_proj.

5. The matrices drawn to scale

Numbers like 16,777,216 are hard to feel. Here every matrix of one layer is drawn with its true proportions: 1,024 rows or columns equal 28 pixels, so the area of a rectangle is proportional to the number of parameters. A matrix is drawn as stored: height = rows = outputs, width = columns = inputs. The thin lines mark bands of 128 (one attention head). Click a matrix to see exactly what its rows and columns mean.

What the picture shows at a glance:

• k_proj and v_proj are slim. Each has a quarter of the rows of q_proj. That is the visible form of grouped-query attention: 8 key/value heads instead of 32.
• The MLP dominates. Its three matrices are 78% of the weights of a layer (151.0 M of 192.9 M). The four attention matrices share the remaining 21.7%.
• down_proj is the gate_proj turned on its side. Same number of parameters, rows and columns swapped.
• The embedding table and the output head are each worth 3.2 layers. Together with the 36 layers they make up the 8.19 B parameters, and 15% of all parameters sit outside the layers.

6. One layer, operation by operation

Now the same layer as a table: every operation in the order it runs, with the weight tensor it uses, the exact shapes going in and out, the number of floating-point operations (FLOPs) and the size of its output. Choose how many sequences you process (B), how many tokens are already in the cache (past) and how many tokens this pass processes (T). Then L = past + T.

Three presets cover the two situations that matter: reading a prompt (many tokens at once, empty cache) and generating (one new token, long cache).

How the FLOPs are counted: a matrix product costs 2 × (outputs) × (multiply-adds), which is exact. The small element-wise operations (norms, RoPE, softmax, SiLU) are rough estimates and marked with ≈. They are tiny next to the matrix products.

What the table teaches:

• Where the arithmetic is. Every matrix product has FLOPs proportional to B × T. The two attention products are proportional to B × T × L. When you generate one token (T = 1) the seven weight matrices dominate. When you read a long prompt (T = L = 8,000) the attention products grow with the square of the prompt and take a visibly larger share. The table counts the full T × L score matrix. Kernels that skip the masked upper triangle do roughly half of these attention products during a prompt pass.
• The weights are read once per pass, whatever B and T are. The memory line “weight matrices, read once per pass” does not change when you change the batch or the number of tokens. Processing more tokens per pass therefore reuses every weight more often. That is the reason a prompt is cheap per token and a single generated token is not.
• The score tensor is huge in principle. For an 8,000-token prompt the matrix [1, 32, 8000, 8000] would hold 2.05 billion numbers (4.1 GB in bf16) per layer. Attention kernels of the FlashAttention family never write it to memory, they compute it block by block. The row only shows what a naive implementation would have to store.
• The cache grows by T rows per pass. The append step writes only the new keys and values, 2 × B × T × 8 × 128 numbers per layer.

7. Attention in exact dimensions: heads, rows and rotations

Attention is the part where names like “head” and “group” stay vague until you see the indices. Two questions have exact answers:

1. Which rows of q_proj, k_proj and v_proj belong to which head?
2. What exactly does RoPE do to the 128 numbers of a head?

Which rows belong to which head

The output of q_proj is a vector of 4,096 numbers per token. It is cut into 32 consecutive bands of 128: numbers 0 to 127 are query head 0, numbers 128 to 255 are query head 1, and so on. k_proj and v_proj produce only 1,024 numbers, cut into 8 bands of 128. The link between the two is plain integer division: query head h uses key/value head ⌊h / 4⌋.

Move the slider to select a query head. The three bars are the outputs of the three projections, aligned so that the four query heads of a group sit exactly above their shared key/value head.

Some exact facts that follow from the picture:

• Cache layout. Per layer the cache holds keys and values of shape [B, 8, L, 128] each. One token adds 2 × 8 × 128 numbers, which is 4,096 bytes in bf16, or 147,456 bytes over the 36 layers.
• Sharing is just indexing. Reference implementations expand the 8 cached heads to 32 by repeating each one four times in a row (heads 0, 0, 0, 0, 1, 1, 1, 1, …). Fast kernels skip the copy and simply read KV head ⌊h / 4⌋.
• What multi-head attention would cost. With 32 key/value heads, k_proj and v_proj would be [4096, 4096] instead of [1024, 4096] and the cache four times larger.
• The causal mask. Query token t (at position past + t) may only look at keys with an index up to past + t. All other scores are set to minus infinity before the softmax, so they get exactly zero weight. During generation (T = 1) the single query sees every cached key, so the mask changes nothing.

How the rotation works

Before the keys are stored and before queries meet keys, RoPE rotates the 128 numbers of each head. The numbers are paired half a head apart: number i is paired with number i + 64, for i = 0 … 63. Each pair is treated as a point in a plane and turned by the angle

φ(p, i) = p × θᵢ        with   θᵢ = 1,000,000 ^ ( −i / 64 )        p = position of the token

a′ =  a·cos φ − b·sin φ        (a, b) = (x[i], x[i + 64])
b′ =  a·sin φ + b·cos φ

So every token position turns pair i by p × θᵢ radians. Pair 0 turns fastest, by exactly 1 radian per position. Pair 63 turns about one millionth of that. The result is a clock with 64 hands at very different speeds. Because rotations compose, the dot product of a rotated query and a rotated key depends only on the distance between the two tokens, which is why the model can use position without storing absolute positions anywhere.

Pick a frequency and a position. The circle shows where the hand points, the bars show the wavelength (the number of positions per full turn) of all 64 pairs on a logarithmic scale.

8. Why these numbers: layers, heads and key/value heads

With the exact shapes in hand, the questions behind the numbers can be asked precisely.

What is a “hidden layer”? A neural network has an input (here: token IDs), an output (here: a score for every dictionary entry) and a lot of computation in between. Everything in between is called hidden, because nobody reads those intermediate vectors directly. In Qwen3-8B the hidden layers are the 36 blocks model.layers.0 … model.layers.35, and hidden_size is the width of the vector that travels between them. The config counts blocks, although every block itself contains several smaller steps.

Why 36 layers? It is a design choice, not a law of nature. One layer does one round of “gather information from the other tokens” (attention) followed by one round of “think about it” (MLP). Stacking layers lets later rounds build on the results of earlier ones: depth buys sequential steps of refinement. The exact number follows from the size budget. The weights of one layer are

attention:  q_proj H² + o_proj H² + k_proj ¼H² + v_proj ¼H²  =  2.5 H²
MLP:        gate 3H² + up 3H² + down 3H²                     =  9   H²
per layer:  11.5 × 4,096² = 192,937,984   (plus 8,448 norm weights)

total:      36 × 192,937,984  +  2 × 151,936 × 4,096   =   6.95 B + 1.24 B   =   8.19 B

The same budget could be spent on fewer, wider layers or on more, narrower ones. The builder below shows how steeply the layer count changes with the width. I cannot tell you why the designers chose exactly this point.

What is an attention head? One independent attention reader. The 4,096 query numbers are cut into 32 bands of 128 (section 7), and each band computes its own scores and its own softmax, so each head can focus on different earlier tokens. Splitting is free in arithmetic: 32 × 128 = 4,096, the same matrix, just indexed in bands. Studies of transformer models in general found heads that follow the previous word, heads that copy repeated text and heads that match a number to its label. I did not verify what the heads of Qwen3-8B specifically do.

Why only 8 key/value heads? Memory. Keys and values of all earlier tokens are kept in the cache, so their size is set by the number of key/value heads:

bytes per token  =  2 (K and V) × 36 layers × n_kv × 128 × 2 bytes
n_kv = 32 (classic multi-head attention):  589,824 bytes  →  4.7 GB for 8,000 tokens
n_kv =  8 (Qwen3-8B):                      147,456 bytes  →  1.2 GB for 8,000 tokens
n_kv =  1 (multi-query attention):          18,432 bytes  →  0.15 GB for 8,000 tokens

Grouped-query attention, the design with 1 < n_kv < n_q, was introduced as a middle way: the paper reports quality close to multi-head attention at a speed comparable to multi-query attention, while pure multi-query attention can cost quality. Eight also divides 32 evenly and is a common head count when a model is split across eight GPUs, but I cannot verify that this was Qwen’s reason.

Assemble it yourself

Press Build it to watch the network being put together piece by piece: first the embedding table, then layer after layer, finally the output head, while the parameter counter climbs towards 8.19 B. Then change the design. The tower is drawn with width = hidden size and height = number of layers. Its layers are split into an attention part (green) and an MLP part (purple) by their share of the weights, and the dashed outline is Qwen3-8B for comparison.

Things to try:

• Keep the parameter count near 8.19 B and change the hidden size. The card “layers needed for 8.19 B” tells you how deep the tower would have to be.
• Raise the number of layers and watch the KV cache per token grow in proportion.
• Raise the KV heads up to the number of query heads (classic multi-head attention) and see the cache explode.

What the builder shows, in short:

• Width versus depth at the same size. About 8.19 B parameters can be reached with roughly 22 layers at width 5,120, 36 at 4,096 or about 66 at 3,072 (if the other ratios are kept). Qwen’s designers picked one point on this line.
• Depth grows the cache. Doubling the layers doubles the KV cache per token.
• Width costs weights quadratically. One layer has about 11.5 × hidden² weights, so a 25% wider model has about 56% more weights per layer.
• Heads are free, KV heads are not. Changing the number of query heads (which follows the hidden size at head size 128) does not change the arithmetic. Changing the number of KV heads changes the cache directly.

9. Check it yourself

Everything in this post can be verified on your own machine. The first snippet reads the shape of every tensor from a safetensors file without loading a single weight. The format starts with 8 bytes that give the length of a JSON header, and the header lists every tensor with its dtype and shape.

import json, struct

def read_header(path):
    with open(path, "rb") as f:
        n = struct.unpack("<Q", f.read(8))[0]       # length of the JSON header
        return json.loads(f.read(n))

h = read_header("model-00005-of-00005.safetensors")
for name, t in h.items():
    if name != "__metadata__":
        print(name, t["dtype"], t["shape"])         # lm_head.weight BF16 [151936, 4096]

The second snippet loads the model and counts the tensors and parameters. (Newer versions of transformers call the argument dtype instead of torch_dtype.)

from transformers import AutoModelForCausalLM

m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="auto")
params = list(m.named_parameters())
print(len(params), sum(p.numel() for _, p in params))      # 399  8190735360

layer = m.model.layers[0]
print(layer.self_attn.q_proj.weight.shape)                  # torch.Size([4096, 4096])
print(layer.self_attn.k_proj.weight.shape)                  # torch.Size([1024, 4096])
print(layer.mlp.down_proj.weight.shape)                     # torch.Size([4096, 12288])

W = layer.self_attn.q_proj.weight
head_13 = W[13 * 128 : 14 * 128]                            # the 128 rows of query head 13: [128, 4096]

10. Limits of this post and sources

What I checked against the repository (2026-10-20):

• The values of config.json.
• The names of all 399 tensors and their grouping into five files, from model.safetensors.index.json.
• The total size of 16,381,470,720 bytes from the same index. It equals 8,190,735,360 parameters × 2 bytes, which is exactly the sum of the shapes used in this post. The five computed file sizes (4.00, 3.99, 3.96, 3.19 and 1.24 GB) agree with the file listing.

What follows from the config but I did not read directly:

• The shape of each individual tensor. I did not open the file headers. Because the sum of all shapes reproduces the total byte count exactly, a wrong shape would have to be compensated by another wrong shape, which is unlikely. The snippets in section 9 settle it in a minute.
• The order and the details of the operations (norm of query and key before RoPE, the pairing of number i with i + 64, query head h reading key/value head ⌊h / 4⌋). They follow the Hugging Face transformers implementation of Qwen3 as I know it. Other inference engines may store weights in a permuted layout, but they compute the same function.
• The vocabulary has 151,936 rows in the embedding table. As far as I know the tokenizer itself defines somewhat fewer tokens and the table is padded up to a multiple of 128.

What is an estimate: the FLOPs of the element-wise operations (norms, RoPE, softmax, SiLU) are rough counts, marked with ≈. The matrix products are exact.

What is not known: why the designers chose 36 layers, 8 key/value heads or a 3× wider MLP, and what the individual heads and layers of this model do. Statements about this come from research on transformer models in general.

Sources

• Model configuration: Qwen3-8B config.json
• Weight index: model.safetensors.index.json and the file listing of the repository
• The file format: safetensors
• Grouped-query attention: Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023)
• The transformer: Vaswani et al., Attention Is All You Need
• Rotary position embedding: Su et al., RoFormer
• RMSNorm: Zhang and Sennrich, Root Mean Square Layer Normalization
• SwiGLU: Shazeer, GLU Variants Improve Transformer