Memory is the weight file, the KV cache, a runtime allowance, and 5% headroom.

A verdict is not a benchmark. It answers whether the published GGUF, at 8,192 tokens, fits in the fast memory of a machine. Specs that could not be read from a config, a GGUF header, or a vendor page were left out.

Worked example: Qwen3 8B on an RTX 4070

The Q4_K_M file is 4.68 GiB (Qwen3-8B-Q4_K_M.gguf). KV cache at 8,192 tokens is 1.13 GiB, from 36 full-attention layers, 8 KV heads, and head dim 128 inconfig.json, stored as fp16. Engine overhead is a flat 0.5 GiB. Five percent headroom brings the total to 6.62 GiB. The 4070 has 12 GB, so the verdict is fits. Active-weight fraction for this dense model is 1.00.

Weights

Weight memory is the byte size of one GGUF file on Hugging Face. It is not parameters times a guessed bits-per-weight. Split archives were skipped. When the recommended file is Unsloth UD-Q4_K_M or OpenAI MXFP4, the page says that name.

KV cache

Full attention bytes are layers × KV heads × head dim × tokens × 2 × 2 bytes. Sliding layers use the same formula with min(tokens, sliding window). Gemma 3 uses sliding_window_pattern 6: every 6th layer is full, and the rest use a 1,024-token window. Gemma 4 follows layer_types from config.json. Full layers use global_head_dim and num_global_key_value_heads when that count is present. When it is null, as on E2B and E4B, full layers keep num_key_value_heads. That matches the E4B GGUF header, which stores one KV head count and a separate sliding key length.

Qwen3.5, Qwen3.6, and Qwen3.8 linear layers do not grow a KV cache with context. Their state is value heads × key dim × value dim, in the config's float32, plus a short convolution of kernel 4 over the Q, K, and V channels. That shape is the one llama.cpp-compatible runtimes use for this architecture.

Verdicts

Fits: total ≤ 85% of fast memory. Tight: total still fits, or the weights fit and at least 1,024 tokens of context fit. Offload: the total fits in VRAM plus system RAM after an 8 GB Windows reserve, but not in VRAM alone. No: it does not fit in that combined pool.

Discrete pages assume 32 GB of system RAM unless you type another number on the homepage. Unified memory, on Macs and handhelds, keeps max(3 GB, 25% of the pool) for the OS, capped at 12 GB. There is no offload tier, because there is no second memory pool.

Speed

The high number is 80% of vendor GB/s, treated as 1e9 bytes per second, divided by the bytes of active weights. The low number is 55% of bandwidth divided by active weights plus the KV cache. Dense models read the whole file. MoE models read an architecture-derived fraction: active experts over total experts, with attention, shared experts, and embeddings always on. Offload splits that traffic between GPU bandwidth and the slower of DDR5-5600 (89.6 GB/s) and a practical PCIe ceiling (24 GB/s for Gen 4, 48 GB/s for Gen 5).

AMD pages use raw GDDR bandwidth. Infinity Cache is tens of megabytes and does not hold a model, so the "effective" gaming bandwidth is not used.

What was left out

Llama 4 Scout has a public GGUF, including a sharded Q4_K_M around 61 GiB, but the official config is gated and the attention pattern is not a standard cache. It is not in the catalog. Ollama commands appear only when the library tag was checked against that model. Mistral Small 3.2 and Devstral Small 2 2512 do not get a command, because the live tags point at earlier builds. Vision projectors are not in the weight total.