GeForce RTX 4060 Ti 16GB · Mistral Small 3.2 24B Instruct · Q4_K_M
Tight. Mistral Small 3.2 24B Instruct at Q4_K_M loads on GeForce RTX 4060 Ti 16GB at 8,192 tokens and uses most of the fast memory, roughly 10–16 tokens per second.
| Piece | Value | Note |
|---|---|---|
| Weights | 13.3 GiB | Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf |
| KV cache | 1.25 GiB | 8,192 tokens |
| Engine overhead | 0.50 GiB | Flat 0.5 GiB allowance |
| 5% headroom | 0.75 GiB | On the sum above |
| Total | 15.9 GiB | Tight |
| Fast memory | 16.0 GiB | 16 GB VRAM |
| Speed | 10–16 tok/s | Dense bandwidth range |
GeForce RTX 4060 Ti 16GB · Mistral Small 3.2 24B Instruct ·GGUF file · Methodology
FAQ
How was this fit calculated?
Weights 13.3 GiB, KV cache 1.25 GiB at 8,192 tokens, engine overhead 0.50 GiB, then 5% headroom, for 15.9 GiB. Fast memory available is 16.0 GiB.
What if I change the quant or the context?
This page is Q4_K_M at 8,192 tokens. A smaller quant or a shorter context lowers the total. The homepage calculator changes both without a new page.
Can I trust the tokens per second?
The range 10–16 tok/s is a bandwidth roofline, 55% to 80% efficient. This dense model reads the whole weight file each token. Measure your own box before you buy around it.
Is GeForce RTX 4060 Ti 16GB assumed to have extra RAM?
Yes. Discrete GPU pages assume 32 GB of system RAM. Offload uses whatever remains after 8 GB for Windows, and it is limited by the slower of DDR5-5600 and the PCIe link.