Meta · 1.2B · Q4_K_M · 0.75 GiB

Llama 3.2 1B Instruct at Q4_K_M needs about 1.58 GiB at 8K context, and the smallest listed machine that fits it is GeForce RTX 3060 Ti.

1B dense instruct model. The architecture numbers are the GGUF header, because Meta's config.json is access-gated.

ollama run llama3.2:1b

Weights: Llama-3.2-1B-Instruct-Q4_K_M.gguf. Architecture: config source. Hugging Face repo created 2024-09-18. License field: llama3.2.

Quants with a measured file size

QuantFileSize
Q4_K_M · recommendedLlama-3.2-1B-Instruct-Q4_K_M.gguf0.75 GiB
Q5_K_MLlama-3.2-1B-Instruct-Q5_K_M.gguf0.85 GiB
Q6_KLlama-3.2-1B-Instruct-Q6_K.gguf0.95 GiB
Q8_0Llama-3.2-1B-Instruct-Q8_0.gguf1.23 GiB

Machines

Same quant and 8,192 tokens on every row.
MachineMemoryTotalVerdictSpeed
Apple M1, 8 GB8 GB unified1.58 GiBFits35–67 tok/s
Apple M2, 8 GB8 GB unified1.58 GiBFits51–99 tok/s
Apple M3, 8 GB8 GB unified1.58 GiBFits51–99 tok/s
GeForce RTX 3060 Ti8 GB GDDR1.58 GiBFits229–444 tok/s
GeForce RTX 30708 GB GDDR1.58 GiBFits229–444 tok/s
GeForce RTX 40608 GB GDDR1.58 GiBFits139–269 tok/s
GeForce RTX 4060 Ti 8GB8 GB GDDR1.58 GiBFits147–285 tok/s
GeForce RTX 50608 GB GDDR1.58 GiBFits229–444 tok/s
GeForce RTX 5060 Ti 8GB8 GB GDDR1.58 GiBFits229–444 tok/s
Radeon RX 66008 GB GDDR61.58 GiBFits114–222 tok/s
Radeon RX 76008 GB GDDR61.58 GiBFits147–285 tok/s
GeForce RTX 3080 10GB10 GB GDDR1.58 GiBFits388–753 tok/s
GeForce RTX 3060 12GB12 GB GDDR1.58 GiBFits184–357 tok/s
GeForce RTX 407012 GB GDDR1.58 GiBFits258–499 tok/s
GeForce RTX 4070 Super12 GB GDDR1.58 GiBFits258–499 tok/s
GeForce RTX 4070 Ti12 GB GDDR1.58 GiBFits258–499 tok/s
GeForce RTX 507012 GB GDDR1.58 GiBFits343–666 tok/s
Intel Arc B58012 GB GDDR61.58 GiBFits233–452 tok/s
Radeon RX 6700 XT12 GB GDDR61.58 GiBFits196–380 tok/s
Radeon RX 7700 XT12 GB GDDR61.58 GiBFits221–428 tok/s
Apple M1 Pro, 16 GB16 GB unified1.58 GiBFits102–198 tok/s
Apple M1, 16 GB16 GB unified1.58 GiBFits35–67 tok/s
Apple M2 Pro, 16 GB16 GB unified1.58 GiBFits102–198 tok/s
Apple M2, 16 GB16 GB unified1.58 GiBFits51–99 tok/s
Apple M3, 16 GB16 GB unified1.58 GiBFits51–99 tok/s
Apple M4, 16 GB16 GB unified1.58 GiBFits61–119 tok/s
GeForce RTX 4060 Ti 16GB16 GB GDDR1.58 GiBFits147–285 tok/s
GeForce RTX 4070 Ti Super16 GB GDDR1.58 GiBFits343–666 tok/s
GeForce RTX 408016 GB GDDR1.58 GiBFits366–710 tok/s
GeForce RTX 4080 Super16 GB GDDR1.58 GiBFits376–729 tok/s
GeForce RTX 5060 Ti 16GB16 GB GDDR1.58 GiBFits229–444 tok/s
GeForce RTX 5070 Ti16 GB GDDR1.58 GiBFits458–887 tok/s
GeForce RTX 508016 GB GDDR1.58 GiBFits491–951 tok/s
Intel Arc A770 16GB16 GB GDDR61.58 GiBFits262–507 tok/s
Legion Go16 GB LPDDR5X-75001.58 GiBFits61–119 tok/s
Radeon RX 6800 XT16 GB GDDR61.58 GiBFits262–507 tok/s
Radeon RX 7800 XT16 GB GDDR61.58 GiBFits319–618 tok/s
Radeon RX 9060 XT 16GB16 GB GDDR61.58 GiBFits164–317 tok/s
Radeon RX 9070 XT16 GB GDDR61.58 GiBFits327–634 tok/s
ROG Ally16 GB LPDDR5-64001.58 GiBFits52–101 tok/s
Steam Deck LCD16 GB LPDDR5-55001.58 GiBFits45–87 tok/s
Steam Deck OLED16 GB LPDDR5-64001.58 GiBFits52–101 tok/s
Apple M3 Pro, 18 GB18 GB unified1.58 GiBFits77–149 tok/s
Radeon RX 7900 XT20 GB GDDR61.58 GiBFits409–792 tok/s
Apple M2, 24 GB24 GB unified1.58 GiBFits51–99 tok/s
Apple M3, 24 GB24 GB unified1.58 GiBFits51–99 tok/s
Apple M4 Pro, 24 GB24 GB unified1.58 GiBFits140–270 tok/s
Apple M4, 24 GB24 GB unified1.58 GiBFits61–119 tok/s
GeForce RTX 309024 GB GDDR1.58 GiBFits478–927 tok/s
GeForce RTX 409024 GB GDDR1.58 GiBFits515–998 tok/s
Radeon RX 7900 XTX24 GB GDDR61.58 GiBFits491–951 tok/s
ROG Ally X24 GB LPDDR5X-75001.58 GiBFits61–119 tok/s
Apple M1 Max, 32 GB32 GB unified1.58 GiBFits204–396 tok/s
Apple M1 Pro, 32 GB32 GB unified1.58 GiBFits102–198 tok/s
Apple M2 Max, 32 GB32 GB unified1.58 GiBFits204–396 tok/s
Apple M2 Pro, 32 GB32 GB unified1.58 GiBFits102–198 tok/s
Apple M4, 32 GB32 GB unified1.58 GiBFits61–119 tok/s
GeForce RTX 509032 GB GDDR1.58 GiBFits916–1775 tok/s
Apple M3 Pro, 36 GB36 GB unified1.58 GiBFits77–149 tok/s
Apple M4 Max 14-core, 36 GB36 GB unified1.58 GiBFits210–406 tok/s
Apple M3 Max 16-core, 48 GB48 GB unified1.58 GiBFits204–396 tok/s
Apple M4 Max 16-core, 48 GB48 GB unified1.58 GiBFits279–541 tok/s
Apple M4 Pro, 48 GB48 GB unified1.58 GiBFits140–270 tok/s
Apple M1 Max, 64 GB64 GB unified1.58 GiBFits204–396 tok/s
Apple M2 Max, 64 GB64 GB unified1.58 GiBFits204–396 tok/s
Apple M3 Max 16-core, 64 GB64 GB unified1.58 GiBFits204–396 tok/s
Apple M4 Max 16-core, 64 GB64 GB unified1.58 GiBFits279–541 tok/s
Apple M4 Pro, 64 GB64 GB unified1.58 GiBFits140–270 tok/s
Apple M2 Max, 96 GB96 GB unified1.58 GiBFits204–396 tok/s
Apple M3 Max 16-core, 128 GB128 GB unified1.58 GiBFits204–396 tok/s
Apple M4 Max 16-core, 128 GB128 GB unified1.58 GiBFits279–541 tok/s

Exact answers

Apple M1, 8 GBfits

FAQ

Why Q4_K_M?

Q4_K_M is the quant this page recommends. The file is Llama-3.2-1B-Instruct-Q4_K_M.gguf at 0.75 GiB. Other single-file quants are in the table when a real size was available.

Where do the architecture numbers come from?

GGUF metadata: llama.block_count, embedding_length, head counts, key_length, context_length. Parameter count is gguf.total from the Hugging Face model API. Source: https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/blob/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf

Does this include a long context?

The fit uses 8,192 tokens, or less when that is all the fast memory can hold. The model file allows up to 131,072 tokens, and KV cache grows as you use more of that.

What is the Ollama command?

ollama run llama3.2:1b. The library tag is Q4-class or the vendor quant. Compare it with the GGUF linked on this page before you treat the sizes as identical.