SEC.03 / LLM Research
LLM Quantization Explained: GPTQ, AWQ, GGUF and QAT, How Much Memory You Need and How to Run Models Locally
A practical guide to quantizing large language models for laptops, phones and edge devices: how quantization works, post-training methods like GPTQ and AWQ, GGUF and llama.cpp, quantization-aware training as used in Apple's 2-bit on-device model, memory rules of thumb, runtimes and quality checks.