LLM Quantization Explained: GPTQ, AWQ, GGUF and QAT, How Much Memory You Need and How to Run Models Locally

A practical guide to quantizing large language models for laptops, phones and edge devices: how quantization works, post-training methods like GPTQ and AWQ, GGUF and llama.cpp, quantization-aware training as used in Apple's 2-bit on-device model, memory rules of thumb, runtimes and quality checks.

PUBLISHED // UPDATED
DURATION // 07 MIN READ
LLM Quantization Explained: GPTQ, AWQ, GGUF and QAT, How Much Memory You Need and How to Run Models Locally
FIG.01 // VISUAL DOCUMENTATION LLM Quantization Explained: GPTQ, AWQ, GGUF and QAT, How Much Memory You Need and How to Run Models Locally

Running a capable language model on a laptop or phone used to be impractical. Today, tools such as Ollama and LM Studio make it routine, and Apple ships a roughly 3-billion-parameter model on its devices. The technique that makes this possible is quantization: storing a model’s weights with fewer bits so it needs less memory and runs faster. This guide explains how quantization works, the main methods and formats, how much memory different models need and how to check that a quantized model is still good enough.

What quantization does

Neural network weights are usually stored as 16-bit floating-point numbers (FP16 or BF16). A 7-billion-parameter model therefore needs about 14 GB just for its weights. Quantization maps those values onto a smaller set of numbers represented with 8, 4 or even 2 bits. Fewer bits mean less memory and less data to move from memory to the processor, which is often the real bottleneck when generating text, so quantized models usually run faster as well as fitting on smaller devices.

The challenge is preserving quality. Not all weights are equally important: some channels and outlier values have an outsized effect on outputs. Modern methods protect those, spread quantization error carefully or let the model adapt to lower precision during training.

Two families of methods

Post-training quantization (PTQ) compresses a model after training, usually with a small calibration dataset. It is fast and cheap, which is why most community-quantized models use it.

  • LLM.int8() made 8-bit inference practical by handling outlier features separately.
  • GPTQ uses approximate second-order information to quantize weights to 3 or 4 bits with small accuracy loss.
  • AWQ protects the most important weight channels based on activation statistics and is widely used for 4-bit inference.
  • llama.cpp’s k-quants (such as Q4_K_M) mix precision across blocks of weights and are the common choice for GGUF files.

Quantization-aware training (QAT) simulates low precision during training or fine-tuning so the model learns to tolerate it. It costs more but can reach lower bit widths. Apple’s 2025 foundation model update says it compressed its roughly 3-billion-parameter on-device model to 2 bits per weight using QAT, combined with other techniques such as key-value cache sharing.

PrecisionMemory for a 7B model (weights only)Typical useQuality impact
16-bitAbout 14 GBReference quality; serversNone
8-bitAbout 7 GBHigh-quality inferenceUsually negligible
4-bitAbout 3.5–4.5 GB depending on formatDefault for laptops and consumer GPUsSmall for most tasks; larger for maths, code and long context
3-bit and belowUnder 3 GBPhones, very constrained devicesNoticeable with PTQ; QAT can do much better

The local AI toolchain

llama.cpp is the C/C++ inference engine behind much of local AI. It runs quantized models on CPUs, Apple silicon and GPUs, and its GGUF file format has become the standard for distributing quantized models.

Ollama wraps llama.cpp-style inference in a simple command-line tool and local server. A single command downloads and runs a model from its library, and applications can call it through an API.

LM Studio offers a desktop app with a model browser, chat interface and local server, making local models accessible without the command line (LM Studio).

Apple MLX is Apple’s framework for machine learning on Apple silicon, with tools for running and quantizing models.

vLLM targets high-throughput serving on GPUs and supports several quantization formats, making it the usual choice for servers rather than laptops.

What you can run on typical hardware

As a rough guide for 4-bit models, assuming some memory is left for the operating system and context:

Device memoryComfortable model sizeExamples of suitable open models
8 GB (phones, entry laptops)1–4B parametersLlama 3.2 1B and 3B, small Gemma and Phi models
16 GB laptops7–14B parametersMistral 7B, Llama 3.1 8B, Phi-4 (14B)
32–64 GB workstations or Macs30–70B parametersLarger Qwen, Llama and distilled reasoning models
Multi-GPU serversLarge and mixture-of-experts modelsFull-size open-weight frontier models

Speed depends on memory bandwidth, the processor and context length, so the same model can feel fast on one machine and sluggish on another. Mixture-of-experts models need memory for all their parameters even though only some are active per token.

Why local deployment matters

Privacy. Data stays on the device, which matters for healthcare, legal, financial and personal use.

Reliability. Local models work offline and are not subject to API rate limits or outages.

Cost. For steady, high-volume workloads, local or self-hosted inference can be cheaper than paying per token, though hardware and maintenance are real costs.

Latency. Removing the network round trip helps interactive uses such as autocomplete and voice.

The trade-off is capability. Local models are smaller than frontier cloud models and generally weaker on complex reasoning and broad knowledge, so many applications combine local models for routine tasks with cloud models for harder ones.

Key methods and primary sources

  • GPTQ (Frantar et al., 2022) — post-training quantization that uses approximate second-order information to quantize weights to 3–4 bits with small accuracy loss (arXiv:2210.17323).
  • AWQ (Lin et al., 2023) — activation-aware weight quantization that protects the most important weight channels, widely used for 4-bit inference (arXiv:2306.00978).
  • LLM.int8() (Dettmers et al., 2022) — 8-bit inference that handles outlier features separately (arXiv:2208.07339).
  • QLoRA (Dettmers et al., 2023) — fine-tuning with a frozen 4-bit (NF4) base model plus low-rank adapters, enabling fine-tuning of large models on a single GPU (arXiv:2305.14314).
  • GGUF and llama.cpp — the file format and C/C++ runtime most commonly used to run quantized models on laptops and phones, with quantization types such as Q4_K_M and Q8_0 (llama.cpp on GitHub).

A rough memory rule of thumb

Weight memory is approximately parameters × bits ÷ 8. A 7-billion-parameter model needs about 14 GB at 16-bit precision, about 7 GB at 8-bit, and about 3.5 GB at 4-bit, plus extra memory for the key-value cache, which grows with context length and batch size. This is why 4-bit quantization is the usual starting point for consumer hardware.

Choosing a runtime

  • llama.cpp / Ollama / LM Studio: Easiest path for local use on CPUs, Apple Silicon, and consumer GPUs.
  • Apple MLX: Apple’s array framework optimized for Apple Silicon, with an mlx-lm package for running and quantizing models (MLX on GitHub).
  • vLLM: High-throughput GPU serving with support for several quantization formats, better suited to servers than laptops (vLLM docs).
  • Mobile: Frameworks such as Google’s MediaPipe LLM Inference API and Apple’s Core ML target on-device deployment.

Quality checks before shipping

Quantization affects tasks unevenly: math, code, and long-context retrieval are often more sensitive than casual chat. Evaluate the quantized model on your own task set, compare against the full-precision version, and watch for increased repetition or formatting errors. Also confirm the model’s license allows your intended commercial use, since quantized files inherit the original license.

Our analysis

Quantization is what turned local AI from a curiosity into a practical option. Four-bit post-training quantization is the sensible default for laptops and consumer GPUs, while quantization-aware training, as in Apple’s 2-bit on-device model, shows how far compression can go when the developer controls training. The key habits are to size models by memory, test quantized versions on your own tasks, especially maths, code and long inputs, and check licences before deploying.