Small Language Models Explained: How Phi, Gemma, Llama 3.2 and Apple's On-Device Models Compete, and When to Use One
Why small language models have become capable enough for real work: data quality, synthetic data and distillation, what Phi-3, Phi-4, Gemma, Llama 3.2 and Apple's on-device model show, where small models win and fall short, and how to decide between a small, large or hybrid setup.
For several years, progress in language models seemed to come mainly from scale: more parameters, more data and more compute. GPT-2 had 1.5 billion parameters and GPT-3 had 175 billion, and later frontier models are believed to be far larger, though most labs no longer disclose sizes. Alongside that race, a second trend has become just as important for everyday use. Small language models, typically from under a billion to around 15 billion parameters, now handle many tasks well enough to run on a laptop or phone. This guide explains why they improved, what the leading examples show and how to decide when one is the right choice.
Why small models got better
Data quality. Microsoft’s Phi series made the case most clearly. The Phi-3 technical report describes phi-3-mini, a 3.8-billion-parameter model trained on 3.3 trillion tokens of heavily filtered web data and synthetic data. Microsoft reported that it rivals models such as Mixtral 8x7B and GPT-3.5 on academic benchmarks, scoring 69% on MMLU, while being small enough to run on a phone. Its successor, the 14-billion-parameter phi-4, uses synthetic data throughout training, and Microsoft says it substantially surpasses its teacher model, GPT-4, on STEM-focused question answering.
Distillation. Smaller models trained on the outputs of larger ones inherit much of their behaviour at a fraction of the cost. DeepSeek’s distilled R1 models, for example, brought step-by-step reasoning to models small enough to run locally.
Specialisation and adapters. A small model fine-tuned for a narrow task such as classification, extraction or routing often matches a general large model on that task. Apple’s on-device model uses small adapter modules fine-tuned for specific features, swapping them in as needed.
Better tooling. Quantisation, which stores weights at lower precision such as 4 bits, and efficient runtimes on phones and laptops have made local deployment far more practical.
Representative small models
| Model | Developer | Size | Notes |
|---|---|---|---|
| phi-3-mini | Microsoft | 3.8B | Trained on filtered web and synthetic data; phone-deployable |
| phi-4 | Microsoft | 14B | Data-quality-focused training with synthetic data; strong on reasoning benchmarks for its size |
| Gemma | Several small sizes | Open-weight models with published terms of use | |
| Llama 3.2 1B and 3B | Meta | 1B, 3B | Lightweight text models designed for on-device use |
| Apple on-device foundation model | Apple | About 3B | Powers Apple Intelligence features with task-specific adapters |
Sources: the Phi-3 and phi-4 reports, Google’s Gemma documentation, Meta’s Llama 3.2 announcement and Apple Machine Learning Research.
Where small models win
Latency. Running locally removes the network round trip to a cloud service, which matters for real-time tasks such as voice interfaces, autocomplete and live transcription.
Privacy and data control. When a model runs on a device or inside your own infrastructure, the data does not need to go to a third party. That is important in healthcare, legal, financial and other settings with strict data rules, and it is central to Apple’s pitch for on-device processing.
Cost at volume. For high-volume, repetitive work such as tagging, extraction, intent routing or moderation pre-screening, a small model can be far cheaper than calling a large model for every request, whether it runs locally or on modest cloud hardware. The exact saving depends on hardware, utilisation and the price of the alternative, so calculate it for your own workload.
Offline reliability. Local models keep working without an internet connection or when an API is rate-limited or down.
Where small models fall short
Small models still generally lag on complex multi-step reasoning, broad factual knowledge, long-context tasks and less common languages. They have less capacity to store knowledge, so they are more likely to give confident wrong answers outside their training focus, and benchmark results achieved through targeted training do not always carry over to messy real-world inputs. Microsoft’s own reports note that phi-3-mini’s limited size restricts how much factual knowledge it can hold, which is why pairing small models with search or retrieval often helps.
The hybrid approach
In practice, many systems combine sizes. A small model handles routine requests quickly and cheaply and escalates harder ones to a larger model. Apple Intelligence works this way: Apple describes a roughly 3-billion-parameter on-device model alongside a larger server model running on its Private Cloud Compute infrastructure, with requests to external models such as ChatGPT requiring the user’s permission.
The same pattern appears in enterprise systems, where a small model routes or classifies requests, extracts structured data or drafts simple responses, and a larger model is reserved for complex reasoning.
How to decide
| Question | If yes | If no |
|---|---|---|
| Is the task narrow and well defined (classify, extract, route)? | A small, possibly fine-tuned model is likely enough | Test a larger model |
| Must data stay on the device or in your infrastructure? | Small local or self-hosted model | Cloud APIs are an option |
| Is volume high and cost per request important? | Small model, with escalation for hard cases | Pay for quality where volume is low |
| Does the task need broad knowledge or long reasoning chains? | Large model, or a small model with retrieval | Small model may suffice |
| Must it work offline or on a phone? | Small quantised model | No constraint |
Before deciding, build a test set of real examples, compare a small and a large model on accuracy, latency and cost, and check each model’s licence, because open-weight licences differ in their commercial terms.
Common questions
What counts as a “small” language model? There is no fixed threshold. The term usually covers models from a few hundred million to roughly 15 billion parameters, small enough to run on a single consumer GPU, a laptop or, at the lower end, a phone.
Can I run one on my own computer? Yes. Tools such as Ollama and LM Studio download and run quantised versions of open models like Phi, Gemma and Llama 3.2 with little setup, though speed depends on your hardware and memory.
Our analysis
Small language models have not made large models obsolete, but they have changed the default for many everyday tasks. Careful data curation, synthetic data and distillation let models of a few billion parameters handle classification, extraction, drafting and on-device assistance well, with clear benefits for latency, privacy and cost. The most robust designs combine small models for routine work with larger models for the hard cases, and teams should choose based on testing with their own data rather than headline benchmarks.