Bolmo in Nature: How Ai2 Converts Subword LLMs Into Byte-Level Models, and When That Matters
Ai2's Bolmo paper in Nature shows how to retrofit existing language models to read raw bytes instead of tokens. What byteification is, how Bolmo, Bwen and Blama perform, what the two-stage process costs, and which tasks benefit.
Most large language models never see text directly. They split it into subword tokens from a fixed vocabulary, which works well for ordinary prose but handles spelling, unusual strings, code identifiers and non-English scripts less gracefully. Byte-level models avoid that by reading the raw bytes of text, but they have historically lagged behind and had to be trained from scratch. On October 7, 2026, Nature published Ai2’s paper “Retrofitting language models to operate over bytes”, which describes a way to convert an existing subword model into a byte-level one with a relatively short additional training run. This article explains the method, the results and where it is likely to matter.
What “byteification” means
Ai2 calls the process byteifying. Instead of throwing away a trained model and pretraining a byte-level model from random initialisation, it keeps the original model’s core (the “global model”) and adds new byte-level components around it:
- a local encoder that reads bytes and builds contextual representations;
- a boundary predictor that decides where to group bytes into patches, which the global model then processes as single steps;
- a decoder that turns the global model’s output back into bytes.
Patching is what keeps byte-level inference practical. Without it, the main transformer would have to process every byte, making sequences several times longer. One of the paper’s technical changes is a non-causal boundary predictor, which looks at more context than earlier designs when deciding where patches end.
Conversion runs in two stages. In stage 1, the original global model stays frozen while the new byte-level parts learn. In stage 2, the whole model trains together. The paper’s compute accounting puts stage 1 at about twice the global model’s FLOPs and stage 2 at about three times. In an ablation that skipped stage 1, the authors added 6.5 billion tokens to stage 2 to match compute, lengthening it by 17%.
The models
According to Ai2’s announcement, Bolmo 1B and Bolmo 7B were first introduced in December 2025, converted from Ai2’s open Olmo models. Ai2 describes them as what it believes are the first fully open byte-level models competitive with strong subword models across a broad range of tasks. With the Nature paper, Ai2 applied the same process to two other families:
| Model | Converted from | Notes |
|---|---|---|
| Bolmo 1B / 7B | Olmo | Fully open weights, data and recipe |
| Bwen 8B | Qwen 3 8B | Ai2’s strongest byteified model so far |
| Blama 8B | Llama 3 8B | Close to its source model |
Ai2 also released the stage 1 checkpoints, so researchers can experiment with the byte-level components without paying for a full conversion.
Key results
The paper reports several findings:
- Better than earlier byte-level models. The byteified models outperformed, on average, all earlier publicly available byte-level models of comparable size. Bolmo 7B scored 16.5 points higher on STEM tasks than BLT 7B, a byte-level model trained from scratch. That gap mainly shows the value of starting from a trained model; it is not a claim that Bolmo beats every subword 7B model on STEM.
- Much better at character-level tasks. Bolmo 7B “greatly outperformed” its source model, Olmo 3, on character understanding, and was better in some coding settings.
- Close to the source model. Bwen 8B reached performance “close to and sometimes surpassing” the original Qwen model.
- Adjustable speed. Byteified models can be sped up further by training with more bytes per patch, which subword models can do only to a limited extent.
- Ecosystem reuse. Existing components from the source model’s ecosystem can be used to adapt a byteified model without extra training cost.
One caution on reading the results: the paper’s ablation figures report 95% confidence intervals over 53,879 evaluation instances, but each plotted point is a single 150,000-step training run. The intervals reflect sampling of test questions, not variation between training runs.
Why this matters
The case for byte-level models has never been mainly about general benchmarks. It is about tasks where subword tokenisation gets in the way:
- Spelling and character manipulation, such as counting letters, fixing typos or handling deliberately misspelled input.
- Code and identifiers, where meaning depends on exact characters and naming conventions rarely match a tokenizer’s vocabulary.
- Scientific strings such as chemical notations and biological sequences, which the paper highlights as a key motivation.
- Non-English text, since subword vocabularies tend to favour English and compress other scripts less efficiently.
The paper’s conclusion lists potential benefits for computational efficiency and energy use, reduced English-centric bias, and fine-grained scientific text. These are described as potential advantages, not demonstrated across the board.
The bigger change is economic. If converting a model costs only a few times the global model’s compute for a short run, byte-level versions can follow new subword models rather than lagging years behind, which has been the main obstacle to adoption.
Byte-level versus subword: a quick comparison
| Aspect | Subword model | Byteified model |
|---|---|---|
| Input unit | Tokens from a fixed vocabulary | Raw UTF-8 bytes grouped into learned patches |
| Spelling and character tasks | Often weak, since characters are hidden inside tokens | Stronger, according to the paper’s character-understanding results |
| New scripts or unusual strings | Split into many rare tokens | Handled as ordinary bytes |
| Speed control | Fixed by the tokenizer | Adjustable through the bytes-per-patch ratio |
| Ecosystem | Mature tooling | Can reuse the source model’s components, per the paper |
How to try it
The Bolmo, Bwen and Blama checkpoints are on Hugging Face, linked from Ai2’s announcement. For a meaningful test:
- Pick tasks where tokenisation is a known problem in your work, such as code with unusual identifiers, multilingual text or structured scientific strings.
- Compare the byteified model directly with its source model on the same prompts, so you isolate the effect of bytes versus tokens.
- Measure latency at your target bytes-per-patch setting; speed depends heavily on that ratio.
- For general-knowledge or multiple-choice tasks, do not expect gains; the source model already has that knowledge.
For a related look at how input representation shapes model performance, see our coverage of two Nature papers on LLM limits in science.
Our analysis
Bolmo does not mean tokenizers are going away, but it removes the strongest practical argument against byte-level models: that they must be trained from scratch and can never keep up. By converting existing models and getting close to their original performance while improving character-level tasks, Ai2 has made byte-level language modelling something researchers and developers can test cheaply. The most promising uses are in code, multilingual and scientific text, where how text is split has always affected results.