Bolmo in Nature: How Ai2 Converts Subword LLMs Into Byte-Level Models, and When That Matters

Ai2's Bolmo paper in Nature shows how to retrofit existing language models to read raw bytes instead of tokens. What byteification is, how Bolmo, Bwen and Blama perform, what the two-stage process costs, and which tasks benefit.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
Workbench with a printed tokenizer vocabulary sheet beside a hex dump of the same sentence in bytes, cool lab light, no logos
FIG.01 // VISUAL DOCUMENTATION Workbench with a printed tokenizer vocabulary sheet beside a hex dump of the same sentence in bytes, cool lab light, no logos

Most large language models never see text directly. They split it into subword tokens from a fixed vocabulary, which works well for ordinary prose but handles spelling, unusual strings, code identifiers and non-English scripts less gracefully. Byte-level models avoid that by reading the raw bytes of text, but they have historically lagged behind and had to be trained from scratch. On October 7, 2026, Nature published Ai2’s paper “Retrofitting language models to operate over bytes”, which describes a way to convert an existing subword model into a byte-level one with a relatively short additional training run. This article explains the method, the results and where it is likely to matter.

What “byteification” means

Ai2 calls the process byteifying. Instead of throwing away a trained model and pretraining a byte-level model from random initialisation, it keeps the original model’s core (the “global model”) and adds new byte-level components around it:

  • a local encoder that reads bytes and builds contextual representations;
  • a boundary predictor that decides where to group bytes into patches, which the global model then processes as single steps;
  • a decoder that turns the global model’s output back into bytes.

Patching is what keeps byte-level inference practical. Without it, the main transformer would have to process every byte, making sequences several times longer. One of the paper’s technical changes is a non-causal boundary predictor, which looks at more context than earlier designs when deciding where patches end.

Conversion runs in two stages. In stage 1, the original global model stays frozen while the new byte-level parts learn. In stage 2, the whole model trains together. The paper’s compute accounting puts stage 1 at about twice the global model’s FLOPs and stage 2 at about three times. In an ablation that skipped stage 1, the authors added 6.5 billion tokens to stage 2 to match compute, lengthening it by 17%.

The models

According to Ai2’s announcement, Bolmo 1B and Bolmo 7B were first introduced in December 2025, converted from Ai2’s open Olmo models. Ai2 describes them as what it believes are the first fully open byte-level models competitive with strong subword models across a broad range of tasks. With the Nature paper, Ai2 applied the same process to two other families:

ModelConverted fromNotes
Bolmo 1B / 7BOlmoFully open weights, data and recipe
Bwen 8BQwen 3 8BAi2’s strongest byteified model so far
Blama 8BLlama 3 8BClose to its source model

Ai2 also released the stage 1 checkpoints, so researchers can experiment with the byte-level components without paying for a full conversion.

Key results

The paper reports several findings:

  • Better than earlier byte-level models. The byteified models outperformed, on average, all earlier publicly available byte-level models of comparable size. Bolmo 7B scored 16.5 points higher on STEM tasks than BLT 7B, a byte-level model trained from scratch. That gap mainly shows the value of starting from a trained model; it is not a claim that Bolmo beats every subword 7B model on STEM.
  • Much better at character-level tasks. Bolmo 7B “greatly outperformed” its source model, Olmo 3, on character understanding, and was better in some coding settings.
  • Close to the source model. Bwen 8B reached performance “close to and sometimes surpassing” the original Qwen model.
  • Adjustable speed. Byteified models can be sped up further by training with more bytes per patch, which subword models can do only to a limited extent.
  • Ecosystem reuse. Existing components from the source model’s ecosystem can be used to adapt a byteified model without extra training cost.

One caution on reading the results: the paper’s ablation figures report 95% confidence intervals over 53,879 evaluation instances, but each plotted point is a single 150,000-step training run. The intervals reflect sampling of test questions, not variation between training runs.

Why this matters

The case for byte-level models has never been mainly about general benchmarks. It is about tasks where subword tokenisation gets in the way:

  • Spelling and character manipulation, such as counting letters, fixing typos or handling deliberately misspelled input.
  • Code and identifiers, where meaning depends on exact characters and naming conventions rarely match a tokenizer’s vocabulary.
  • Scientific strings such as chemical notations and biological sequences, which the paper highlights as a key motivation.
  • Non-English text, since subword vocabularies tend to favour English and compress other scripts less efficiently.

The paper’s conclusion lists potential benefits for computational efficiency and energy use, reduced English-centric bias, and fine-grained scientific text. These are described as potential advantages, not demonstrated across the board.

The bigger change is economic. If converting a model costs only a few times the global model’s compute for a short run, byte-level versions can follow new subword models rather than lagging years behind, which has been the main obstacle to adoption.

Byte-level versus subword: a quick comparison

AspectSubword modelByteified model
Input unitTokens from a fixed vocabularyRaw UTF-8 bytes grouped into learned patches
Spelling and character tasksOften weak, since characters are hidden inside tokensStronger, according to the paper’s character-understanding results
New scripts or unusual stringsSplit into many rare tokensHandled as ordinary bytes
Speed controlFixed by the tokenizerAdjustable through the bytes-per-patch ratio
EcosystemMature toolingCan reuse the source model’s components, per the paper

How to try it

The Bolmo, Bwen and Blama checkpoints are on Hugging Face, linked from Ai2’s announcement. For a meaningful test:

  1. Pick tasks where tokenisation is a known problem in your work, such as code with unusual identifiers, multilingual text or structured scientific strings.
  2. Compare the byteified model directly with its source model on the same prompts, so you isolate the effect of bytes versus tokens.
  3. Measure latency at your target bytes-per-patch setting; speed depends heavily on that ratio.
  4. For general-knowledge or multiple-choice tasks, do not expect gains; the source model already has that knowledge.

For a related look at how input representation shapes model performance, see our coverage of two Nature papers on LLM limits in science.

Our analysis

Bolmo does not mean tokenizers are going away, but it removes the strongest practical argument against byte-level models: that they must be trained from scratch and can never keep up. By converting existing models and getting close to their original performance while improving character-level tasks, Ai2 has made byte-level language modelling something researchers and developers can test cheaply. The most promising uses are in code, multilingual and scientific text, where how text is split has always affected results.