OPTICAL EDITION // 2026.11.12 / SIGNAL-TO-NOISE ARCHITECTURE
f/1.4 — ∞ / EDITORIAL STANDARD
01 // Dispatches 02 // AI Tools 03 // Generative 04 // LLM Research 05 // Systems 06 // Dossier

Apple MLX Runs a Large Open Model on Four Mac Studios at 36 Tokens/s: When a Desk Cluster Makes Sense

Apple's MLX team ran Meta's Muse Spark Open on four Mac Studios at more than 36 tokens per second. The memory math, how the Thunderbolt 5 RDMA pipeline works, its limits, and when a desk-sized Mac cluster makes sense.

PUBLISHED // UPDATED
DURATION // 07 MIN READ
DESK // The Focus Digital Editorial Team
QUADRANT // SEC.03 (LLM Research)
Two desktop workstations linked by a short Thunderbolt cable on a lab bench, cool overhead light, no logos
FIG.01 // VISUAL DOCUMENTATION Two desktop workstations linked by a short Thunderbolt cable on a lab bench, cool overhead light, no logos

Apple’s machine learning team has shown that four Mac Studios linked over Thunderbolt 5 can run Meta’s Muse Spark Open, a very large open-weight model, at more than 36 tokens per second. The demonstration relies on several pieces Apple has built over the past year: M5 Ultra hardware, distributed inference in the MLX framework, and remote direct memory access (RDMA) over Thunderbolt 5. It shows that frontier-scale open models can run on desk-sized hardware, but it also comes with specific conditions that matter for anyone considering a similar setup.

This article explains what Apple reported, why the model needs multiple machines, how the two-machine and four-machine configurations work, and when a small Mac cluster is a sensible choice.

What Apple reported

Apple’s research post on MLX distributed inference states that a cluster of four Mac Studios generates more than 36 tokens per second on Muse Spark Open at an 8,192-token context, which Apple describes as “well above average human reading speed.” Reading speed is a useful reference for interactive use, but it is not a standard benchmark, and the figure applies to that specific context length.

Apple also reports a 4.6× improvement in startup time (time to first token) relative to a comparison configuration that is not fully detailed. Time to first token and generation speed are different measures, so the 4.6× figure should not be read as a speed-up in token generation.

Why one Mac Studio is not enough

Apple’s post sets out the memory arithmetic:

Model and precisionApproximate memory for parametersFits one 512GB Mac Studio?
Qwen and Mistral models in the sizes Apple discussesUnder 512GBYes
Muse Spark Open, native BF16More than 1.6TBNo
Muse Spark Open, 4-bit NVFP4About 493GBNo, once intermediate values are included

Apple describes Muse Spark Open as a model of more than 800 billion parameters, nearly a trillion. Even at 4-bit precision, the weights alone come close to the 512GB of unified memory in a single M5 Ultra Mac Studio, leaving no room for activations and the key-value cache. Splitting the model across machines is therefore a necessity rather than an optimization.

Unified memory is the reason this is possible at all. Because the CPU and GPU share one memory pool, a Mac Studio with 512GB behaves more like a single dense compute node than a GPU with limited onboard memory. Apple also lists mixture-of-experts (MoE) support in MLX as part of this year’s work; MoE models activate only some of their parameters for each token, which helps make very large models practical at modest throughput. Apple’s post does not specify how many experts are active per token.

The building blocks

Apple credits several developments for making the cluster work:

  • M5 Ultra and Neural Accelerators in the M5 GPU family
  • RDMA over Thunderbolt 5, providing low-latency connections between several Macs
  • Distributed inference in MLX, including MoE implementations
  • JACCL, an open-source collectives library

How the two-machine setup works

TechTimes’ October 15 report describes the two-machine version as pipeline parallelism: the model’s transformer layers are split between two Mac Studios, and RDMA carries the activation data from the first machine to the second without passing through the CPU. From the developer’s perspective, the setup is a single mlx.launch command pointed at a hostfile listing the two machines, with no MPI configuration or hand-written parallel code required.

JACCL, the library underneath, provides collective operations such as AllReduce and AllGather. According to TechTimes, it is named after Jack Beasley (Thunderbolt 5 RDMA) and Angelos Katharopoulos (MLX), as a deliberate nod to NVIDIA’s NCCL, and was introduced at WWDC 2026. MLX calls it automatically.

Meta contributed GPU kernel optimizations for M5 Ultra. TechTimes cites launch-blog figures of 1.03× to 3.95× on key compute kernels. These are kernel-level measurements, not end-to-end generation speed, so they should not be combined into a single overall speed-up.

TechTimes also notes that the approach is not limited to Muse Spark Open: any open-weight model whose weights fit in the combined unified memory of a Thunderbolt-linked cluster can, in principle, use the same path. The practical test is simple addition: the quantized weights, plus the key-value cache, plus activations, must fit in the total memory available.

Week-in-review context

Computerworld’s roundup placed the announcement alongside Apple Intelligence plans for China and described the setup as “Mac Muse AI supercomputers.” The China news is a separate product and compliance story; for infrastructure planning, the relevant material is Apple’s research post and the technical details above.

Limits to keep in mind

Context length. The 36 tokens-per-second figure applies at 8,192 tokens. Longer prompts increase memory use and the amount of data passed between machines, so performance at, for example, 32,000 tokens will differ. Measure at the context length you actually use.

Pipeline behavior. Pipeline parallelism across two machines is different from tensor parallelism inside a multi-GPU server. If one machine stalls, there is no replica to take over; the pipeline stops. Collective operations can still hang, and cable quality and connection stability become part of operations.

No job scheduler. mlx.launch with a hostfile is a developer command, not a cluster scheduler like Slurm. If several people share the machines, decide in advance who controls them and when.

Power and placement. Apple’s post does not publish power figures for the cluster. Teams placing several Mac Studios together should plan power and cooling with facilities staff rather than assuming office conditions are sufficient.

Licensing. Fitting the weights in memory does not settle whether you can use them. Check the model’s license and access terms before planning a deployment.

When a desk cluster makes sense

SituationReasonable approach
You have one 512GB M5 Ultra Mac Studio and a model that fitsRun it on one machine and measure your own startup time and speed
You have two machines and a model of roughly 500GB at 4-bitUse the two-machine pipeline; test with your real prompts and context length
You have four machines and need very large models interactivelyYou may reach performance near Apple’s 36 tokens per second at 8K context; verify locally
Your team needs shared, scheduled, audited inferenceA managed GPU cluster or cloud service is usually a better fit
Your stack is vLLM on NVIDIATreat JACCL as background; it is not a drop-in replacement for NCCL on Linux

A small Mac cluster is well suited to research groups, individual developers, and small teams that want to experiment with large open models locally, keep data on hardware they control, and avoid per-token API costs. It is less suited to shared production services that need scheduling, redundancy, and compliance controls.

For more on running open models, see our articles on how open-weight models are taking share of AI usage and small models such as MiniCPM5-2B. For benchmark interpretation, see our analysis of IBM’s learned-guidelines agent on τ²-bench.

Bottom line

Apple’s demonstration shows that a model approaching a trillion parameters can run at interactive speed on four Mac Studios, thanks to unified memory, 4-bit quantization, MoE support, and RDMA over Thunderbolt 5. The key number to remember is the memory math: about 493GB for Muse Spark Open’s weights at NVFP4, which is why a single 512GB machine is not enough. For developers and small teams already working in MLX, the setup is now relatively simple to try; for shared production use, the usual requirements for scheduling, redundancy, and governance still apply.

Sources

CROSS-REFERENCE //

Adjacent Dispatches

FULL ARCHIVE →
[01] SEC.03

How LLM Hallucination Detection Works: Attention Graphs (CHARM), Output Signatures (LOS-Net), Semantic Entropy and Practical Safeguards

A guide to detecting LLM hallucinations: white-box methods that read attention and activations (CHARM, ICLR 2026), black-box methods using output probability distributions (LOS-Net), sampling-based checks like semantic entropy and SelfCheckGPT, their limits, and the safeguards production systems should use.

OPEN DOSSIER →
[02] SEC.03

AI Is Solving Math Problems Humans Couldn't — But It's Not the Whole Story

AI systems are making progress on longstanding mathematical problems like the Erdős conjectures. Here's what's happening, what's working, and where the limits still lie.

OPEN DOSSIER →
[03] SEC.03

Bolmo in Nature: How Ai2 Converts Subword LLMs Into Byte-Level Models, and When That Matters

Ai2's Bolmo paper in Nature shows how to retrofit existing language models to read raw bytes instead of tokens. What byteification is, how Bolmo, Bwen and Blama perform, what the two-stage process costs, and which tasks benefit.

OPEN DOSSIER →