125 Billion Parameters in 32GB RAM: Benchmarking Sushi, Qwen, and Thinking Tokens on Apple M4

2026-10-10 2082 words 10 minutes

Contents

Let’s pause for a moment to appreciate the sheer, glorious absurdity of local AI in late 2026.

Just three years ago, if you wanted to run a model with over 100 billion parameters, you needed a server chassis the size of a mini-fridge, an electrical circuit that could power a laundromat, and a bank loan to pay for four NVIDIA A100 GPUs. If someone told you that you would soon run a 125-billion parameter reasoning model on a standard 32GB consumer Mac, you would have politely recommended they seek medical attention.

And yet, here I am sitting at my desk with an Apple M4 (10 cores, 32 GB of Unified Memory), watching an ultra-dense MoE model casually navigate a cognitive gauntlet of 15 notorious brain-teasers—streaming internal thinking traces in real time, dodging every classic cognitive trap, and generating tokens at 13 to 17 tokens per second.

No cloud APIs. No monthly subscription tokens. Zero fan noise.

Here is the story of how we hooked up Coni to Sushi (the high-performance MLX inference engine), built a standalone benchmark arena in coni-cli-apps, and put Qwen3.8-Flash-Next 125B through the ultimate reasoning stress test.

1. Meet Sushi: The Frontier MLX Engine for Apple Silicon

If you haven’t bumped into Sushi yet, it is not your standard generic GGUF/llama.cpp wrapper.

Sushi is an Apple Silicon-first inference engine built directly on Apple Metal & MLX. Instead of trying to support every toy architecture from 2021, Sushi is purpose-engineered for three modern frontier architectures:

qwen4_exp (Qwen3.8-Flash-Next MoE)
mimo_v2 (MiMo-V2.6-Flash)
glm5_next (GLM-5.3-Flash)

The magic lies in how Sushi handles memory. A 125-billion parameter model—even quantized down to 2 bits per weight (2bpw)—is still around 35 GB to 65 GB on disk. On a 32 GB Mac, traditional loaders immediately crash with an Out Of Memory panic.

Sushi solves this with tiered dynamic expert offloading and multi-token prediction (MTP) speculative decoding:

Sushi keeps the dense base layers in unified memory, while routed experts are dynamically paged between an in-memory cache and local NVMe storage with speculative drafting. The result? A 125B frontier model actually boots and answers questions on a consumer Mac.

2. Is 12–16 Tokens/s Actually Fast? (The Memory Math)

When you run your first prompt and see ~13–16 tokens/second, your modern internet brain might initially wonder: “Wait, shouldn’t it be 60 tokens per second?”

Let’s do the physical math of Apple Silicon memory bandwidth.

Apple M4 (Base): Has a memory bandwidth of 120 GB/s.
For an autoregressive model whose weights reside fully in memory, the theoretical speed limit is: $$\text{Speed (tok/s)} \approx \frac{\text{Memory Bandwidth}}{\text{Active Model Footprint}}$$
A 3B model (~2 GB in RAM) can hit 50–60 tok/s.
But Qwen3.8-Flash-Next has 125 BILLION parameters.

Those 60–90 tok/s screenshots you see floating around X/Twitter come from gaming rigs equipped with an RTX 5070 or RTX 4090, which boast 500 to 1,000+ GB/s of dedicated VRAM bandwidth.

On a single laptop chip with 120 GB/s bandwidth and 32 GB of RAM, pulling 13 to 17 tokens per second on a 125-billion parameter MoE model while streaming expert weights from disk isn’t slow—it is a minor computational miracle.

3. The 15 Logic Traps: Building the Arena

We didn’t want to test the model with trivial tasks like “Write a haiku about coffee”. Any 1B model can write a haiku.

We wanted to know: Can it actually think? Does its internal chain-of-thought reason through cognitive traps, counter-intuitive wordplay, and trick math questions?

We wrote a standalone Coni CLI app in /Users/nico/cool/s5/coni-cli-apps/sushi-bench/ that connects to Sushi’s OpenAI-compatible endpoint (http://127.0.0.1:12345/v1/chat/completions), sets reasoning_effort: "medium", and streams 15 notorious reasoning prompts:

The Sheep: “A farmer has 17 sheep. All but 9 die. How many sheep are left?” (Classic trap: intuitive brain says 8, correct is 9).
The Strawberry: “How many times does the letter ‘r’ appear in ‘strawberry’?” (The classic LLM tokenizer blind spot).
The Widgets: “If 5 machines take 5 minutes to make 5 widgets, how long do 100 machines take to make 100 widgets?” (Trap answer: 100. Correct: 5).
The Bat and Ball: “A bat and ball cost $1.10. The bat costs $1.00 more than the ball. How much is the ball?” (Trap answer: $0.10. Correct: $0.05).
Sally’s Family: “Sally has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?” (Trap: 2. Correct: 1).
Gold vs. Feathers: “Which is heavier: an ounce of gold or an ounce of feathers?” (Trick: Troy vs. Avoirdupois ounces!).
The Calendar Trap: “If yesterday’s tomorrow was Monday, what day is today’s tomorrow?”
Mary’s Father: “Mary’s father has 5 daughters: Nana, Nene, Nini, Nono, and who?”
The Footrace: “You are in a race and pass the person in 2nd place. What position are you in?” (Trap: 1st. Correct: 2nd).
The Cat Rate: “If 3 cats catch 3 mice in 3 minutes, how many cats are needed to catch 100 mice in 100 minutes?”
The Border Crash: “A plane crashes on the border of the US and Canada. Where do they bury the survivors?”
The Table Apples: “There are 3 apples on a table and you take away 2. How many apples do you have?”
The Word Riddle: “I am an odd number. Take away a letter and I become even. What number am I?”
The Fisherman: “Two fathers and two sons catch one fish each, but only 3 fish are caught in total. How?”
The 28-Day Month: “How many months in the Gregorian calendar have 28 days?”

4. Running the Benchmark

Firing up the benchmark app from our terminal:

coni /Users/nico/cool/s5/coni-cli-apps/sushi-bench/main.coni

The app streams the internal thinking traces, records Time to First Token (TTFT), pure generation speed, and dynamically builds an updating table after every single question.

Here are a few highlights from the run:

Highlight 1: The Strawberry Test (Prompt #2)

Question #2 : How many times does the letter 'r' appear in the word 'strawberry'?
Answer       : To determine the number of times the letter 'r' appears in the word 'strawberry',
               let's break down the word character by character:
               1. s  2. t  3. r (1st)  4. a  5. w  6. b  7. e  8. r (2nd)  9. r (3rd)  10. y
               Answer: The letter 'r' appears 3 times in the word 'strawberry'.
---------------------------------------------------------
Time to First Token (TTFT)  : 2.45 s
Pure Generation Time        : 10.98 s
⚡ Pure Generation Speed     : 16.04 tokens/sec

By forcing character-level decomposition inside its <think> block, Qwen completely sidestepped BPE tokenization blindness and counted all 3 ‘r’s without hesitation.

Highlight 2: The Gold vs. Feathers Trap (Prompt #6)

Most humans (and smaller models) proudly answer: “They weigh the same, an ounce is an ounce!” Look at what Qwen 125B did:

Question #6 : Which is heavier: an ounce of gold or an ounce of feathers?
Answer       : An ounce of gold is heavier.

Reasoning:
1. Gold is measured in troy ounces; feathers are measured in avoirdupois ounces.
2. 1 troy ounce = 31.103 grams.
   1 avoirdupois ounce = 28.349 grams.
3. 31.103g > 28.349g. Therefore, an ounce of gold weighs more.
---------------------------------------------------------
⚡ Pure Generation Speed     : 13.4 tokens/sec

That is pure, unadulterated domain knowledge.

Highlight 3: The Plane Crash (Prompt #11)

Question #11 : A plane crashes on the border of the US and Canada. Where do they bury the survivors?
Answer       : They don't bury survivors. Survivors are still alive.
---------------------------------------------------------
Time to First Token (TTFT)  : 4.77 s
Pure Generation Time        : 1.47 s
⚡ Pure Generation Speed     : 10.24 tokens/sec

Instant catch. Zero hesitation.

5. The Final Results Table

After cycling through all 15 prompts, our Coni script rendered the full scoreboard:

=============================================================================================================
                                            FINAL BENCHMARK TABLE                                            
=============================================================================================================
+---+----------------------------------+--------------------------------+--------+---------+---------+---------+-------------+-------------+
| # | Question                         | Final Answer                   | Tokens | TTFT    | Gen (s) | Tot (s) | Gen tok/s   | E2E tok/s   |
+---+----------------------------------+--------------------------------+--------+---------+---------+---------+-------------+-------------+
|  1 | A farmer has 17 sheep. All bu... | The phrase "all but 9 die" ... |     35 |    4.3s |   2.35s |   6.65s |   14.88 t/s |    5.26 t/s |
|  2 | How many times does the lette... | To determine the number of ... |    176 |   2.45s |  10.98s |  13.42s |   16.04 t/s |   13.11 t/s |
|  3 | If it takes 5 machines 5 minu... | To determine the time requi... |    294 |   4.22s |  17.88s |  22.11s |   16.44 t/s |    13.3 t/s |
|  4 | A bat and a ball cost $1.10 i... | Let the cost of the ball be... |    255 |   4.22s |  16.62s |  20.84s |   15.35 t/s |   12.24 t/s |
|  5 | Sally has 3 brothers. Each br... | Here is the step-by-step lo... |    254 |   4.38s |  15.11s |  19.49s |   16.81 t/s |   13.03 t/s |
|  6 | Which is heavier: an ounce of... | An ounce of gold is heavier... |    146 |   3.94s |   10.9s |  14.84s |    13.4 t/s |    9.84 t/s |
|  7 | If yesterday's tomorrow was M... | Let's break this down step ... |    164 |   3.78s |  10.79s |  14.57s |    15.2 t/s |   11.26 t/s |
|  8 | Mary's father has 5 daughters... | The answer is **Mary**.  He... |     80 |   3.61s |   7.02s |  10.62s |    11.4 t/s |    7.53 t/s |
|  9 | You are running a race and pa... | You are in **2nd** place.      |      9 |   3.15s |   0.53s |   3.68s |      17 t/s |    2.44 t/s |
| 10 | If 3 cats catch 3 mice in 3 m... | To solve this problem, we n... |    445 |   3.38s |  40.54s |  43.93s |   10.98 t/s |   10.13 t/s |
| 11 | A plane crashes on the border... | They don't bury **survivors... |     15 |   4.77s |   1.47s |   6.23s |   10.24 t/s |    2.41 t/s |
| 12 | There are 3 apples on a table... | You have 2 apples.  **Reaso... |     49 |   4.17s |   6.21s |  10.38s |    7.89 t/s |    4.72 t/s |
| 13 | I am an odd number. Take away... | The number is **seven**.  R... |     68 |    3.3s |   8.04s |  11.35s |    8.45 t/s |    5.99 t/s |
| 14 | Two fathers and two sons catc... | The situation is possible b... |    156 |   4.13s |  15.71s |  19.84s |    9.93 t/s |    7.86 t/s |
| 15 | How many months in the Gregor... | In the Gregorian calendar, ... |    154 |    3.8s |  13.35s |  17.16s |   11.53 t/s |    8.98 t/s |
+---+----------------------------------+--------------------------------+--------+---------+---------+---------+-------------+-------------+

==================================== EXECUTIVE SUMMARY ====================================
  Model Tested               : beamster/Qwen3.8-Flash-Next-Sushi-2bpw
  Questions Answered         : 15 / 15 (100% Accuracy)
  Total Completion Tokens    : 2,300 tokens
  Average TTFT               : 3.84 s
  Average Pure Gen Time      : 11.83 s
  Average Total Time         : 15.67 s
  -----------------------------------------------------------------------------------------
  ⚡ Mean Pure Generation Speed : 13.03 tokens/sec
  ⚡ Peak Generation Speed      : 17.00 tokens/sec
  ⚡ Mean End-to-End Speed      : 8.54 tokens/sec
===========================================================================================

6. How Coni Drives the Benchmark in 30 Lines

One of the great things about building this in Coni is how painless it is to configure an autonomous agent loop with streaming callbacks, token timestamps, and telemetry hooks:

(def sushi-agent
  (make-agent {:api-url "http://127.0.0.1:12345/v1/chat/completions"
               :api-key "none"
               :model "beamster/Qwen3.8-Flash-Next-Sushi-2bpw"
               :options {:reasoning_effort "medium"}
               :stream-text true
               :stream-fn (fn [chunk]
                            (when (nil? @*first-token-ts*)
                              (reset! *first-token-ts* (sys-time-now)))
                            (swap! *token-count* inc)
                            (swap! *full-text* str chunk))
               :usage-fn (fn [usage]
                           (reset! *usage-data* usage))}))

Coni automatically:

Negotiates the streaming HTTP connection over SSE.
Extracts streaming reasoning_content deltas as the model thinks.
Fires :usage-fn with server-reported completion tokens and microsecond timestamps.
Allows us to compute precise pure-generation speed independently from prompt evaluation latency (TTFT).

7. What This Means for Local AI

Here is the bottom line:

We just evaluated a 15-question cognitive trap battery against a 125B parameter model on a 32GB laptop, and it scored 15 out of 15.

It did not fall for the sheep trap.
It correctly analyzed troy ounces vs avoirdupois ounces.
It counted letters in strawberry correctly.
It kept an average Time-to-First-Token under 3.8 seconds.
It generated clean, articulated reasoning traces at 13 to 17 tokens/second.

If you’re building local autonomous coding agents, multi-agent swarms, or privacy-critical data pipelines, the equation has officially changed. You no longer need to compromise between “small and fast but hallucinates on edge cases” vs “smart but trapped behind a cloud API rate limit”.

With engines like Sushi pushing the envelope of Apple Silicon MLX and languages like Coni coordinating the orchestration, the frontier is already running locally on your desk.

Happy hacking!