Running GGUF Models Natively with Coni and Apple MLX

Picture this: You see a shiny new Open Source LLM drop on HuggingFace. It promises to be the next big thing in local reasoning. You want to try it out on your MacBook. So you spin up your terminal.

You pip install some packages. You get an error about torch being compiled without Metal support. You switch to conda. You update your environment. You wait 15 minutes for 4 gigabytes of dependencies to download. You run the Python script. You get a ValueError: weight shape mismatch because your transformers library is exactly 0.0.1 versions out of date. You update transformers, which breaks accelerate.

Deep breath.

What if running a cutting-edge Large Language Model on your Mac was as simple as executing a single, statically-compiled binary? What if you could hook directly into the GPU without ever touching a Python virtual environment?

Welcome to the Native Coni MLX Bridge.

Escaping Python Dependency Hell

When we started building Agent Studio, we needed an orchestration layer that was portable, deterministic, and extremely lightweight. We wanted to build autonomous, multi-agent orchestrations that just worked out of the box. Asking users to maintain brittle Python virtual environments, manage CUDA/Metal drivers, and fight with dependency resolution was a non-starter.

To achieve this, we bypassed Python entirely.

Coni is a fast, standalone, Clojure-like interpreter written in Go. To bring LLMs natively into this ecosystem, we built a CGO (C to Go) bridge that connects the Coni execution engine directly to Apple’s MLX C++ API.

The result is a paradigm shift in local inference. Coni can now natively parse, load, and execute HuggingFace GGUF models directly on Apple Silicon GPUs. There is no Python runtime overhead. There are no dependencies. You just pass a .gguf file to a Coni script, and Apple’s Unified Memory Architecture handles the rest. Because CPU and GPU share the same memory pool on M-series chips, we don’t even have to transfer tensor data across a slow PCIe bus. The model just loads into RAM and the GPU instantly begins evaluating the matrix math.

The Benchmark: Balancing Speed and Reasoning

By hooking directly into Apple’s Metal backend, we unlock ridiculous inference speeds. But “speed” in the world of LLMs is entirely relative to the architecture and parameter count of the model you choose to load. Let’s look at three distinct weight classes we tested natively in Coni.

The Speed Demons: Qwen2.5 (0.5B) and TinyLlama (1.1B)

For small, rapid-response models like Qwen2.5 0.5B or TinyLlama 1.1B, the speed is absolutely staggering. On an M4 Mac, we consistently hit over 200 tokens per second.

At that speed, the model generates entire paragraphs of text faster than you can even read the first sentence. It looks like the text is just instantly materializing on your screen. But why would you use such small models?

In a multi-agent system like Agent Studio, you don’t need a massive 70-Billion parameter model to do simple routing or classification. You can use TinyLlama as a lighting-fast “router agent” that reads a user’s prompt and decides which tool to call in 15 milliseconds. It acts as the nervous system of the application—blindingly fast, highly specialized, and practically free to run.

The Heavy Thinker: Qwen3 (4B)

But what about more complex, heavy reasoning tasks? We recently tested the brand new Qwen3 4B Instruct architecture natively in Coni.

Here is the raw output from our poem-generator.coni script running the Qwen3 4B model (using the q8_0 GGUF quantization format):

[Metal GPU] Loading native GGUF from disk: models/qwen3-4b-instruct-2507-heretic-q8_0.gguf

Enter a theme for your poem: waterfall
[Config] Detected GGUF architecture: qwen3  (Family: qwen3 )

[Composing...]

Beneath the hush of endless blue,  
The ocean breathes—a rhythm deep and true.  

No shore can hold its song,  
It flows like memory—soft, vast, and free.  

And when the moon dips low,  
The ocean whispers—I am the world’s first breath.

[Generation complete. Hit token evaluation bound. Speed: 14.3 t/s. Total tokens: 101]

At 14.3 tokens per second, a heavy 4-Billion parameter reasoning model feels incredibly snappy and responsive. It trades the 200+ tok/s blazing speed of TinyLlama for deep reasoning, context awareness, and beautiful prose. Best of all? It runs completely silently on a laptop without making the GPU cooling fans scream.

The Forward Pass & True WebSocket Streaming

When you run LLMs through generic HTTP REST APIs (like Ollama or OpenAI), you are often subjected to the dreaded “Time To First Token” (TTFT) latency. Even with HTTP chunked streaming, you are dealing with network overhead, JSON serialization, and buffering delays.

Because Coni executes the forward pass loop natively in C++, we have full control over the evaluation loop. Every single time the MLX engine evaluates a token on the GPU, it is instantly yielded back to the Coni runtime in the exact same memory space. This gives us true, zero-latency per-token streaming.

This is especially powerful when paired with Coni’s native concurrency primitives (Goroutines and Channels) and its built-in WebSocket library (libs/ws).

In projects like Agent Studio, we don’t wait for a response. We simply pipe the output of the native MLX loop into a background channel, and broadcast the generation to the browser in real-time as the tokens are born.

Here is what it looks like to stream an LLM directly to a web client with zero overhead:

;; Inside a Coni WebSocket handler
(let [prompt-toks (sys-tokenizer-encode tk-path prompt)]
  
  ;; Create a buffered channel to hold the incoming tokens
  (let [out-chan (chan 100)]
    
    ;; Spawn a background worker (Goroutine) to run the MLX forward pass.
    ;; llm/generate-fast evaluates tokens natively and pushes them into out-chan.
    (spawn (fn [] 
             (llm/generate-fast prompt-toks map-obj 1024 tk-path config nil 0 out-chan)))
    
    ;; In our main thread, we loop over the channel and stream tokens 
    ;; through the websocket the instant they arrive from the GPU.
    (loop []
      (let [token (<! out-chan)]
        (when (not (nil? token))
          (ws/send-text ws-client (sys-tokenizer-decode tk-path [token]))
          (recur))))))

Look closely at that code. There is no intermediate web server. There is no Global Interpreter Lock (GIL) forcing threads to wait in line. There is no REST API. The GPU computes a matrix multiplication, identifies the token ID, pushes it into a Go channel, and a WebSocket blasts it to the browser.

It is a beautiful, uninterrupted pipeline of data. And it makes building interactive CLI tools, multi-agent swarms, or local web chatbots feel perfectly fluid to use.

By stripping away the layers of abstraction that have plagued the AI ecosystem, we haven’t just made local inference easier—we’ve made it fundamentally better.