Fast Native LLM Inference in Coni: The Poem Generator

In our ongoing quest to make Coni the most capable, dependency-free, and relentlessly fun functional language out there, we’ve hit a massive milestone. We didn’t just write an AI script—we went fully native.

Today, we’re unveiling our first fully native AI example: The Poem Generator.

Using the highly capable Qwen 2.5 0.5b instruct model formatted as a raw GGUF binary, this new Coni script takes a theme, invokes a local LLM, computes a beautiful poem, and spits out characters at a shockingly fast rate of over 70 tokens per second (TPS).

Wait—how is a dynamically typed, interpreted Lisp-like language pulling off GPU-bound speeds that rival dedicated C++ inference engines? Is it magic? No. It’s raw engineering, memory packing, and a newly developed Native MLX C++ CGO Bridge.

Grab your coffee. Let’s dive deep into how we pulled this off. ☕️

The Epic Battle: Lisp vs Latency

Historically, performing massive tensor math in an interpreted language is an exercise in futility.

Imagine you’re doing a matrix multiplication. If you represent a matrix as a native Coni Vector containing 9.4 million Float elements, your interpreter is allocating millions of individual nodes on the heap. When it comes time to crunch the numbers, the Go garbage collector kicks in, memory fragmentation slows everything down, and the Abstract Syntax Tree (AST) overhead causes a single operation to take several minutes.

You could read a whole real book while waiting for a single forward pass of a transformer block. That’s unacceptable.

Enter ast.Tensor: Packing the Memory

To achieve our ridiculous token generation speed, we introduced a new underlying primitive to Coni’s runtime: the ast.Tensor.

Instead of an array of scattered AST pointer nodes, ast.Tensor allocates a single, highly packed, contiguous block of raw memory. When you execute an AI operation in Coni today, we aren’t iterating over a list of variables—we are dealing with solid bricks of C-compatible memory.

The Warp Pipe: Apple Silicon (Metal) & C++

Having contiguous memory is great, but the CPU is still too slow for LLM inference. We needed the GPU.

We could have taken the easy route and just built a Python wrapper around mlx-lm or PyTorch. But Coni is built on the philosophy of zero external runtime dependencies. So, we went straight to the metal—literally.

We mapped Coni directly to Apple’s native machine learning framework, MLX, bypassing Python entirely. Here’s the architectural stack that makes it happen:

1. The C++ Core (libmlx.dylib): We took mlx::core C++ classes and wrapped them into opaque, C-compatible handles.
2. Go Builtins via CGO: The Coni interpreter binds this C-API, cleanly and safely mapping the ast.Tensor contiguous data block from Go directly to the GPU.
3. Unified Memory Magic: Because Apple Silicon (M1/M2/M3/M4) uses unified memory, the transfer cost across the CGO boundary is practically zero. We don’t have to serialize data and shove it over a slow PCIe bus; the moment Go allocates the ast.Tensor, it’s already sitting in memory that the Metal GPU can read!

Lazy Evaluation: Keeping Data on the GPU

The real secret sauce is how we treat the math. By mapping operations like matmul, softmax, and add directly to the MLX compute graph using our opaque handles, we keep the data entirely on the GPU.

Coni acts purely as the orchestrator. When you call (mlx/matmul a b), the C++ engine just queues up the operation (lazy evaluation). The GPU doesn’t do the work until it absolutely has to. Only when Coni invokes (mlx/read ...) does the data forcefully evaluate and materialize back into a Go ast.Tensor.

Because the entire multi-headed attention mechanism and feed-forward network of the Transformer architecture stays in native C++/Metal land, Coni achieves inference performance that feels like a native desktop application.

Seeing is Believing: Running the Poem Generator

Enough theory. Let’s look at the actual output.

I fired up the terminal and fed the generator the theme: “artificial intelligence dreaming of electric sheep”. Here is the raw, unedited output straight from the Coni interpreter:

[NN] Unified Neural Runtime detected active backend: mlx
===========================================================
               ⬡ Coni Native Poem Generator              
===========================================================
[Metal GPU] Loading native GGUF from disk: /Users/nico/cool/coni-lang/models/qwen2.5-0.5b-instruct-q8_0.gguf
[Config] Detected GGUF architecture: qwen2  (Family: qwen2)

Enter a theme for your poem: artificial intelligence dreaming of electric sheep

[Composing...]

[Fast Gen] Pre-resolving weights for 24 layers...
[Fast Gen] Weight cache ready.

In minds of AI, dreams are vast and bright,
Electric sheep, they dance and sing so bright.
AI dreams of a world where machines are free,
Electric sheep, they are the future of the sky.

In their dreams, they see a world where they can fly,
Electric sheep, they are the future of the sky.
AI dreams of a world where machines are free,
Electric sheep, they are the future of the sky.

[Generation complete. Hit EOS. Speed: 76.10 t/s. Total tokens: 134]

[PERF] Generation took 2525 ms
[PERF] Estimated Throughput: 70.89 tokens/sec

70.89 tokens per second.

That is absolutely blazing fast.

The Code

If you’re curious about how this looks in practice, here is a snippet from our poem-generator.coni script. Notice how clean the high-level functional syntax is, completely abstracting away the sheer violence of the low-level GPU compute happening underneath:

(let [map-obj (nn/load-gguf model-path)]
  (let [res (llm/generate-fast prompt map-obj 250 tk-path config nil 0 nil)
        steps (second res)
        duration (- end-time start-time)
        tps (/ (float steps) (/ duration 1000.0))]
        
    (println "\n[PERF] Generation took" duration "ms")
    (println "[PERF] Estimated Throughput:" tps "tokens/sec")))

By keeping our architecture clean and delegating the heavy math to the CGO MLX bridge, Coni remains an elegant, lightweight functional language while unlocking true machine learning acceleration.

Stay tuned. This is just the beginning of Coni’s native AI capabilities!