Running GGUF Models Natively with Coni and Apple MLX

Tue, 28 Jul 2026 10:00:00 +0900

Picture this: You see a shiny new Open Source LLM drop on HuggingFace. It promises to be the next big thing in local reasoning. You want to try it out on your MacBook. So you spin up your terminal.

You pip install some packages. You get an error about torch being compiled without Metal support. You switch to conda. You update your environment. You wait 15 minutes for 4 gigabytes of dependencies to download. You run the Python script. You get a ValueError: weight shape mismatch because your transformers library is exactly 0.0.1 versions out of date. You update transformers, which breaks accelerate.

Fast Native LLM Inference in Coni: The Poem Generator

Sat, 25 Jul 2026 11:45:00 +0900

In our ongoing quest to make Coni the most capable, dependency-free, and relentlessly fun functional language out there, we’ve hit a massive milestone. We didn’t just write an AI script—we went fully native.

Today, we’re unveiling our first fully native AI example: The Poem Generator.

Using the highly capable Qwen 2.5 0.5b instruct model formatted as a raw GGUF binary, this new Coni script takes a theme, invokes a local LLM, computes a beautiful poem, and spits out characters at a shockingly fast rate of over 70 tokens per second (TPS).

Mlx on NicoLabs

Running GGUF Models Natively with Coni and Apple MLX

Fast Native LLM Inference in Coni: The Poem Generator