Building Native Parallel Downloads in Coni

One of the great things about building your own language and tooling is the ability to rethink how operations are performed. In our latest commits, we decided to tackle a major bottleneck in our build process: downloading Maven dependencies.

The Problem with Platform-Specific Scripts

Previously, the download-url-to-file function in Coni relied on shelling out to platform-specific tools:

• On Linux/macOS, it spawned a curl process.
• On Windows, it invoked a massive powershell command using System.Net.WebClient.

While this worked, it had several drawbacks. First, shelling out to external processes is slow and resource-intensive. Second, the dependency on external tools meant that subtle differences in curl versions or Windows security protocols could cause unexpected failures. Most importantly, downloading artifacts sequentially using these shell commands meant that resolving a large Maven project would take entirely too long.

The Native Solution

In commit 9ac76d12, we introduced sys-http-download, a native builtin implemented directly in the Go evaluator. This new builtin uses Go’s standard net/http client.

Because it’s natively built into the evaluator, we avoid the overhead of process creation entirely. We even added some robust retry logic directly into the Go implementation to handle rate-limiting from Maven Central (which frequently throws 429 Too Many Requests errors if you hit it too fast without a recognizable User-Agent).

Supercharging with Goroutines

With the native builtin ready, the real magic happened when we applied Coni’s concurrency primitives. Coni supports Go-style concurrency natively using go, chan, <! (read from channel), and >! (write to channel).

Instead of downloading artifacts one by one, we set up a worker pool right inside the standard library:

(let [total-count (count missing)
      result-ch (chan total-count)
      task-ch (chan total-count)]
  ;; Push all tasks to the task channel
  (loop [rem missing]
    (if (not (empty? rem))
      (do 
        (>! task-ch (first rem))
        (recur (rest rem)))))
  (close! task-ch)
  
  ;; Spawn 8 worker goroutines to process tasks concurrently
  (loop [i 0]
    (if (< i 8)
      (do
        (go 
          (loop []
            (let [item (<! task-ch)]
              (if (not (nil? item))
                (do
                  (>! result-ch (download-file-single item repos))
                  (recur))))))
        (recur (+ i 1)))))

  ;; Await results and show progress
  ...)

We cap the worker pool at 8 goroutines to prevent overwhelming the network or triggering aggressive rate limits.

The Result

The speedup is massive. What used to be a long, serialized sequence of shell invocations is now a swift, parallelized operation handled entirely inside the Coni runtime.

By dogfooding our own concurrency primitives, we proved that Coni’s channel-based architecture isn’t just a fun toy—it’s robust enough to handle the language’s own core I/O tasks efficiently.