compare_reviews.clj — A tiny, configurable LLM-powered reviewer in Clojure

2025-08-27 1372 words 7 minutes

Contents

Overview

This tool assembles Markdown from patterns, calls an LLM with a structured prompt, and writes a report as Markdown (and optionally PDF). It’s driven by EDN config, meaning you can change behavior by editing data, not code.

Why it’s cool:

Configuration-as-data (EDN): composable, Git-friendly, reproducible.
Clear pipeline: collect → call LLM → write → optional PDF and summary.
Smart outputs: timestamped filenames avoid collisions with minimal friction.
Extensible “agent” model: the LLM call is just a map merged into agent/call.
Minimal code, lots of leverage (Pandoc, glob patterns, simple IO).

High-level data flow

flowchart TD
    A[Start] --> B[load-config (EDN path or map)]
    B --> C[mru/aggregate-md-from-patterns]
    C --> D[agent/call (merge model + {:system :pre :prompt})]
    D --> E[resolve-output-file]
    E --> F[Write main .md]
    F --> G{pdf?}
    G -- Yes --> H[Pandoc md->pdf]
    G -- No --> I[Skip]
    F --> J{summary?}
    J -- Yes --> K[agent/call for summary]
    K --> L[Write _summary.md]
    J -- No --> I
    H --> I[Done]
    L --> I

Code walkthrough (what each function does)

(ns margin-mania.reporting.compare-reviews
  (:require [clojure.edn :as edn]
            [pyjama.core :as agent]
            [margin-mania.reporting.utils :as mru]
            [pyjama.tools.pandoc]
            [clojure.java.io :as io])
  (:import (java.io File PushbackReader)
           (java.time LocalDateTime)
           (java.time.format DateTimeFormatter)))

pyjama.core/agent: abstraction over the LLM call (agent/call).
mru/aggregate-md-from-patterns: globs files and concatenates Markdown (plus optional metadata).
pyjama.tools.pandoc: converts Markdown to PDF.

load-config

(defn load-config [cfg]
  (cond
    (string? cfg)
    (with-open [r (io/reader cfg)]
      (edn/read (PushbackReader. r)))

    (map? cfg) cfg

    :else (throw (ex-info "Unsupported config type" {:given cfg}))))

Accepts either a path to EDN or a pre-built map.
Encourages configuration-as-data and REPL ergonomics.

Timestamp and output resolution

(defn ^:private timestamp []
  (.format (LocalDateTime/now)
           (DateTimeFormatter/ofPattern "yyyy-MM-dd_HH-mm-ss")))

(defn resolve-output-file
  "Return the actual File to write to.
   If out-file is a directory or has no extension, use <dir>/<yyyy-MM-dd_HH-mm-ss>.md."
  [out-file]
  (let [f (io/file out-file)
        as-dir? (or (.isDirectory f)
                    (not (re-find #"\.[^/\\]+$" (.getName f))))]
    (if as-dir?
      (io/file f (str (timestamp) ".md"))
      f)))

If :out-file is a directory or lacks an extension, it auto-generates a timestamped filename, e.g. 2025-08-27_16-30-12.md.

Summary file helper

(defn ^:private summary-file
  "Given the primary output file, return the summary file: <same path> with `_summary.md`."
  ^File [^File final-file]
  (let [parent (.getParentFile final-file)
        name (.getName final-file)
        base (if (re-find #"\.md$" name)
               (subs name 0 (- (count name) 3))
               name)]
    (io/file parent (str base "_summary.md"))))

Takes the main report path and returns the companion summary filename (e.g., report.md → report_summary.md).

The main engine: process-review

(defn process-review
  "If :summary true, performs a second LLM call over the first call's output and writes
   `<previous out-file>_summary.md`."
  [{:keys [patterns model out-file system pre pdf summary]}]
  (let [combined-md (mru/aggregate-md-from-patterns patterns)
        result-1 (agent/call
                   (merge model
                          {:system system
                           :pre    pre
                           :prompt [combined-md]}))
        final-file (resolve-output-file out-file)
        out-1-str (with-out-str (println result-1))]
    ;; write main result
    (io/make-parents final-file)
    (spit final-file out-1-str)

    ;; optional PDF for main result
    (when pdf
      (pyjama.tools.pandoc/md->pdf
        {:input  final-file
         :output (str final-file ".pdf")}))

    ;; optional summary step
    (when summary
      (let [sum-pre "Generated a short summary, (with title and points just like a ppt slide)  of %s"
            result-2 (agent/call
                       (merge model
                              {:system system
                               :pre    sum-pre
                               :prompt [out-1-str]}))
            sum-file (summary-file final-file)]
        (io/make-parents sum-file)
        (spit sum-file (with-out-str (println result-2)))))

    ;; return the path(s) for convenience
    {:out     (.getPath final-file)
     :summary (when summary (.getPath (summary-file final-file)))
     :pdf     (when pdf (str final-file ".pdf"))}))

Aggregates input Markdown per :patterns, then calls the LLM once to produce the main report.
Writes the main .md, and if :pdf true, renders a PDF via Pandoc.
If :summary true, performs a second LLM call on the first output and writes ..._summary.md.
Returns a map of produced paths for convenience.

Notes:

:pre strings often contain %s. Formatting is handled by the agent layer (pyjama), which composes :system, :pre, and :prompt.
println around the result ensures a trailing newline and converts the result object to text; with-out-str captures it.

Entry point

(defn -main [& _]
  ;(process-review "resources/reporting/edn_config.edn")
  ;(process-review "resources/reporting/bad_review.edn")
  (process-review (load-config "resources/reporting/yourown.edn")))

Swap the EDN file to change the job, no code changes required.

Configuration (EDN) at a glance

Key	Type	Required	Meaning
:patterns	vector of globs or maps	yes	What to aggregate into one Markdown blob for the prompt.
:model	map	yes	LLM backend and options, e.g. `{:impl :chatgpt :model "gpt-5" :temperature 1}`
:out-file	string or File	yes	Output path; directory or no-ext yields `timestamp.md` inside.
:system	string	rec.	System prompt guiding the LLM’s persona/role.
:pre	string	optional	Prompt preface/template (often contains `%s`).
:pdf	boolean	optional	If true, also render a PDF via Pandoc.
:summary	boolean	optional	If true, run a second LLM pass to produce `_summary.md`.
:query	string	no	Present in some configs but not used here (potential future use).

Example: a “bad review” config

{:patterns ["docs/review/small/*.md"]
 :model    {:impl :chatgpt :temperature 1 :model "gpt-5"}
 :out-file "docs/review/review-comparison-worst-2.md"
 :query    "highlight the bad review"
 :system   "You are extremely good at making text comparison and selecting the best option."
 :pre      "Given the text below, %s, give its name and why. Use a table if needed. \n %s"}

Example: “your own” config with richer patterns and summary

{:patterns [{:pattern "src/margin_mania/reporting/compare_reviews.clj" :metadata "The main clojure code"}
            {:pattern "resources/reporting/*.edn" :metadata "EDN based config file to use with the main clojure code."}
            {:pattern "docs/gen/yourown/2025-08-27_16-30-12.md" :metadata "Generated markdown output file for the yourown.edn input config."}]
 :model    {:impl :chatgpt :temperature 1 :model "gpt-5"}
 :out-file "docs/gen/yourown"
 :summary  true
 :system   "You are a technical R&D researcher for a consulting company. Output is markdown, formatted for Hugo, with headers and tags."
 :pre      "Given the text below, explain the Clojure code and why it's so cool. Use tables if needed. Add code sample to make it easy to understand. Include mermaid diagram. Show input/output of the program. \n %s"}

How output filenames are decided

Provided :out-file	Is directory or no extension?	Actual main output example
“docs/gen/yourown”	Yes	docs/gen/yourown/2025-08-27_16-30-12.md
“docs/gen/”	Yes (dir)	docs/gen/2025-08-27_16-30-12.md
“docs/review/out.md”	No	docs/review/out.md

If :summary true, the summary file is the same base with _summary.md appended, e.g., out.md → out_summary.md.

Returned paths from process-review:

Key	Value (if present)
:out	path to the main .md
:summary	path to the `_summary.md`
:pdf	path to the `.pdf` (if `:pdf`)

Minimal end-to-end example (input and output)

Assume two tiny Markdown inputs:

docs/review/small/a.md

# Service A
- Latency: 120ms
- Errors: 0.3%

docs/review/small/b.md

# Service B
- Latency: 90ms
- Errors: 1.2%

EDN config (input):

resources/reporting/example.edn

{:patterns ["docs/review/small/*.md"]
 :model    {:impl :chatgpt :temperature 0.2 :model "gpt-5"}
 :out-file "docs/gen/example"
 :system   "You compare services objectively and concisely."
 :pre      "Compare the following notes and pick the best option. Use a short table then a verdict.\n%s"
 :summary  true
 :pdf      false}

Run from REPL:

(require '[margin-mania.reporting.compare-reviews :as cr])

(def paths
  (cr/process-review
    (cr/load-config "resources/reporting/example.edn")))

;; Example returned value:
;; {:out "docs/gen/example/2025-08-27_16-35-00.md"
;;  :summary "docs/gen/example/2025-08-27_16-35-00_summary.md"}

Example output (main .md, truncated):

docs/gen/example/2025-08-27_16-35-00.md

| Service | Latency | Errors |
|---------|---------|--------|
| A       | 120ms   | 0.3%   |
| B       | 90ms    | 1.2%   |

Verdict:
- If latency is critical and error budget can tolerate ~1.2%: choose Service B.
- If reliability is paramount: choose Service A.
Overall: Service A is safer; Service B is faster but error-prone.

Example summary (second LLM pass):

docs/gen/example/2025-08-27_16-35-00_summary.md

Title: Service Comparison — TL;DR

- A is more reliable (0.3% errors).
- B is faster (90ms).
- Pick A for stability; B for speed when errors are acceptable.

Files produced:

docs/gen/example/2025-08-27_16-35-00.md
docs/gen/example/2025-08-27_16-35-00_summary.md

If :pdf true, you’d also get:

docs/gen/example/2025-08-27_16-35-00.md.pdf

A tiny “use it now” code sample

From a REPL (no EDN file at all):

(require '[margin-mania.reporting.compare-reviews :as cr])

(cr/process-review
  {:patterns ["docs/review/small/*.md"]
   :model    {:impl :chatgpt :temperature 0.5 :model "gpt-5"}
   :out-file "docs/gen"               ; directory-ish → timestamped .md
   :system   "You are a helpful reviewer."
   :pre      "Summarize and recommend.\n%s"
   :summary  true
   :pdf      false})

Why this design works so well

Declarative flexibility: Switching tasks is as simple as swapping EDN files.
Safe defaults: Smart filename resolution prevents accidental overwrites.
Composability: :patterns, :system, :pre, and :model are plain data—easy to test and iterate.
Extensible pipeline: Adding steps (like the summary pass or PDF rendering) is localized and clear.
Production-friendly: io/make-parents, explicit return values, and Pandoc integration make it practical.

Tips and gotchas

Install Pandoc if using :pdf true.
:query appears in some configs but is not used by process-review here; it might be used upstream (mru) or be reserved for future logic.
Large inputs: consider chunking or filtering in mru/aggregate-md-from-patterns to keep prompts within model limits.
%s placeholders in :pre: the formatting/templating is handled by the agent layer (pyjama), so keep consistent with how your agent expects :pre and :prompt.

Happy reviewing!