Benchmarking local models

The benchmark measures how well a free local model does the work Action Stations asks of it, on your own computer, and produces a result file you can submit so the published table (docs/benchmarks/) and the per-memory recommendations grow. It costs nothing: it runs only against LM Studio or Ollama on this machine, with no cloud model and no cloud judge.

It drives the built app headless through the same IPC the screens use, in a throwaway settings folder, so your own settings, projects and API keys are never read or changed.

What it runs

Three tasks, on the fictional example projects that ship with the app (examples/):

Task	What the model does	How it is scored
Ideation	Proposes six scenario ideas from an example’s plans	Valid ideas, no near-duplicates, cited sections that resolve, spread of threat families, the source check on why each idea teaches
Drafting	Writes a scenario from a fixed brief: overview, roles, phases, every MSEL phase, the documents and the review pass (no injects or media)	Stages that succeed, the app’s own lint checks, citations (resolved, coverage, and rows the source check labels grounded), a structure rubric drawn from the Scenario 3 reference MSEL
Critique	Reviews a hand-corrected example scenario with planted defects (an out-of-order row, a duplicate, a missing decision point, an unknown facility identifier, an implausibly fast restoration…)	Share of planted defects a note names or the revision removed, minus notes on rows with nothing wrong, minus damage to correct rows the revision rewrote or dropped

Critique is advisory: it is reported next to the floor but does not decide it (see below). Every score is 0 to 100 and computed without a model. The overall score weights drafting 0.5, critique 0.25 and ideation 0.25. The cases, weights and thresholds are in bench/suites/quick.json and bench/suites/standard.json; the planted defects are in bench/defects/.

Two suites:

• quick: one case per task on the City of Millbrook water corpus, plus a second critique case. A check that a model works at all; under an hour for a 9B model on a 32 GB Mac.
• standard: three corpora, five briefs and six critique cases, three repeats with fixed seeds (11, 23, 37). This is the suite the published table uses; plan for an evening.

Before you start

1. Build the app: pnpm install then pnpm build:app.
2. Start LM Studio or Ollama and have the model you want to test downloaded.
3. On Apple Silicon, test the MLX build of a model (safetensors in lms ls --json), not the GGUF build, unless its profile says otherwise (preferredBuild). The benchmark applies the model-card settings in Model profiles and records the profile in the run file (model.inference). BENCH_PROFILE=off runs without them, BENCH_EFFORT=low|medium|high|off sets the reasoning level and BENCH_DRAFT=<model id> names a draft model for speculative decoding.
4. Give the model a 32K context. The suites run at 32,768 tokens, which the MSEL stage needs.
   • Ollama: nothing to do; the benchmark sets num_ctx for each request.
   • LM Studio: load the model with lms load <model id> --context-length 32768 -y. The benchmark warns when the loaded context is smaller, and long stages then fail and count against the model.

Run it

pnpm bench -- --models ollama:qwen3.5:9b --suite quick
pnpm bench -- --models lmstudio:qwen/qwen3.5-9b,ollama:qwen3:30b --suite standard

Option	Meaning
--models <ref,…>	ollama:<name from ollama list> or lmstudio:<model id from lms ls>
--suite quick|standard	Default quick
--repeats N	1 to 3; default from the suite (quick 1, standard 3)
--out <dir>	Where run files and the generated projects go; default .tmp/bench
--base-url <url>	A non-default port of LM Studio or Ollama, still on 127.0.0.1
--embedder local|hash	local (default) searches the plans with the bge-small-en-v1.5 model the app uses, downloaded once (about 130 MB) into .cache/bench/models after you confirm; hash runs fully offline with weaker search and is marked in the result
--concurrency N	Cases run in parallel, 1 to 8, default 1 (serial). On Apple silicon with LM Studio, parallel cases did not finish sooner (see the spec), so use more only on a server that batches requests. Each worker is its own app instance, so keep N at or below the parallel slots the model server offers (LM Studio: 4 by default). LM Studio may share a model’s loaded context between its parallel slots, so load it with --context-length <32768 x N> --parallel <N>; the benchmark warns when it is not. Wall times per case are longer under load
--triage	The standard suite at one repeat, with a pass or miss against the floor at the end. A quick screen before the full run; triage runs are not published and the report ignores them
--yes	Confirm the one-time embedder download without a prompt
--resume	Continue after an interruption from <out>/state.json
--clean	Delete the generated projects in <out>/work/ after the run

| --judge lmstudio:<model> | Optional local judge: scores how plausible up to 30 cited MSEL rows are, quoting the row and passage (a score with a quote that is not in the text is dropped). Local models only, off by default, never part of the overall score. The scores are marked publishable only when the judge agrees with the reference rows in bench/calibration/rows.json (scored by a hosted reference model, accepted by the maintainer; see its README) (weighted kappa 0.6 or more, 60 rows); without that file they are kept but marked unpublished. Measured once (2026-11-03, old sampling settings) with qwen/qwen3-30b-a3b-2507 as judge on the 60 rows: kappa 0.039, which does not clear 0.6, so no judge scores are published yet | | --determinism | Runs two repeats at the same seed and reports whether their scores and lint counts match (replaces --repeats) |

With more than one worker, the model’s speed (output tokens per second) is measured in a separate pass: one ideation request on its own after the cases, so other cases sharing the model do not slow it. The per-task wall times in the run file are measured under load.

After repeat 2 of a three-repeat run, a model whose two repeats both miss the floor on the same criterion (a task under 50, or drafting reliability under 90%) is stopped, since the median of three can no longer reach it. The run file keeps the two repeats and records the reason in stoppedEarly, and the terminal summary shows it. Use --triage to screen models cheaply, and the standard suite with three repeats to publish the rest.

Judge scores and the determinism result appear in the run file (judge, determinism) and the terminal summary.

The benchmark refuses any endpoint that is not on this machine; there is no option to allow one. It also refuses to run when prompt overrides are present, so every run uses the shipped prompts. After the first case it prints an estimate of the time left.

When a model finishes, the terminal shows a short table and the path of its run file, <date>-<model>-<machine>.json. The generated projects stay in <out>/work/ so you can open them in the app and see what the model wrote.

Read the result

• Median per task and overall: with repeats, the middle of the repeats.
• Range: the lowest and highest overall score over the repeats. A range wider than 15 points is marked unstable.
• Drafting reliability: the share of pipeline stages that succeeded. A stage that fails after its repair round is recorded in failures with the error, and the case goes on as far as the app allows.
• details in each task holds per-case scores, lint counts per code, the structure items that passed and the planted defects found or missed.

Submit a run

Run files hold scores, model metadata (name, quantization, size, digest) and coarse hardware (platform, chip or GPU name, RAM and VRAM rounded to GB). They never hold a host name, user name or file path, and the report refuses a file that does.

1. Run the standard suite with the local embedder.
2. Copy the run file to docs/benchmarks/runs/.
3. Run pnpm bench:report. It validates every run file, merges runs of the same model and writes docs/benchmarks/model-benchmarks.json and the tables in docs/benchmarks/README.md.
4. Open a pull request with the run file and the regenerated report. The maintainer reruns any run that would change a recommendation before merging it.

Runs from another suite version or scoring version are kept but listed as archived, never ranked with the current ones. Changing a case, weight or threshold in a suite file bumps its suiteVersion.

Recommendations per memory size

For each memory tier of the setup screen (under 10 GB, 10–24 GB, 24–72 GB, 72 GB and up), the report recommends the model with the highest median overall score that fits the smallest machine in the tier and passes the floor: drafting and ideation at least 50, drafting reliability at least 90%, and stable. Critique is not part of the floor. The report shows its recall, the share the revision fixed and the share of correct rows it damaged, and warns when recall is under 50 or damage is over 10%: such a model can still draft well, but review the critique notes and the revised rows before trusting them. “Fits” is the size on disk plus an estimate of the context cache (a tenth of the weights at 32K); the lowest tier assumes 5.5 GB. A tier with no such model says so. The setup recommendations in the app are updated by hand from this report.

Tabletop Benchmark (public view)

The same runs feed Tabletop Benchmark, the public benchmark at https://github.com/actionstations-ai/tabletop-bench. pnpm bench:export prints the hash of each suite’s task set and, with --out <dir>, writes the task-set files; every run file records that hash as taskSetHash. To submit a run there, use that repository’s tb run and tb add; it checks the hash against the checkout you run from.

Tabletop Mini Benchmark (about ten minutes per model)

The Mini answers “will this model work for tabletop generation?” with four model calls on fixed inputs from the bundled examples, each run twice, and one number with a stated meaning:

Mini score = Quality x Completion

• Completion is the chance that a full scenario drafts with no failed stage. It is the share of Mini calls that did not fail (a call fails only after the app’s normal repair policy, or if it stalls or loops), raised to the 14 model calls a scenario makes. It is shown with the observed count, such as “0 failures in 10 calls”. The repair rate is reported apart and is not a failure.
• Quality (0 to 100) is the share of fixed criteria the replies meet, against the Scenario 3 reference: a full 18-row MSEL phase on a long prompt (row count, pace, purpose-tag mix, citations resolved and supported, structure, lint), the handbook’s completeness, and each planted defect the review finds.

Shown next to the score and not part of it: review damage (the share of correct rows the review rewrote) and draft time (minutes for a full scenario on this hardware, from the model’s measured speed and the token budget of a scenario). A slow draft time is reported, never a failure.

Band	Requirement
Recommended	completion >= 90%, quality >= 80, damage <= 5%
Usable with editing	completion >= 75% and quality >= 60
Not recommended	anything else

Completion falls fast: one failure in eight calls leaves about 15%. About ten calls detect frequent failures, not rare ones.

Probe	Call
Format	scenario overview (city-water)
Long prompt	the last MSEL phase of chemical-rmp, stretched to 18 rows
Long output	controller handbook (city-water)
Review	critique of the county-911 scenario with planted defects

There is no time limit on a call. A call ends only when the model stops producing tokens (60 seconds after it started, or 10 minutes while the prompt is read) or repeats the same text; the run file keeps the raw end of the reply. Every figure shows its range over the two runs, and models whose ranges overlap are a tie.

pnpm build:app
pnpm bench:mini -- --models lmstudio:qwen/qwen3.5-9b,lmstudio:google/gemma-3-12b

A list of models runs one after the other. For LM Studio models the runner loads each one at 32,768 tokens (lms load), runs the Mini and unloads it; pass --no-load if you load models yourself and --stop-server to stop the LM Studio server at the end. Ollama models are used as loaded. pnpm bench -- --suite mini … is the same command. One run file per model is written to --out (default .tmp/bench-mini). --held-out adds the maintainers’ held-out review case.

pnpm bench:export -- --suite mini prints the Mini task set’s hash; run files record it as taskSetHash. pnpm bench:mini:replay <mini-run-dir> scores saved Mini runs again with no model call and names the calls it cannot recompute.

Re-scoring saved runs

pnpm bench:rescore <run-dir> [<run-dir> ...] scores the critique cases of saved runs again from their work/ folders (the scenario before the critique, after its revision, and the stored notes), with no model call, and prints the old and new critique, overall and floor result. Use it after a scoring change to see what moved. bench/suites/held-out.json holds critique cases on examples the standard suite does not use; they are not in the exported task set.