Command Squad
A pixel-art tactics game where you never click a soldier. You type or say an order, and a small language model on your own device turns it into a program that your squad runs. It is also my testbed for low-latency, on-device inference.
The idea
In most tactics games you click units. In Command Squad you lead three to five soldiers only with words, typed or spoken: "Two of you flank left, the rest hold the bridge until I say go." While you type or talk, the battle slows to 25% speed, but it does not stop. So a slow parser costs you real time in the game, and latency stops being a number on a dashboard and becomes gameplay.
The game has 10 missions in one countryside biome, three soldier roles (rifleman, heavy, medic), snipers and machine-gun nests, and three stars per mission.
How an order becomes an action
Every order compiles to a short program in a small command language, and a deterministic simulation written in Rust runs it. The order above becomes:
let a = pick(squad, 2)
move(a, to=left_flank)
hold(squad - a, at=bridge, until=signal("go"))Parsers try each order in turn, cheapest first:
- A rule parser. Hand-written, in Rust compiled to WebAssembly. It answers in well under a millisecond and handles most everyday orders.
- An on-device model. Qwen2.5-0.5B fine-tuned on order-and-program pairs, quantized to 4 bits (about 400 MB). It runs in the browser with WebGPU, and a grammar constrains its output, so it can only write valid programs.
- A validator. Every program is type-checked against the live game state before the squad moves. When no parser is sure, the game asks a question or offers its best guess as "Did you mean…?".
Measuring it
The evaluation suite came before the models: 500 everyday orders, 300 orders that combine known parts in new ways, 150 adversarial orders where the right answer is often to ask, and 71 orders from an AI playtest. The main metric is execution match: the predicted program and the reference program run in the headless simulation, and they must leave the squad in the same state. Two programs that are written differently but do the same thing both count as correct.
| Parser | Everyday orders | New combinations | Latency (p50) |
|---|---|---|---|
| Rule parser (hand-written, WebAssembly) | 65.7% | 52.3% | 0.1 ms |
| Qwen2.5-0.5B, few-shot, no training | 3.5% | 0.0% | 258 ms |
| Qwen2.5-0.5B, fine-tuned (distilled), 4-bit | 87.4% | 89.7% | 142 ms |
| Claude Haiku 5.5, the teacher | 86.4% | 78.3% | ~5 s* |
| Cascade: rules → 0.5B → Haiku | 92.3% | 89.3% | 88 ms |
Execution match on held-out test orders. Latency measured on an Apple M2 Pro (llama.cpp with Metal for the small models). *The Claude latency includes about 3.5 s of command-line start-up.
Few-shot prompting alone did not work for a 0.5B model. After fine-tuning on about 20,000 order-and-program pairs, where Claude Haiku wrote 35% of the orders and a filter kept only pairs whose program really did what the order said, the small model matched its teacher on everyday orders and beat it on new combinations. The cascade sends only about one everyday order in eight to the cloud, at about $0.05 per 1,000 orders.
Two honest caveats. Claude drafted most of the evaluation orders, so the Claude numbers may look better than they should. And the public build does not ship the distilled model: Anthropic's usage policy asks for permission before training a model on Claude outputs, so players get a model trained only on template-generated orders, which scores 82.7% on the same test.
On your device
The public build runs entirely on the player's machine: the rule parser and the simulation as WebAssembly, the 0.5B model through WebGPU (usually under a second per order on an M2 Pro), and push-to-talk speech recognition with whisper.cpp in WebAssembly (the text is ready about half a second after you let go of the key). The browser downloads the model only when you click to allow it. The Mac app has the model inside and works with no internet connection.
How it was built
Command Squad started as a 12-month learning plan: evaluations first, then distillation, on-device inference, GPU serving and, at the end, my own inference engine in Rust.
The stack: Rust for the simulation, the command language and the rule parser (compiled to WebAssembly, with Python bindings for the evals); TypeScript and Canvas for the game; Python for the evaluation harness and the data pipeline; MLX for LoRA fine-tuning; llama.cpp and wllama for inference; whisper.cpp for speech; Tauri for the Mac app.
Next on my list are the parts I want to own end to end: the evaluation harness, the fine-tuning setup, and a Rust inference engine for this model, so that the game's own parser runs on code I wrote.
