← sources

I fine-tuned a 2B decision model to play Doom on my Mac in 43 minutes (open source, MLX)

source: reddit (reddit:LocalLLaMA)published 2026-11-09 18:04ingested 2026-11-10 02:42

retained text

You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon).

How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions:  
\- Is a monster right at the crosshair, so that a shot now would hit it? (yes/no)  
\- Where is the nearest monster: left, centre, right, none?  
\- Is a monster dangerously close? (yes/no)

A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision.

The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames:  
\- "a shot would hit": precision 0.79 → 0.92  
\- where the nearest monster is: 92.7% → 96.8%  
\- dangerously close: 93.9% → 98.3%

In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just **pip install mlx-decision**.

Repo: [https://github.com/armbues/mlx-decision](https://github.com/armbues/mlx-decision)

I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next.

document fetched 2026-11-10 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xryvbg/i_finetuned_a_2b_decision_model_to_play_doom_on