Jacob Kunselman
Updated 2026-10-23

The AI models I run at home

Open models on one laptop, kept ready for when the cloud is down.

Most of my AI work runs on frontier models in the cloud. These are the backup that keeps me working when those models or the internet are down, and they never leave the house. They are not a replacement for the frontier models.

The setup

One gaming laptop with a 16 GB RTX 5080 Laptop GPU. Everything runs through llama.cpp, one model at a time, and a model loads when something asks for it.

Reading the numbers. Size is the model file on disk. Speed is how fast it writes an answer, in tokens per second (a token is about three-quarters of a word), measured on this laptop. Context is how much text it can keep in mind at once; 64k tokens is roughly a 150-page book. Every language model here is an open-weight release from Hugging Face, squeezed (“quantized”) to fit a 16 GB card.

What each one is for

Qwen 3.5 9B Heretic v2 everyday

Quick answers, and reading photos of part tags, receipts and checklists. 8-bit, with vision.

9.5 GB + 0.6 GB vision · ~66 tok/s · 128k

trohrbaugh/Qwen3.5-9B-heretic-v2

Qwen 3.8 27B GSQ-RCO IQ2_S agent · new

The agent model: coding jobs on my own repositories, web research with sources, browsing, and answering from my notes. Stock Qwen 3.8 27B at 2 bits with a smarter bit allocation per layer, plus multi-token prediction. It took over the job on 2026-10-10.

9.6 GB · ~63 tok/s · 64k

ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Qwen 3.8 27B GSQ-RCO IQ3_XXS CAD · new

Designs printable parts in OpenSCAD and checks them with the CAD tools: it builds, looks at four views, measures, and runs a print check. The same stock model as the agent at 3 bits, with vision and multi-token prediction, thinking at most 8,192 tokens a turn. It took the job on 2026-10-14 after beating its abliterated twin at the same size. How it does.

10.4 GB + 0.9 GB vision · 64k

ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Qwen 3.8 27B RVN Heretic GSQ-RCO hard questions

Harder questions, uncensored, with room for long documents. 3-bit, with multi-token prediction.

10.6 GB · 54–72 tok/s · 64k

0bserverx/Qwen3.8-27B-Heretic-GSQ-RCO-GGUF

Whisper large-v3-turbo speech to text

Voice memos from my phone. Run with whisper.cpp.

1.6 GB · a short clip in under 3 s

openai/whisper-large-v3-turbo

Kokoro 82M text to speech

Reads an answer aloud. It runs on the processor, not the GPU.

0.3 GB · 8 s of speech in about 1.3 s

hexgrad/Kokoro-82M

Image and video models pictures

Pictures with FLUX.1-dev (8-bit) and Z-Image Turbo, short videos with Wan 2.1 VACE 1.3B, all run in ComfyUI.

~11–20 GB per set, text encoder included · an 81-frame 832×480 clip in ~8 min

the original releases

Retired on 2026-10-16: nine models that nothing used any more. They were the Huihui 35B-A3B, both Gemma 4s, the previous 3-bit agent model and a larger copy of it, an older file of the RVN Heretic model, and the leftover Saluki and Bonsai test files from the tests below. About 96 GB.

Tried on
2026-10-10

Each candidate got the same tests as the agent model it would replace: speed and memory at growing context, a fact hidden deep in a long document, the real coding test three times, and six tool-use tasks.

Qwen 3.8 27B GSQ-RCO IQ2_S chosen

Passed the coding test 3 of 3, with the fastest runs of any model, and 6 of 6 tool tasks. Fits whole on the card at 64k with about 2 GB to spare. Now the agent model.

9.6 GB · ~63 tok/s · 64k

Underdog Saluki 27B 1.0 runner-up

A Qwen 3.8 27B fine-tune at 2 bits, also tried with a multi-token prediction head grafted on. Passed 3 of 3 and 6 of 6, but slower, and the only one that fits 96k.

7.9 GB (8.3 GB with MTP) · 42 tok/s, 56–66 with MTP · ~96k

ConwayResearch/Underdog-Saluki-27B-1.0 · Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF

Ternary Bonsai 2 27B not used

A ternary compression of Qwen 3.8 27B, about 1.75 bits per weight, that needs its maker's own llama.cpp fork. The fastest here, but it passed the coding test only 1 of 3: one run spent its whole budget thinking, another ran out of time.

7.0 GB · ~82 tok/s · 64k tested

sudoingx/Ternary-Bonsai-2-27B-PTQ1_0-MTP-GGUF, from PrismML's Ternary Bonsai 2

Tried on
2026-10-23

NVIDIA's new Nemotron 3.5 Bolt E4B against the agent model, on 66 questions graded by code and six real code edits graded by hidden tests. Bolt answers about four times faster; the 27B was right 91% of the time to its 74%, and solved 10 of 12 edits to its 4, in about the same time. The 27B stays the agent model. The method and the results.

Designing parts
2026-10-13

The same laptop now designs printable parts offline: a model writes OpenSCAD, builds it, looks at four views of the result and fixes what it sees. Five of these models got six parts to make, from an L-bracket to a 1/2-inch NPT coupling, and every part was graded by measuring it. The prompts and how each model did.

2026-10-14: the stock 27B, with tools to measure its own parts (calipers, an axis profile, numbered cross-sections and a print check), went from 18 to 27 of 28 harder parts fully right. The harness and the results, round by round.

What I learned

  • Never make a model fit by force. llama.cpp checks whether a model and its context fit in the graphics memory that is actually free; I have it leave about 1 GB for Windows. In October a test loaded a model past that check, and Windows crashed outright instead of the model failing to load. Under WSL, graphics memory that runs over spills through the virtualisation layer and takes Windows down with it. Now every model loads through the check: if it doesn't fit, part of it runs on the processor and it gets slower, but it never crashes the machine.
  • Windows keeps a slice of the card you can't see. Linux's view of the GPU showed 16.3 GB; the memory actually free was 14.6–15.0 GB.
  • Multi-token prediction is free speed. The model guesses the next couple of words and checks them in one step. It doesn't change the answers, and it makes the models here 20–55% faster.
  • 2 bits can be enough. A well-made 2-bit 27B model passed the same real coding test as the 3-bit one, with room to spare on the card.
  • Turn thinking off to look things up. With the model's thinking on, a question about my own notes ran four minutes without an answer before I stopped it. With it off, the same model answered five test questions in 4–40 seconds each. Finding and quoting a line isn't reasoning.
  • Test on real work. My yardstick is one real change to one of my own tools, graded by a separate checker with 14 checks, plus six small tool-use tasks. Benchmarks on a model card don't tell me whether it can do my work.