A 7-billion-parameter coding model, quantized to 4-bit, weighs about 4.5 GB on disk. It loads on a laptop with 16 GB of RAM and answers in real time with nothing crossing the network. For most of the last two years, the catch was obvious: it wasn't very good. You ran a small model locally when you cared more about privacy or offline access than about the answer being right, and you paid for that choice on every hard prompt.
That trade is inverting, and faster than the "just call the frontier API" reflex has caught up with. The distance between a hosted model and one you can run on your own hardware is closing on precisely the tasks that dominate real work — structured extraction, tool calls, code edits, repetitive multi-step reasoning — and a training result published this autumn shows how much capability we were leaving unclaimed at the small end.
"Small" finally has a working definition
The useful definition isn't a round parameter number, it's a hardware test. NVIDIA's 2025 position paper on small models frames it as a deployment property rather than a benchmark tier:
A SLM is a language model that can fit onto a common consumer electronic device and perform inference with latency sufficiently low to be practical when serving the agentic requests of one user.
By that measure, "as of 2025, we would be comfortable with considering most models below 10bn parameters in size to be SLMs." That threshold is exactly the line that matters for a local setup. A sub-10B model in 4-bit fits in the RAM you already have; a 70B model does not, and a 175B one was never going to. So the same definition that describes a small model also describes the machine on your desk.
The economics follow from the size. The same paper puts serving a 7B model at 10–30x cheaper than a 70–175B model in latency, energy, and FLOPs. Run it locally and the marginal cost per token isn't lower — it's zero. No metering, no rate limits, no egress.
The capability objection is the one that's dying
The honest counterargument has always been quality: small models can't reason, so they're a toy. That is the claim a recent paper ("Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning," arXiv 2510.25992, accepted at ICLR 2026) directly attacks — and it does so on a 7B model, the kind that runs locally.
The setup names the two failure modes small models hit. Reinforcement learning with verifiable rewards (RLVR) stalls when the model almost never samples a correct solution, so there's nothing to reward. Plain supervised fine-tuning overfits long demonstrations by imitating them token by token. Supervised Reinforcement Learning (SRL) splits the difference: it reformulates a problem as a sequence of discrete actions, makes the model write an internal reasoning monologue before each one, and rewards it on how closely each action matches an expert's — step by step, rather than pass/fail on the final answer. That gives a useful gradient even when every rollout is wrong.
The results are the part worth internalizing. Training Qwen2.5-Coder-7B-Instruct on 5,000 verified expert trajectories — decomposed into 134,000 step-level items — SRL produced up to a 3.7-point gain on math-reasoning benchmarks and a 14.8% resolve rate on SWE-bench oracle file editing, about a 74% relative jump over the supervised baseline. The framing the authors use is the one that matters here: SRL lets small models learn problems that were "previously unlearnable" by either SFT or RLVR alone. The ceiling on a 7B model was a training-method problem, not a fundamental one.
What a serious local setup actually looks like
Start it and you'll see how little ceremony is involved:
# pull and run a 7B coder locally; ~4.5 GB at 4-bit quantization
ollama run qwen2.5-coder:7b
# or expose it on localhost for your tools to call
ollama serve
curl http://localhost:11434/api/generate \
-d '{"model":"qwen2.5-coder:7b","prompt":"Refactor this function..."}'
The mistake is treating this as all-or-nothing. The stronger pattern — the one the NVIDIA paper spells out as an LLM-to-SLM conversion path — is heterogeneous. Log what your system actually asks a model to do, cluster the calls, and you'll find most of them are narrow and repetitive: format this, extract that, pick a tool, apply an edit. Those are the calls you move to a local SLM, optionally fine-tuned with LoRA in a few GPU-hours. You keep a large model on call only for the genuinely open-ended "decide and plan" moments. Most requests never leave the machine; the few that need a bigger brain still can.
Where a big model still earns its keep
This isn't a claim that small wins everything. Broad world knowledge, long-context synthesis across many documents, and the rare hard planning step are still where a frontier model pulls ahead, and pretending otherwise gets you a worse product. The point is narrower and stronger: for the high-volume, well-scoped calls that make up the bulk of a real workload, a local SLM is now good enough that routing them to a paid API is a habit, not a requirement. Quantization noticeably degrades some models below 4-bit, and a 7B model will confabulate on facts it never saw — so measure your own task, don't assume the benchmark transfers.
The takeaway
Pick one repetitive model call in something you run — a classifier, an extractor, a code-edit step — and move it to a local 7B model this week. Measure resolve rate or accuracy against whatever API you're paying for now. If it holds, you've cut a recurring cost to zero, removed a network dependency, and kept the data on your own disk. The default is quietly flipping: the local model is becoming the first thing you reach for, and the hosted call the exception you justify.
Sources: Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning (arXiv 2510.25992) · Small Language Models are the Future of Agentic AI (arXiv 2506.02153)