The Local AI Stack: What Runs on Your Own Machine

Overview

Running AI models on your own hardware has moved well past the enthusiast stage. Ollama, the most widely used local model runner, reached 52 million monthly downloads in the first quarter of 2026 — up from 100,000 in the same quarter of 2023, a 520-fold increase in three years.

The hardware caught up alongside it. AI PCs are projected to make up 54.7% of total PC shipments in 2026, having grown from 15.6% in 2024 and 31.0% in 2025, with Microsoft's Copilot+ specification requiring a neural processing unit delivering 40 or more TOPS.

The capability question has a precise answer, and it depends entirely on the task. Small models now land within 3 to 8 percentage points of frontier models on structured extraction, simple code completion and intent classification. On long-document summarisation and multi-step reasoning the gap widens to 15 to 20 points.

That split is the whole practical picture. For a large class of everyday work, a model running on your laptop is genuinely competitive, private and free to run. For the hardest reasoning work, the cloud still wins clearly.

Ollama Went From 100,000 to 52 Million

The adoption curve for local AI tooling is steeper than almost anything else in the sector. Ollama recorded 100,000 monthly downloads in the first quarter of 2023 and 52 million in the first quarter of 2026.

The reason it became the default is friction removal rather than capability. Running a language model locally previously meant understanding model formats, compiling runtime backends, matching quantisation levels to hardware and configuring inference parameters. Ollama reduced that to pulling a model and running it.

This matters more than raw performance for adoption. The models had been openly available for years before usage grew like this; what changed was that using them stopped requiring specialist knowledge.

Where Small Models Genuinely Compete

The gap between small local models and frontier cloud models has narrowed to the point where, for specific work, it is difficult to justify the cloud.

On the tasks that dominate practical automation — JSON extraction, simple code completion and intent classification — small models land within 3 to 8 percentage points of frontier systems. For most production purposes that difference is invisible.

On some narrow benchmarks small models now win outright. Microsoft's Phi-4, at 14 billion parameters, scores 80.4% on the MATH benchmark, ahead of GPT-4o at 74.6% and Claude 3.5 Sonnet at 78.3%. A model small enough to run on a laptop outperforming much larger cloud systems on a mathematics evaluation would have seemed implausible two years ago.

Where They Still Lose Badly

The same comparison run on harder tasks produces a very different answer, and this is the part usually left out of local-AI advocacy.

For long-document summarisation and multi-step reasoning, the gap between small and frontier models jumps to 15 to 20 percentage points. That is not a rounding error — it is the difference between a tool you can trust with a complex task and one you must check.

The pattern is coherent. Tasks with a well-defined output format and limited context are close to solved at small scale. Tasks requiring sustained reasoning across a long context, holding multiple constraints simultaneously, remain strongly dependent on model size. Anyone choosing local over cloud should be choosing per task, not as a blanket policy.

What You Can Actually Run on 8GB

Hardware requirements are the practical constraint, and memory rather than raw compute is what decides whether a model runs at all.

Why 70B Models Aren't Practical at Home

The models that would close the reasoning gap are precisely the ones most people cannot run, which is the central tension in local AI.

A 70-billion-parameter model needs roughly 48 GB of memory even at aggressive quantisation. That means two 24 GB consumer GPUs, a workstation card costing several thousand dollars, or offloading layers to system RAM — which works and slows generation to the point of frustration.

So the practical local ceiling sits well below the point where models become strong at multi-step reasoning. The 15 to 20 point gap is not a temporary artefact of immature software; it reflects the fact that the models capable of closing it do not fit on the hardware most people own.

What Quantization Costs You

Quantisation is what makes local inference possible at all, and it is a trade rather than a free optimisation.

The technique converts model weights from high-precision representations into lower-bit ones, cutting memory footprint and computational demand substantially. Four-bit quantisation reduces a model's memory requirement by roughly three quarters compared with full precision.

Quality degrades with aggressiveness. Moderate quantisation costs little measurable accuracy, and heavier compression produces noticeable deterioration — more errors, weaker instruction-following, reduced coherence over long outputs. Benchmark scores quoted for a model generally reflect full precision, so the version running on your laptop is usually somewhat weaker than the number that persuaded you to download it.

AI PCs Pass Half the Market

The hardware base is expanding quickly. AI PC share of total shipments is projected at 54.7% in 2026, up from 31.0% in 2025 and 15.6% in 2024, with Counterpoint putting AI-advanced PC penetration near 59%.

The commercial side is moving faster than consumer. Gartner forecasts that essentially all commercial PC purchases will be AI PCs by the end of 2026, and Intel expects AI PCs to exceed half the total market in the same period.

Much of this is passive rather than chosen. As NPUs become standard silicon, buyers acquire AI-capable hardware by replacing an old laptop rather than by seeking the capability. The installed base will grow regardless of whether owners ever deliberately run a local model.

The 40 TOPS Line

Microsoft's Copilot+ specification created a concrete hardware threshold: a processor or system-on-chip with an NPU capable of 40 or more TOPS, plus a minimum of 16 GB of RAM.

Features including Recall, Cocreator, Windows Studio Effects, live captions and automatic super resolution are designed around that level of on-device processing.

The 16 GB memory floor is arguably the more consequential half. Neural processing performance determines how fast a model runs; memory determines whether it runs at all. A machine with a fast NPU and 8 GB of RAM is still limited to small models, which is why the RAM requirement — not the TOPS figure — is what actually governs local AI capability.

AI Is Making AI PCs More Expensive

The most awkward development in this market is that AI demand is pricing the hardware out of reach faster than it is making it capable.

IDC expects PC unit shipments to fall 11.3% year over year in 2026, from 284.7 million to 252.53 million, while total market value rises 1.6% on higher prices. Warnings have gone as far as a potential 9% market contraction driven by memory costs.

The cause is AI itself. Data centre demand for memory has driven RAM prices sharply upward, and RAM is precisely what local inference depends on. The result is a market where the share of AI-capable PCs rises while the number of PCs sold falls — and where the 16 GB of memory needed to run a decent local model has become materially more expensive because of the same boom that made anyone want it.

Conclusion

Local AI is bigger than its coverage suggests. Ollama alone went from 100,000 to 52 million monthly downloads in three years, AI PCs are on track to exceed half of all shipments in 2026, and a 14-billion-parameter model now beats far larger cloud systems on at least one mathematics benchmark.

The honest limitation is task-shaped rather than general. Within 3 to 8 points of frontier models on extraction, classification and simple code completion; 15 to 20 points behind on long-document work and multi-step reasoning. The models that would close that gap need around 48 GB of memory, which is well beyond a normal laptop.

The sensible position is not local versus cloud but knowing which is which. Structured, repetitive, privacy-sensitive work runs well on your own machine at no marginal cost. Hard reasoning does not — and the RAM shortage created by AI data centres is making the hardware that would change that steadily more expensive.

Data Source and Attribution

OllamaMicrosoft Copilot+ DocsCounterpoint Research

Local model runner download figures come from published Ollama usage reporting. AI PC market share projections come from IDC and Counterpoint Research, with commercial adoption forecasts from Gartner and Intel. Copilot+ hardware requirements come from Microsoft's published developer documentation. Model performance comparisons, benchmark scores and task-level capability gaps come from published small language model evaluations. VRAM and quantisation requirements reflect documented hardware guidance for local inference.

FactsFigs reviews, cleans, and cross-checks every source dataset before shaping it into a data story. Each visualization is created and designed in FactsFigs Design Studio — an internal tool developed and owned by FactsFigs — and is the original work of a FactsFigs author, not an AI-generated copy of any existing graphic. Individual assets within a visual may or may not be produced with AI tools, but the design of the visual itself is solely FactsFigs' own.

Figures are estimates at the time of publication, provided for information only — nothing here is financial advice or a guarantee of accuracy.

2026-07-20