If you want to run models on a laptop, the specification that decides what you can do is not the CPU, the core count, or the token throughput on the box. It is memory available to the accelerator, and how much of it the model plus its context actually occupies.
The rough arithmetic
A quantised model needs roughly its parameter count in billions, multiplied by the bytes per weight, plus overhead for context. At four bits per weight, a seven-billion-parameter model lands near four gigabytes; a fourteen-billion model near eight; a thirty-billion model near eighteen. Context, the key-value cache, adds more as the conversation grows.
What each tier gets you
- 8 GB: small models, short context, good for summarising and rewriting.
- 16 GB: the practical sweet spot for daily drafting and code assistance at moderate context.
- 24 GB and above: larger models and long documents, at the cost of heat and battery.
These are starting points, not guarantees. Quantisation quality, the runtime, and the context length all move the boundary.
Why the box numbers mislead
Vendors quote TOPS, an integer throughput figure that says little about the memory bandwidth that actually limits token generation. A laptop can have an impressive accelerator and still crawl because it cannot feed it fast enough.
Thermals and battery are the real ceiling
A thin laptop will sustain a fraction of its peak for more than a minute. If you plan to run long jobs, expect the machine to throttle, and expect the fan to be audible. Local inference is a sustained workload, not a burst.
What to buy
Buy the memory, not the badge. A mid-range machine with 16 GB of usable accelerator memory will run more useful models than a premium machine with 8 GB. Then check one review that measures sustained tokens per second after five minutes, not the launch figure.
Bottom line: decide which model size you need first, add context overhead, and buy slightly above that. Everything else on the specification sheet is secondary for local AI work.
