Near-frontier AI intelligence now runs on a GPU you can buy today — and for me, that’s the most exciting story in a very busy few weeks of AI announcements.
The race among frontier labs keeps raising the bar for what AI can do, and that’s genuinely exciting. But the releases I keep coming back to are the open-weight ones — specifically, the models coming out of Alibaba’s Qwen lab.
As a consultant and software engineer working with many clients, I’ve had the opportunity to discuss and demonstrate to some of them that local AI could be a serious alternative to the major providers they are used to working with.
Last week alone, Alibaba’s Qwen made two major releases: first, the Qwen 3.8 27B model, and a few days later, Qwen 3.8 Flash Next. These models are impressive for many reasons, but a few stand out in particular:
- They can run on small to medium-sized hardware.
- They are versatile, all-around models with vision capabilities and excellent agentic scores.
- They perform well across many tasks, from coding and writing to document analysis, OCR, and more.
What amazes me is that, according to the Artificial Analysis indexes, these models are scoring on par with what frontier models achieved approximately six to eight months ago. This means that what was frontier-class intelligence only six to eight months ago is now available in an open-weight model that most companies can afford to run locally—and it is a much smaller model than the frontier alternatives.
Looking at the chart above, we can see that Qwen 3.8 27B (where 27B stands for 27 billion parameters, a relatively small model by today's standards) is in the same league as Google’s Gemini models. Meanwhile, the latest member of the family, Qwen 3.8 Flash Next (a 120B Mixture-of-Experts model with 6B active parameters and a 51B n-gram layer), competes with frontier models such as the latest GPT and Opus.
Why does a local model matter?
Benchmarks are only an indication of a model’s level of intelligence; real-world tasks can tell a different story. With previous Qwen generations, however, my everyday use largely confirmed what the benchmarks suggested. My first tests with the new models show the same pattern: Qwen 3.8 models are excellent workhorses for everyday tasks, whether parsing documents, powering RAG and agentic systems, summarising and translating content, or editing code.
Does this mean that a local model can completely replace a cloud subscription? Probably not, but local models can complement frontier models for many tasks and help reduce cloud bills. Beyond cost and performance, local AI brings another set of advantages: data sovereignty, governance, predictable costs, customisation, lower latency, and greater operational control. In some cases, companies do not want to—or are completely prohibited from—sharing content with external entities for security or privacy reasons. This is another area where local models can help.
What does it take to run these locally?
The short answer: less than you’d expect, with a payoff in speed that beats the cloud.
In my tests on a single-GPU system with a 24-core AMD Threadripper, 128 GB of RAM, and an NVIDIA RTX PRO 6000, Qwen 3.8 27B with FP8 quantization produced 30–50 tokens per second, even under 16 concurrent requests. Flash Next with NVFP4 quantization ran at 150–180 tokens per second, remaining above 110 tokens per second with 16 concurrent requests—that’s more than a paragraph per second, far faster than you can read. Cloud services can be slower, although they can, of course, scale far beyond a single local system. That’s frontier-adjacent intelligence, at speeds that can outperform cloud inference, running on hardware sitting under your desk.
What you’d actually need:
- Qwen 3.8 27B: runs on a single 24 GB consumer GPU (RTX 4090/5090) with quantization; a 32–40 GB card is more comfortable for concurrent workloads and the full 260k-token context. Previous-gen enterprise cards like the L40S work too.
- Flash Next: more demanding — an RTX PRO 6000 (96 GB VRAM) with 128 GB system RAM to host the offloaded layers — but it plays in a completely different performance league.
One reason the newer Blackwell cards (RTX 5090, RTX PRO 4000/5000/6000) pull ahead is native NVFP4 hardware acceleration. Pairing NVFP4 quantization with their high memory bandwidth is what delivers the throughput above.
The architecture explains the speed difference. The 27B is a dense model — all 27 billion parameters are read to produce every token. Flash Next, despite holding 120B + 51B parameters, is a mixture-of-experts design that activates only 6B per token, and it uses the extra 51B n-gram layers to correlate groups of tokens — a preview of the upcoming Qwen 4.0 generation.
Conclusions
Given the promising results from my recent tests, I’m already updating my local stacks with the new models to benefit from their improved intelligence and speed. I can only reaffirm my previous advice: Qwen models should be seriously considered by companies that want to experiment with AI and agentic workflows without huge budgets or sending private information to external cloud providers. Even when providers offer strong contractual privacy guarantees, keeping sensitive data entirely inside the company's security boundary removes a category of third-party exposure.
Credit is due to Alibaba’s Qwen team and to the other labs investing significant resources in developing these models and making them available to the community under permissive open licenses. Their work is rapidly changing what is practical outside the large AI clouds.
The interesting question, then, may no longer be whether local models can replace frontier models entirely. It is how often companies actually need frontier intelligence. If most everyday AI workloads can run locally—with predictable costs, high performance, and data remaining inside the organisation—cloud models can increasingly become the escalation path rather than the default. The rain of increasingly capable open-weight models isn't going to make the cloud disappear, but it may move a surprising amount of AI back to the ground.
Author: Fabio Marini, IT Consultant and Senior Solution Architect at Impresoft GN Techonomy
