A mini PC on a workshop shelf casting a shadow the size of a data center
FN·074 · The gap between the box on your shelf and the frontier is now a few index points

For most of the last three years, the honest answer to "should I run an open model instead of paying an API" was no. Not because open weights were bad, but because the gap was embarrassing. You would take a model that scored somewhere around a third of what the frontier scored, wire it into a workflow, and then spend a month apologizing for it.

That answer changed this summer, and it changed quietly enough that most people have not updated. As of today, the closed frontier sits at roughly 53 on the Artificial Analysis Intelligence Index. The best open weight model, Moonshot's Kimi K3, sits at 50.2 on the open-weights tracking at theopenweights.com. Zhipu's GLM-5.3 is right behind it at 48.6.

Call it 6%. That is not a benchmark curiosity. It is the difference between "open models are a toy" and "open models are a procurement decision." I build local AI agents that run on mini PCs my clients own outright, so this is not an abstract argument for me. It changes what I can promise, and it changes where I still have to say no.

But there is a catch buried in that number, and it is the whole reason this post exists. The model that closed the gap is a 2.8 trillion parameter machine. You are not running it on a mini PC in a back office. The thing that actually ships on your hardware is a much smaller model, and the interesting question is how much of that 50.2 you get to keep when you shrink down to something that fits under a desk.

Key Takeaways

The frontier gap closed at the top of the open tier, not at the consumer tier. Those are two different stories and conflating them is how people end up disappointed.

Point Details
The gap is real but narrow Closed frontier around 53 on the Artificial Analysis Intelligence Index; Kimi K3 at 50.2 open weights, GLM-5.3 at 48.6.
The best open model is not a local model Kimi K3 is 2.8T total parameters with 104B active across 896 experts. Weights are public, but the hardware to serve it is data center class.
Architecture beats raw size on your box Generation is memory-bandwidth-bound, so a sparse MoE with few active parameters can be roughly an order of magnitude faster than a dense model of the same total size on the same machine.
Memory is the gate, not the GPU brand A workable rule of thumb at 4-bit: gigabytes needed is about parameter count in billions times 0.6, plus headroom for context.
Local is a control decision first You run locally for data residency, cost predictability, and no vendor deprecating your model. You do not run locally to beat the frontier on reasoning.

What Does 6% Behind Actually Mean?

Composite benchmark indexes are useful and also easy to over-read, so let me be precise about what I am claiming.

The Artificial Analysis Intelligence Index is a composite. It averages a basket of evaluations covering reasoning, coding, agentic tool use, and long-context work, and it gets revised as the old evals saturate. A single number from a composite tells you roughly where a model sits in the pack. It does not tell you whether the model will handle your specific job, and it definitely does not tell you which model writes better copy for a roofing company in Las Vegas.

So when I say 6%, I mean this: on a broad basket of hard tasks, the best publicly downloadable weights now land close enough to the best proprietary model that the difference is no longer the deciding factor for most business workloads. Two years ago the gap was wide enough that you could feel it in every response. Now you have to go looking for it, and you find it in specific places: multi-step agentic reliability, the hardest reasoning tails, and the polish of the surrounding product.

That last one matters more than people admit. When you pay for a frontier API you are not just buying weights. You are buying a serving stack that has been tuned by a team, a tool-use harness that has been debugged against millions of real calls, safety filtering, uptime, and a support path when something breaks at 2am. Download the weights and all of that becomes yours to build. The 6% is the model gap. The engineering gap is larger and it is the part nobody benchmarks.

I still find the number encouraging, because the direction of travel is what a small shop like mine plans around. If open weights were 40% behind and holding, building on them would be a bet against the market. At 6% and narrowing, building on them is just where the industry is going.

What Is the Best Open Source LLM Right Now?

By the composite scores, Kimi K3 from Moonshot AI. The weights went up on Hugging Face on July 26, 2026, and the model card is unusually specific about the architecture, which is worth reading if you care about how these things are built.

It is 2.8 trillion total parameters with 104 billion activated per token, routed across 896 experts with 16 selected per token. The context window is 1,048,576 tokens, which is the real million-token figure rather than a marketing round number. It ships with quantization-aware training baked in, using MXFP4 weights with MXFP8 activations, and it is natively multimodal across text, images, and video via a 401M-parameter vision encoder.

Right behind it is GLM-5.3 from Zhipu at 48.6, and the tier below that is populated by Alibaba's Qwen family and DeepSeek. One important caveat on terminology: several of these are open weight rather than open source in the strict sense. Kimi K3 ships under its own license rather than a standard permissive one. The weights are downloadable and deployable, the training data and full pipeline are not, and the license carries conditions worth reading before you build a commercial product on it. Compare that to OpenAI's gpt-oss models, which are Apache 2.0 with no copyleft and no patent trap, and you can see why "open source LLM" as a phrase has gotten sloppy.

For a client build, I care about the license more than the leaderboard. A model two points lower on an index but clean under Apache 2.0 is often the better engineering answer, because I can hand the whole thing to the customer without a lawyer in the loop.

Why Is the Top Open Tier Almost All Chinese Labs?

Look at the top of the open-weight table and it is Moonshot, Zhipu, Alibaba, and DeepSeek. That is not a coincidence and it is not a fluke of one release cycle. It has been the shape of the open tier for a while now.

The strategic logic is fairly legible. If you are not the incumbent selling the most expensive API in the world, giving weights away is a way to become the default substrate that everyone builds on. It buys distribution, developer mindshare, and a research reputation that recruiting runs on. American labs at the frontier have the opposite incentive: their revenue is the API, so releasing the thing that competes with the API is a harder internal argument. gpt-oss was a real and welcome exception, but it was released well below the flagship line.

For an operator, this raises a practical question rather than a geopolitical one. If you are deploying weights from a Chinese lab on hardware you own, on your own network, with no outbound calls, the data never leaves your building. That is the entire point of running locally. The weights are a file. They compute, they do not phone home, and you can verify that with a firewall rule and a packet capture rather than by trusting anyone's terms of service.

Where I do get careful is with anything regulated or contractual. Some clients have procurement language about model provenance, and some industries have auditors who will ask. If that is you, the Apache 2.0 American-lab options exist for exactly this reason, and the two-point difference on a benchmark will not be what your business lives or dies on.

Can You Actually Run the Best Open Source LLM Yourself?

No. Not the top of the table, anyway, and I want to be blunt about this because there is a lot of content out there that blurs it.

"Open weights" means the file is downloadable. It does not mean the file is servable on your hardware. A 2.8 trillion parameter model, even at aggressive 4-bit quantization, needs well over a terabyte of fast memory just to hold the weights, before you allocate anything for the key-value cache that a long context requires. That is a multi-node data center deployment with high-bandwidth interconnect. It is a real thing that real companies do, and it is not a thing that happens in the back office of a dental practice.

The MoE architecture helps with compute but not with capacity. Only 104B of those 2.8T parameters activate per token, which is why the model is economical to serve at scale. But all 2.8T parameters have to be resident somewhere the router can reach quickly, because the router picks different experts for every single token. Sparse activation cuts your bandwidth bill per token. It does not cut your memory bill.

So the honest framing is a two-tier world. There is the frontier open tier, which you rent from an inference provider or self-host on serious infrastructure, where the 6% gap headline is true. And there is the local tier, which runs on hardware you can buy and put in a closet, where the models are much smaller and the gap to the frontier is considerably wider than 6%. Both are real. Only one of them is what I install for clients.

Which Open Weight Models Fit the Box You Own?

Four workshop shelves holding progressively larger boxes with glowing orange cores
Memory tiers · 16GB, 32GB, 128GB, 512GB decide what fits

Local model selection is a memory problem before it is anything else. The question is never "which model is best," it is "which is the strongest model whose weights and working context fit in the memory this machine has, at a speed a person will tolerate."

The rule of thumb I use for sizing: at roughly 4-bit quantization, gigabytes needed is about the parameter count in billions times 0.6. A 30B model wants somewhere around 18GB for weights. Then you add headroom for the context cache, the operating system, and whatever else is on the box. That is why a 16GB machine and a 24GB machine are genuinely different tiers rather than a small upgrade.

Memory available What fits comfortably What it is good for
16GB gpt-oss-20b (21B total, 3.6B active, MXFP4, Apache 2.0) Classification, routing, extraction, summarizing a call transcript, drafting a reply for a human to approve.
24 to 32GB qwen3-coder:30b, Qwen3.6-35B-A3B class sparse MoE models Serious agent work: tool calling, structured output, code edits, multi-step workflows with retries.
128GB unified (AMD Strix Halo class) gpt-oss-120b (117B total, 5.1B active) The best local quality most small businesses will have a use for, still on a single small box.
512GB unified (Mac Studio M5 Ultra) Very large open models at 4-bit, including 400B-class weights Research, evaluation, running something close to frontier-adjacent quality on a desk.

That last row is new. Apple announced the M5 Ultra on August 25, 2026, with configurations up to 512GB of unified memory at up to 1.2TB/s of bandwidth. Unified memory is the reason a Mac punches above its GPU spec for this work: the model does not have to fit in a separate VRAM pool, and the bandwidth number is what actually determines your tokens per second.

Most of my client builds land in the 16GB to 32GB tier, on a small x86 mini PC, because the jobs those businesses need done are narrow. If you want the hardware side of that in detail, I wrote up the box selection in best mini PC for an AI agent and the full local inference walkthrough in the mini PC guide to local inference.

Why Does a MoE Model Run So Much Faster Than a Dense One?

Two locomotives on parallel tracks, one hauling a hundred cars and crawling, the other hauling three and racing ahead
Dense vs. mixture-of-experts · same engine, fewer cars per token

This is the single most useful thing to understand about local inference, and it is the thing that trips up almost everyone sizing their first rig.

Token generation is memory-bandwidth-bound, not compute-bound. To produce one token, the machine has to read the parameters it needs out of memory and multiply them. On consumer hardware, the multiplying finishes long before the reading does. The GPU or the NPU sits there waiting on memory. So your generation speed is roughly a function of how many bytes of parameters have to be pulled per token, divided by how many bytes per second your memory can deliver.

A dense model reads every parameter for every token. A sparse mixture-of-experts model reads only the active experts. So a 30B model with 3B active parameters is reading roughly a tenth of the bytes per token that a dense 30B reads, on the same box, at the same quantization. The speed difference in practice tends to be around an order of magnitude, sometimes better once the runtime is tuned.

The practical consequence: on a 128GB Strix Halo class machine, a dense 70B at 4-bit is roughly a handful of tokens per second, slow enough that a person watching it type will go get coffee. A 30B-A3B sparse MoE on the same machine runs at conversational speed, comfortably faster than most people read. Treat both of those as approximate ranges rather than specs, because your numbers will depend on quantization, runtime, context length, thermals, and whether anything else is competing for the machine. I benchmark our own rigs before quoting a client a latency number, and I would encourage you to benchmark yours rather than trusting any figure you read online, including mine.

What this means for model shopping is counterintuitive. Total parameter count tells you whether the model fits. Active parameter count tells you how fast it will feel. Two models with the same total size can behave like completely different products on the same box, and the sparse one is usually the right answer for anything interactive.

Why Run an LLM Locally At All?

If open models are still a bit behind, and running them is a hardware problem, why bother? Four reasons, in the order my clients actually care about them.

Data residency. Some businesses cannot send customer conversations to a third party, either because of regulation or because of a contract they signed. A medical practice, a law firm, a company handling someone else's client list. For those shops the question is not whether the frontier model is smarter. It is whether the data can leave the building at all. If it cannot, local is not a preference, it is the only option.

Cost shape. An API bill scales with usage, which is fine until the agent starts working and the volume goes up. Hardware you bought is a fixed thing that already happened. The economics flip somewhere in the middle, and the crossover comes sooner than people expect once a workflow is running all day. More importantly, a fixed cost is easier for a small operator to plan around than a variable one.

Nobody deprecates your model. This one gets underrated until it bites you. Hosted models get retired, quantized differently, silently retuned, or repriced. If you tuned your prompts and your evals against a specific version, a vendor's roadmap becomes your maintenance schedule. A file on your own SSD does not change unless you change it. For an agent that has been carefully tuned around one business's workflow, that stability is worth real money.

Latency and offline operation. No round trip to a data center, no dependency on the shop's internet being up. For anything that has to work when the connection does not, this is the whole game.

None of those four reasons is "the local model is better." They are all reasons about control. That is the honest sales pitch, and it is the one I make. If your only criterion is maximum intelligence per query, pay for the frontier API and be happy. I have written more about how this plays out in a real deployment in the local AI agent on a mini PC note.

What It Doesn't Change

Here is the part I would want to hear if I were the one buying.

A local 30B is not a frontier model. The 6% headline belongs to a 2.8 trillion parameter machine. The model on your mini PC is two orders of magnitude smaller and the gap is correspondingly larger. It will be excellent at narrow, well-specified jobs and it will fall apart on open-ended reasoning that a frontier model handles casually. Scope the work to what the model can actually do and it is reliable. Ask it to be a general-purpose genius and it will disappoint you.

You own the maintenance too. "You own the rig" cuts both ways. The runtime gets updates. Quantizations improve. A better model in the same memory tier lands every few months and someone has to evaluate whether swapping is worth it. Disks fail, fans clog, a power cut needs a clean restart. This is not heavy work, but it is not zero, and pretending otherwise would be dishonest. That is what an optional managed plan is for, and it is also entirely fine to handle it yourself if you have someone technical in the building.

Benchmarks are not your workload. An index score is a proxy. The only evaluation that matters is a set of real examples from your business, with the answers you would have wanted, run against each candidate model. I build that eval set before picking a model, every time. It usually takes an afternoon and it has changed my mind more than once.

The hard part was never the model. Getting a model to answer well is maybe a fifth of the work. The rest is plumbing: reading from the systems the business already uses, writing back to them, handling the case where the API is down, deciding what escalates to a human, logging enough to debug it in three months. That work is the same whether the weights are open or closed, and it is where the schedule actually goes.

How I Pick a Model for a Client Build

The process is boring, which is the point. It runs roughly in this order.

  1. Write the eval first. Twenty to fifty real examples from the client's actual work, with the output I would accept. No model gets picked before this exists.
  2. Set the memory budget. Determined by the box, and the box is determined by where it has to live and what it has to survive. A closet in a shop is a different spec than a rack.
  3. Filter by license. Anything that cannot be cleanly handed to the customer as part of a system they own is out, regardless of score.
  4. Prefer sparse over dense in the same memory tier. Almost always the right call for interactive work, for the bandwidth reasons above.
  5. Benchmark on the actual hardware. Tokens per second, time to first token, and behavior under the longest context the workflow will really see. Numbers from a review site are a starting hypothesis, not a spec.
  6. Run the eval on the top two or three and pick on results. Not on leaderboard position.

Field note: the most common mistake I see people make when they try this themselves is buying a machine with plenty of raw GPU compute and not enough memory bandwidth. They get a box that trains and renders beautifully and generates tokens at a crawl, because generation was never a compute problem. Check the memory bandwidth number on the spec sheet before you check anything else. It is the single figure that predicts how the thing will feel to use.

The second most common mistake is picking the biggest model that technically fits. A model that fills 90% of your memory leaves nothing for context, and the moment a real conversation gets long the machine starts swapping and the whole thing falls over. Leave room. A slightly smaller model with comfortable headroom beats a bigger one that thrashes, every time.

If you want this handled rather than researched, that is what a custom build is. I scope one workflow, pick the model against your own examples, deploy it on hardware you own, and hand you the whole thing. Same person for the scoping and the code, because there is only one of me.

Worth Watching: Five Videos That Show the Gap Closing

Reading benchmark tables is one thing. Watching someone load a 120B model on a mini PC and get usable speed is what usually convinces an owner. These five are the ones I send people, in the order I would watch them.

None of these channels are affiliated with Pulp AI Studio. Their numbers are their own; benchmark your rig before you quote any of them to a client.

Sources

FAQ

What Is the Best Open Source LLM in 2026?

By composite benchmark score, Kimi K3 from Moonshot AI at 50.2 on open-weight tracking, with GLM-5.3 from Zhipu at 48.6 behind it. Both are far too large to run on consumer hardware.

How Far Behind Are Open Source LLMs?

About 6% at the very top of the open tier, comparing roughly 50.2 for the best open weights against roughly 53 for the closed frontier on the Artificial Analysis Intelligence Index. The gap is much wider for the smaller models that actually fit on a local machine.

Why Run an LLM Locally?

Four reasons: customer data never leaves your building, cost is fixed rather than metered, no vendor can deprecate or retune the model you tuned your workflow around, and it keeps working when the internet does not.

How Much Memory Do I Need to Run a Local LLM?

A rule of thumb at 4-bit quantization: gigabytes needed is about the parameter count in billions times 0.6, plus headroom for context and the operating system. A 20B-class model fits in 16GB, a 30B-class model wants 24 to 32GB.

Is a Mixture-of-Experts Model Faster Than a Dense One?

Yes, substantially, at the same total size on the same hardware. Generation is limited by memory bandwidth, and a sparse model reads only its active parameters per token, so it moves far fewer bytes than a dense model of the same size.

Are Open Weight Models the Same as Open Source?

Not always. Open weight means the model file is downloadable and deployable. Training data and pipeline are usually not published, and licenses vary from fully permissive Apache 2.0 to custom terms with conditions worth reading before commercial use.