How OpenAI's 120B Model Fits on One GPU: Mixture of Experts Explained

0:00The Number That Doesn't Fit

A hundred and twenty billion parameters, running on one graphics card. That's OpenAI's open model, gpt-oss — and its little brother runs on a sixteen-gigabyte laptop. That shouldn't work. A model that size, saved the way models usually are, is more than two hundred gigabytes. That card holds eighty. And yet it runs. The reason is a trick with a strange name: mixture of experts. Some of the biggest open models of the last two years are built on it — DeepSeek's, OpenAI's — and it has quietly changed what the size of a model even means. A hundred and twenty billion is how much room it takes. Five billion is how much work it does. Those used to be the same number. Now they're not — and the gap between them is the whole story. So: what a parameter is, how the trick splits a model, the catch nobody puts in the headline — and what it means for the model you'd run on your own machine.

1:05Dense — Every Light On

Start with what a parameter is. A model is a giant grid of numbers — billions of them — and each number is a parameter: a value it learned during training. Your prompt goes in as numbers, gets multiplied through that grid one stage at a time — engineers call them layers — and out the far end comes a guess at the next word. Technically the next piece of a word — a token — but let's just say word. Now the cost, and there are two of them. First, room. Saved the usual way, each parameter takes two bytes. So count the parameters, double it, and that's your gigabytes. Second, work. In a normal model, every word you generate passes through every single parameter. Every light on, every time. So the work per word is the same count as the room. Meta's Llama is the famous example. Its biggest version has four hundred billion parameters — over eight hundred gigabytes just to hold it. Meta had to shrink the numbers down to eight bits — one byte instead of two — just to squeeze the thing into one server. And Meta picked this, on purpose, over a mixture of experts, because it was more stable to train. That's the plain deal — engineers call it dense: the room it takes and the work it does are the same number. Keep those two side by side. Now let's break them apart.

2:50Experts — The Split

Here's where those hundred and twenty billion parameters actually live in gpt-oss. Almost all of them sit in one part of each layer — the big block that does the heavy lifting. So cut that block up. Instead of one enormous block per layer, gpt-oss has a hundred and twenty-eight smaller ones, side by side. Each one is called an expert — and the name is a bit of a lie. An expert isn't a specialist in law, or code. It's just one slice of the grid. In front of the slices sits a router: a tiny extra piece that looks at the incoming word and gives every expert a score. Then the rule — and this is the entire trick — only the top four fire. The word goes through those four slices, their answers get blended, and the other hundred and twenty-four sit dark. And that happens in every layer, with a fresh pick each time. Now add up only the parameters one word actually touches — four experts per layer, plus the small stuff around them — and you get about five billion. That's what's called the active parameters. The total is still a hundred and twenty billion. But the work per word is now about a twentieth of that. The work just collapsed. Now the room.

4:10The Catch — Memory Doesn't Shrink

So the work per word fell from a hundred and twenty billion to five. Did the room fall with it? No. Not by a byte. And the reason is the router. It picks four experts per word, per layer — and it can't know which four until the word shows up. The next word wants a different four. Watch a whole sentence go through: first word, four experts light in a layer. Second word, four different ones. Third, another set again. By the end of the sentence, any of the hundred and twenty-eight could have been called — so none of them could have been unloaded in between. That's what your card is doing while an answer types out: holding all of them, using four. From one word to the next, any expert can get called. So all hundred and twenty-eight have to be loaded, sitting in memory, ready to go — even though at any instant, only four of them are lit. That's the deal mixture of experts makes, and it's the part the headlines leave out: much less work per word, but not one byte less room. Per word, your card does about as much arithmetic as it would for a five-billion model. But it still has to hold all hundred and twenty billion. So the two numbers drifted apart. The big number — the one in the name — is the room. The small number, the active one, is the work. And that puts us right back at the problem we started with. Two hundred gigabytes of model. Eighty gigabytes of card.

5:48Quantization — The Squeeze

The fix for room is the oldest fix in the book: store each number with fewer bits. That's quantization — rounding: keep the first few digits of every number and drop the rest. Two bytes per parameter becomes one, or half of one, and the model gets a little less precise for it. Meta did that to Llama to fit a server. OpenAI went further — and the reason it could is that split from earlier. The experts are more than ninety percent of everything the model stores. Shrink them, and you've shrunk the model. So the experts in gpt-oss are saved in a four-bit format — about a quarter of the usual space — and everything else is left alone. The file on disk lands just over sixty gigabytes. And sixty fits on an eighty-gigabyte card, with room to spare. That's how the big one fits. A hundred and twenty billion parameters, a handful working at a time, most of them squeezed small — and a model that should need three cards runs on one. Now the small one. Same recipe, fewer experts. On disk it's about thirteen gigabytes. That fits a sixteen-gigabyte laptop. And per word, it does the work of a model a fifth its size. So when a model page shows you two numbers — which one are you supposed to read?

7:10Two Numbers — What You Store, What You Run

Once you see the two numbers, the last two years make sense. DeepSeek's V3 is the same trick, bigger. Six hundred and seventy billion total. Thirty-seven billion doing the work. And DeepSeek reported training it in about a tenth of the graphics-card hours that Meta's dense Llama took. Different chips, different data — but on paper, that's the bill. That's the pull: not a smarter model — a cheaper one per word, paid for in memory. So, choosing a model for your own machine. Every model now has two numbers, and they answer two different questions. The total tells you whether it fits: double it for gigabytes — less if it's quantized — and compare that to your RAM. The active number tells you how much work each word costs. So read a model page in that order. Total first — does it fit? Active second — what does a word cost? And if the page shows only one number, it's a dense model: the same number does both jobs. That's the whole reading skill. It takes ten seconds, and it's the one most people skip. A twenty-billion model that fits on your laptop and does the work of a four-billion one — that's not a marketing line. That's the router, waking four of its thirty-two experts, layer after layer, while the rest wait in memory for their turn. The number on the box is the room it takes. The number underneath is the work it does.