Qwen on Your GPU vs the Frontier: What a Local LLM Actually Gives Up

0:00The Actual Difference

Here is a sum. One, plus eleven, plus one hundred and eleven, and so on, three hundred numbers long. Then add up every digit of the total. One tester gave it to the small model on a home graphics card, and it came back with one thousand two hundred and twenty-nine. Now give it to Claude or ChatGPT, which live on servers you will never see. One thousand two hundred and thirty. One of those is wrong, and it is the one under your desk. Both are the same kind of machine, a transformer. Both chop your words into small pieces called tokens, and both push tokens back out, one at a time. On paper, the model in your own computer and the hosted ones look alike. So what is the actual difference? Why do the hosted models such as Astra or Fable answer so well, and so fast, when the one under your desk is doing the very same job? The answer comes down to five physical things. The companies behind ChatGPT and Claude never publish what is inside Astra or Fable. So every number about what is inside a model comes from Alibaba. Alibaba shipped a small model that runs on a desktop card. On some of Alibaba's own coding tests, it beats an earlier Claude Opus. It shipped a big one too, the kind that needs a datacenter, and published the full weights of both. Now follow the same sum as it goes out twice. One copy goes to the small model on the card in your room. One copy goes to a big model such as Astra or Fable. At five places the two copies come apart: memory, rounding, the read, the conversation's notes, and where the small model learned its tricks. But first, what is a model made of?

2:25Inside the File

Before the two copies go anywhere, open the small model's file and look at what is inside. It is a long list of numbers called weights, billions of them, and each is a plain decimal. The file writes each weight in two bytes, the same space the word hi takes in a text file. Two letters, or a single weight, two bytes either way. Now count them. Alibaba's small model has twenty-seven billion weights. Twenty-seven billion, times two bytes each, is fifty-four billion bytes. A billion bytes is a gigabyte, so the small model is a fifty-four gigabyte file. That is the whole model, a download about the size of a large game. Now do the same sum for the big model, which has two point four trillion weights. At two bytes each, the whole file comes to nearly five terabytes. It is the same kind of file, with the same kind of plain decimals inside, and there are just far more of them. So that is what a model is, physically: a file full of weights, with a size you can work out yourself on the back of an envelope. Now all of those bytes have to fit somewhere before a single token comes out.

3:58The Memory

Every one of those bytes has to sit in the graphics card's memory before the first token comes out. Watch the fifty-four gigabytes land on your card, which holds twenty-four gigabytes. The file does not fit, and we will fix that in a minute. Now the big model's five terabytes. That is not a bigger card, that is a different kind of building. Nvidia's own recipe for running it names a liquid-cooled rack, a tall box full of its GPUs. It spreads the model across a block of them. The big model's weights are split into a crowd of specialists. For any one token, only a few of them do the work, about ninety-five billion weights out of the two point four trillion. That trick is called mixture of experts, and we covered it in the Mixture of Experts video. But the next token might need any of the others, and there is no time to fetch them from disk. So all of them stay loaded. Think of a hospital: one patient sees one doctor, but the building keeps every specialist on site, because nobody knows who walks in next. The small model is small enough to carry home in a single file, and the big model is bolted to the floor of a building. The sum is still waiting on a card that is far too small.

5:39The Rounding

So how does a fifty-four gigabyte model fit on a twenty-four gigabyte card? You round it. Each weight's bytes hold sixteen bits of detail, and rounding, or quantization, keeps the weight but writes it with about four bits instead, a few weights keeping a little more. Look at one weight as it gets rounded: the long decimal snaps to the nearest step on a much shorter ruler. Do that twenty-seven billion times and the file shrinks from fifty-four gigabytes to nineteen, and now it fits with room to spare. That nineteen-gigabyte copy is the file sitting on your disk. So what did you give up? The benchmark numbers Alibaba published came from the unrounded model, and the file on your card is a rounded copy of it. Does that rounding cost you the answer? In one August test, the four-bit copy passed a long coding benchmark about as often as the original. The rack has the memory to run the weights exactly as Alibaba shipped them. Round the weights harder, down to one bit, and the answers collapse to guessing. At four bits the copy keeps its answers, and only thinks longer. The sum now runs on the rounded copy, while the rack runs the original.

7:14The Read

Now watch the answer come out, one token at a time. To produce each token, the card reads every weight once, all nineteen gigabytes, and then writes a single token. Then it reads all nineteen gigabytes again for the next token, and again for the one after that. Nvidia's engineers put it plainly: how fast the memory can be read decides the speed, and the chip's arithmetic mostly sits waiting. Your card moves about a thousand gigabytes a second, and a thousand divided by nineteen is about fifty. So on your desk, alone, the ceiling is roughly fifty tokens a second at best, and that is before the conversation's notes take their share. Now put the rack beside it. Nvidia's rack holds seventy-two GPUs, and together they read memory about five hundred times faster than your card. But the raw speed is not even the trick. The trick is that the rack never reads the weights for one person. It stacks thousands of questions together and answers all of them from a single pass over the weights. Your sum rides somewhere in the middle of that stack. That is how a model that big can be sold by the million tokens at all. The read is shared, so each question pays for only a sliver of it. So at home, one full read of the weights answers one person. In the rack, one read answers thousands of people at once.

9:06The Notes, and the Teacher

There is something else on your card, and it grows while you type. Every token of your conversation leaves a note in memory, so the model does not reread the whole chat for each new token. Engineers call that store the KV cache, and Nvidia's formula says it grows in a straight line with the length of the conversation. The longer the conversation, the more of your card the notes fill up. Watch it on your card: the weights already take nineteen gigabytes, and the cache takes whatever is left. When that runs out, the conversation stops, or the model starts forgetting the beginning. Alibaba says the small model can hold about two hundred and fifty thousand tokens. On your card, the notes run out of room long before that. The rack has room, and the hosted models sell that room in different ways. Claude Fable holds a million tokens at a flat price. Astra holds about a million too, but OpenAI's own page says that past two hundred and seventy-two thousand tokens, it charges double for the input. Even the datacenter pays for those notes. So how did a small model get good enough to beat Opus on anything? Partly by studying the big models' output, the way a student learns from a teacher's worked answers. Anthropic's report this month says Alibaba harvested over a hundred and fifty-one million exchanges with Claude to train earlier Qwen models, and that is Anthropic's claim, not ours. The report says a student trained that way gains even on tasks it never practised, and that the safety training does not copy across with it. Now both copies of the sum are back. The right answer is one thousand two hundred and thirty, and Claude and ChatGPT both gave it. In that tester's run, the small model worked for twenty-seven thousand tokens and said one thousand two hundred and twenty-nine. Somewhere along the way it slipped, and the whole answer is wrong. The rounding did not do that. You saw the rounded copy pass the same test as the original. An older, smaller cousin with eight billion weights forgot the question. It invented an earlier conversation and answered a question nobody asked. A thirty-billion cousin works each token with only three billion of them. It thought for ten minutes, answered a different sum, and then repeated its answer over and over. Alibaba's own tables put the big model ahead on every hard test the two share. A small model also thinks longer, and on a long sum it still slips, or loops. That is what the extra weights buy. In a long conversation, your card runs out of room for its notes, and the rack does not. And the rack answers fast, and sends you a bill. Here is what you keep. The nineteen gigabytes on your desk are yours, and nobody can reprice them, change them, or switch them off.