How ChatGPT Runs 800 Million Users on One Postgres Database

0:00Ten People

How would you build a system that eight hundred million people use every day? ChatGPT is one, and it runs on one database. Do not start at eight hundred million, though. Start with ten people and one box, as a thought experiment. Everything lives on that box: the app, and the database behind it. The database is Postgres, the same Postgres OpenAI runs, on Microsoft's cloud. Now watch what one message costs, slowly, because the whole video rests on it. You press send, and before the app can save anything, the app asks the database who you are. Then which plan you are on, and then this chat's settings. Three questions out, three answers back, and only then does the app write your message. So one send is several reads first, and one write last. Reads and writes are going to behave very differently as the crowd grows. Ten people send a message: ten little bursts of reads, ten writes, and the box answers every one before anyone notices a wait. This is the whole system, and it is enough. Then a hundred people come; the box is busier, and it still answers. When the box gets warm, do the boring thing first, the thing OpenAI did first too. Make the box bigger, and make the app ask the database for less. A bigger box buys a long time, but a bigger box is still one box, and every box has a ceiling. The next wave finds it.

2:01Ten Thousand, and the Doors

Ten thousand people, and not spread across the day. A post goes viral, and ten thousand people press send in the same minute. Every one of those sends needs a connection to the database, and Postgres treats a connection as a heavy thing. Opening one takes about fifty milliseconds. And the box can hold only five thousand at once. Think of each connection as a door: ten thousand sends, five thousand doors. The first five thousand get in, the rest queue and then time out, and the people who got refused press send again. Notice that the database was not tired, because it still had room to answer. The box ran out of doors, not out of work. OpenAI has been hit exactly this way, by a connection storm that used every connection there was. The fix is a doorman. A small program called PgBouncer sits between the app and the database, and keeps a few dozen real connections open all the time. When the app needs the database, the doorman lends it one open connection for one job, and takes it back the moment the job is done. The app never opens a fresh connection again. Fifty milliseconds at the door becomes five. And the ceiling stops mattering, because ten thousand sends now take turns on a few dozen doors, very fast. Now walk your message through it. You press send, and the app asks the doorman for a connection and gets one already open. The app runs its three reads and its one write on it, and hands it back. Five milliseconds at the door instead of fifty, and the door is free again for the next person. Same box, same wave, one doorman. Ten thousand goes through clean, and then the next wave arrives.

4:22A Million, and the Reads

A million people, and the doorman is fine; connections were never the limit again. This time the database itself is the limit. Every send is several reads and one write, so a million people chatting is millions of reads, and one machine has to compute every answer. The box's processor pins at full, and every answer, reads and writes alike, slows down. You cannot buy a box big enough. But look at what the reads are asking: who you are, your plan, this chat's settings. A copy of the database holds exactly the same data, so a copy can answer those. But a copy cannot take a write. If two copies each accepted a message, the two copies would hold two different truths, and nobody could say which one was real. One place has to own every write. So the design splits the two halves of every send. One database takes every write, and from here on we call it the writer, and copies take the reads. Now follow your message through it. The app reads who you are, your plan and this chat's settings from a copy, and writes the message to the writer. The writer passes that change to every copy, in order, a moment later, and the next read sees it. Growth on the read side is now easy: more people, more copies. OpenAI runs nearly fifty copies, spread across regions, so your read is answered by a copy near you. Here is the moment to sit with. For a heartbeat after you send, a copy may not have your message yet. The writer has it, and the copies get it a moment later. That small lag is the price of copies, and for a chat it is a price nobody notices. Reads spread, writes stay. A million goes through clean, and the next wave is a real number.

6:52A Hundred Million, and the Same Question

A hundred million people, and OpenAI has seen that many arrive in a single week. Nearly fifty copies now answer every read by hand, and most reads ask the same question. Who you are and which plan you are on is the same answer on every message you send. So a hundred million people keep asking the copies the same thing. And adding copies is not free any more, because the writer sends every change to every copy itself. Past some number, the writer does nothing but feed copies. The fix is a cache in front of the copies, a fast store that remembers recent answers. Your second message of the day never asks a copy who you are, because the cache already knows. Now walk your message through it. You press send. Who you are: the cache has it. Your plan: the cache has it. This chat's settings: the cache has it, so not one read reaches a copy. Then the write goes to the writer, as always. OpenAI says the cache serves most of the read traffic. That is what makes nearly fifty copies enough for millions of queries, or questions, a second. The copies are now only for what the cache forgot. And then the cache has its own bad day. The cache is its own set of machines, holding answers in memory only. A cache machine restarts, or an update replaces it, and its memory starts empty. Everything that machine knew is gone in the same instant. Every read the cache was answering now misses, finds nothing, and hits the copies in the same second. OpenAI calls that a cache-miss storm. Look at where the storm is worst: the entries many people share, one plan's limits, one hot chat. Thousands of readers miss the same entry at once, and every one runs the same query on a copy. The guard is a lease, a hold that works one entry at a time. The first reader to miss on an entry takes the hold and goes to the copy alone. Every other reader that misses the same entry waits for that one answer. One query fills the cache, and everyone waiting gets it. One reader goes, everyone else waits. A hundred million goes through clean, storm included. And still, every write in all of that landed on one machine.

10:05The Writer

Look at what the last three fixes never touched. The doorman, the copies and the cache all serve reads. The writer is still one box, and it takes every write from a hundred million people. Why does a write cost the writer so much? When the app changes one field in a row, Postgres copies the whole row to a new version, and leaves the old one lying there. Under heavy writing, the writer spends its time copying rows and cleaning up old versions. Even the writer's own reads have to step over the dead ones. That is the one weakness this whole design is built around. And a write storm is how a new feature breaks you: a launch that writes more than anything before it, aimed at the one machine nothing copies. OpenAI's rule for the writer is: keep it as empty as possible. Empty means headroom, so a writer that idles has room when a spike comes, and a writer near full has none. So, write less: bugs that wrote the same thing twice got fixed, and a write that can wait, waits. And no new tables on the writer. Any new feature that needs its own tables of data gets those somewhere else. Somewhere else is for the heavy writing that splits cleanly, per user, say, so no single machine has to see all of it. That writing goes to Cosmos DB, a sharded store, which means many small databases behind one name, each owning one slice. A write goes only to its slice, and more writes just means more slices. Writes spread there the way reads spread across copies here. The message itself stays on the writer, because your message must be one truth, and hundreds of places in the app's code assume one database. So here is the honest version of the hook. OpenAI did not split the Postgres database. OpenAI moved the heaviest writing out of it, and kept one writer for everything that stayed. Commenters on Hacker News said the quiet part: that is sharding, just not of this database. Fair enough. Why not shard Postgres itself? Rewriting hundreds of places in the code would take months or years, and reads are most of the work anyway. OpenAI says it may still do it one day. Now walk your message one more time. Your three reads go to the cache, or a copy, and your one write, the message itself, still goes to the writer. But the heavy writing behind a new feature never comes near the writer; it goes to the sharded store. The writer only ever sees the writes that must be one truth. Keep the writer empty. The writer holds, and now the last wave, the real one.

13:41Eight Hundred Million

Eight hundred million people. This is not a thought experiment any more; this is the machine OpenAI runs. Then came the week that tested it. When ChatGPT launched ImageGen, its image maker, over a hundred million new people signed up in a single week. Writes jumped more than ten times, aimed at the writer. OpenAI's database outages have often gone the same way. A spike hits the database, answers slow down, and slow answers become timeouts. The app retries the timed-out requests, and the retries land on top of the spike. That loop is the outage, and OpenAI's guards are built to break it. Rate limits, caps on how much gets through, stand at four checkpoints: the app, the doorman, the proxy and the query. No spike reaches the database whole. Next, the app waits longer before each retry, so a retry never tips the spike over. Behind the writer sits a standby: a copy kept in step at every moment, ready to take over if the writer dies. Reads keep working from the copies the whole time. The spike is cut four times before it reaches the database. That week, the writer was the part that hurt, because the writer is the one part nothing copies. One outage of the worst kind, the only one in a year. One bad week in a year, on the one machine nothing copies. It held, six waves and five fixes later. Now look at the whole machine at once.

15:48The Whole Machine, One Message

This is the whole machine now, the one OpenAI runs. Send one last message and follow it through everything. You press send, and your request meets the first checkpoint, a rate limit, and passes. The app asks the doorman for a connection and gets one that is already open. Your three reads go to the cache, and the cache knows who you are and your plan. Watch: the cache has forgotten this chat's settings, so one reader goes to a copy near you, and the cache learns the answer for everyone after you. Your one write goes to the writer, the only machine that takes writes. The writer passes the message to nearly fifty copies a moment later, and the standby behind the writer already has it. Notice that nothing about your message touched the sharded store; that is where the heavy writing of new features went. The answer comes back to you. Reads spread, write kept, and that is the whole design. This machine answers millions of queries a second for eight hundred million people. It is up five nines of the time: almost never down. Load grew more than ten times in the last year, and it is still growing. So the answer to how you get from ten people to eight hundred million is not a hundred databases. It is one writer doing as little as possible, so it has room left when a week like ImageGen arrives, and copies for everything else. Scale the reads, and spare the writer.