Inside Netflix's Live Streaming System: One Fight, 65 Million Screens

0:00One Viewer: video in pieces over HTTPS

On November fifteenth, twenty twenty-four, Mike Tyson walks into the ring, and sixty-five million screens are watching Netflix at the same moment. How would you build a system that streams live, and still works when millions of people watch the same stream at the same moment? Here is how Netflix built it. We do not start there, though. We start with one viewer, one film, and one server. The film is a file on that server's disk. Netflix keeps every film in several sizes, a small one for a phone and a big one for a 4K television. Now watch what one press of play does, slowly, because the whole video rests on it. You press play, and your screen asks the server for the film. The server does not send the whole film at once. It sends the film in pieces, and each piece is a few seconds of video. Your screen plays one piece while the next one is on its way. The pieces travel over HTTPS, the same way an ordinary web page reaches your browser. So one press of play is a screen asking a server for short pieces, one after another. Keep that picture in mind, because everything after this is that same press of play, by more people. Now ten viewers press play at once. Ten screens ask, ten rows of pieces go out, and the server answers every one without a wait. This is the whole system, and for ten viewers it is enough.

1:49A Million Viewers: the Open Connect CDN

Now a million screens press play. One server cannot push that many pieces, so most of those screens would wait. And the problem is bigger than a million viewers pressing play. The second problem is where those viewers are. The viewers are spread all over the globe, and every piece has to cross the world to reach each one. So how does Netflix solve this? Netflix puts the server near you. Since twenty eleven, Netflix has built its own delivery network, called Open Connect. Today Open Connect is more than eighteen thousand servers, at more than six thousand locations around the world. Many of those servers sit inside the network of your internet provider, the company you pay for your connection. Netflix gives the provider the server for free, and the provider installs it inside its own network. More than a thousand providers have done this. So when you press play, your pieces come from a server that is already inside your provider's network. Look at what that changes: the pieces no longer cross the world, because the server is close to you. But a server near you can only send a film it already has. Here is how the film gets there. Netflix can predict what people near each server will watch tomorrow, and at what time of day. So every night, in the quiet hours, Netflix copies those films onto the servers. When you press play in the evening, the film has been sitting on the server near you since the night before. The film reached your neighbourhood before you asked for it. One question is still open, though: which of eighteen thousand servers does your screen ask?

3:48Who Picks the Server: the steering service

The servers only keep films and send pieces, and the servers do not decide anything. The deciding happens somewhere else, in Netflix's cloud. Netflix's cloud is the Netflix software that runs in Amazon's data centres. Every server reports up to the cloud. Each server says whether it is healthy, and which films it is holding. Now walk through the press of play again. You press play, and your screen does not ask a server first; your screen asks Netflix's cloud. Netflix runs a service in its cloud for exactly this job, called the steering service. The steering service looks at what every server reported. It picks healthy servers near you that have your film, and it sends the addresses of those servers back to your screen. Now the cloud's part is done. Your screen takes an address and asks that nearby server directly for the film. The pieces come from that server, close to you. So here there are two machines doing two jobs. The cloud decides, and the server sends. Now run the million viewers through it. A million screens ask the cloud, and a million answers come back. Then each screen gets its pieces from a server nearby. This is ordinary Netflix, where every film was already on a server near you before you pressed play. All of this works because the film was copied the night before. A live event cannot be copied the night before, because it has not happened yet.

5:38Live: encoder, packager, origin

In a live event, the picture is being made right now, in a boxing ring. The piece your screen wants next has not been made yet. So first, watch how a piece of the fight gets made. The cameras in the stadium send the picture to Netflix's cloud. There, a program called an encoder makes copies of the picture in every size, from a phone's to a 4K television's. Then a program called a packager cuts each copy into pieces of exactly two seconds. These are the same kind of pieces the ordinary system already sends. Netflix also kept HTTPS for live. Faster ways to send live video exist, but HTTPS works on every device Netflix already runs on. So the fight rides the same servers as the films. Now follow one new piece of the fight. The packager puts each new piece in one place in Netflix's cloud, and Netflix calls that place the origin. For a moment, the origin is the only machine in the world that has the piece. Now you press play on the fight. Your screen asks the server near you for the newest piece, and that server does not have it, because nobody copied it there overnight. So the server has to go and get it. It asks a server above it, and that server asks a server above it. At the top are a few dozen top servers, and only those are allowed to ask the origin. Then the piece comes back down the same chain: from the origin to a top server, to your server, to your screen. Why not let all eighteen thousand servers ask the origin directly? Because the origin would be hit eighteen thousand times for every piece. This way, the origin is asked only a few dozen times. Here is the difference from a film. A film sits on the server near you for hours before you press play. With the fight, a piece is made, and two seconds later sixty-five million screens are asking for it. Every one of those pieces has to come from one place, the origin.

8:11The Origin: from S3 to Live Origin

Netflix's first origin was the simple choice. It was Amazon's storage service, S3, the same place Netflix keeps its films. The packager saved each new piece there, and the top servers fetched it from there. For small early events, this worked. Then the events got bigger, and a difference between a film and a live piece started to matter. A film is saved long before anyone watches it. If saving that film is slow, or fails and has to be tried again, nobody notices. A live piece has no such slack. Screens need it about two seconds after it is made. If saving that piece is slow even once, the screens have nothing to play, and the picture freezes. S3 did not break, but S3 had been doing a different job, holding films that are saved once and then sit still. A live event needs a new file every two seconds, and that file can never be late. So Netflix built its own origin, called Live Origin. Look inside it, because it has two parts. The first part is a database called Apache Cassandra, which saves each piece quickly and safely. The second part is a memory cache, which keeps a copy of each piece ready to hand out. Saving goes to the database, and handing out comes from the memory. So when the top servers all ask for one piece at once, nearly every request is answered from the memory. The database is left free to save the next piece on time. Netflix also made the asking simple. Every piece has a number, and every screen and every server is told at the start how to work out, from the clock, which number is due right now. So nobody has to ask what the newest piece is. Screens always want the newest piece of the fight, so sometimes a request arrives a split second too early. A top server asks for the next piece, and the next piece is still being saved. Ordinary storage would answer, I don't have it. That answer goes all the way back down the chain to your screen. Your screen then has to ask again, and the whole trip is repeated. Live Origin does not answer, I don't have it. It knows the next piece is due any moment, so it holds the request and waits. The instant the piece is saved, Live Origin sends it. One request, one answer, and no wasted trip. So the piece is saved once at the origin. The top servers get it the instant it exists, and pass it down the chain to your screen. That handles the servers, but it does not yet handle the viewers, who all press play in the same minute.

11:26The Same Minute: retry storms and load shedding

With a film, people start watching at different times. One viewer presses play early in the evening, and another presses play late at night. The presses reach Netflix's cloud a few at a time. A fight has a start time. So tens of millions of people press play inside the same minute. Every one of those presses goes to Netflix's cloud first, just as it did in chapter three. But the start is not the worst wave. The worst wave comes when something goes wrong. When the picture freezes, people press play again. Netflix measured this: a freeze as short as thirty seconds brings ten times the normal traffic, from viewers restarting the stream. So a small problem creates a much bigger wave of its own, and Netflix has three answers. First, Netflix spreads the retries out, so when a screen has to try again, Netflix's server tells it how long to wait first. That way the retries do not all land together. Second, look at the gateway, which is the front entrance of Netflix's cloud, where every request arrives first. When a spike hits, the gateway turns away the least important requests, such as loading things in advance. That lets the important requests through, the ones that keep the video playing. Third, the origin can tell the servers to come back in five seconds. This is the web's standard reply for busy, try later, and its code is five oh three. For those five seconds the servers stop repeating the less important requests, and the origin catches up. Netflix also plans for the crowd. Before each event, Netflix forecasts how many people will watch, and gets enough machines ready in advance. It shares that forecast with the internet providers, so the providers are ready too. And if that is still not enough, Netflix turns off features to save its machines for the video, and here are two examples. First, your home screen normally shows titles picked just for you. Under heavy load, Netflix makes the home screen less personal, which is less work. Second, Netflix also has a feature that remembers the exact minute where you stopped watching, so you can continue later. Under heavy load, Netflix stops saving that. And Netflix can lower the very top picture quality too. A slightly less sharp picture for everyone is better than a frozen picture for some. So Netflix handles the crowd by making the less important requests wait, and by switching off what the video does not need. That way, the fight keeps playing. One weak point is left, and this time the crowd is not the cause. The weak point is the picture itself, on its way from the stadium.

14:46Two of Everything: redundant paths and rehearsal

The picture from the stadium travels along one path. First comes the feed from the cameras. Then comes the encoder, which makes the picture in every size. Then comes the packager, which cuts it into two-second pieces. If any one of those breaks, the picture freezes for everyone. And we just saw what a freeze does: ten times the traffic. So Netflix builds the whole path twice. Two feeds leave the stadium, and two encoders and two packagers run in two separate groups of data centres. Both paths run for the whole event. Watch the two paths side by side. The two paths cut the fight at exactly the same moments, because both encoders follow the same clock. So a piece from path A and the matching piece from path B show the same two seconds, and either one can be used. Live Origin receives pieces from both paths, and for each piece, Live Origin takes the first good copy. If one path breaks, Live Origin simply keeps using the other path. Your screen does nothing and notices nothing. Netflix does not wait for the night to find out whether all this works. Netflix rehearses, and it records the traffic from past events and plays it back against the machine. To copy the rush at the start of an event, Netflix also runs software that behaves like Netflix devices. Between the two, Netflix can press play up to a hundred thousand times a second. The teams at Netflix rehearse too, breaking something on purpose and practising the fix, before a real crowd is watching. On the night, a control centre watches the event. Screens and servers send in up to thirty-eight million reports a second. The people in the room see a problem within seconds, before a small freeze can grow into a wave of retries. By fight night, the failures have already happened, on purpose, in rehearsal.

17:07Fight Night: 65 million screens

It is fight night, November fifteenth, twenty twenty-four. Follow a single piece of the fight through the finished machine, as Netflix describes it today. Tyson steps forward, and two feeds leave the stadium. Two encoders make the picture in every size, and two packagers cut the same two seconds into a piece. Live Origin takes the first good copy, saves it, and keeps a copy in memory. The top servers that were already waiting get the piece at once, and pass it down the chain. Your screen asks the server near you, and gets the piece. So do sixty-five million other screens, at the same moment. Netflix said around one hundred and eight million people watched across the night. The night was not perfect, and news reports said many viewers had buffering and sound problems. Netflix has kept working on this machine since. That Christmas, Netflix streamed two NFL games, and the bigger one averaged about twenty-four million viewers. A year later, the bigger Christmas game averaged more than twenty-seven million. Now look back at where we started. We started with one viewer, one film and one server. The fight works the same way: a screen asks a nearby server for a two-second piece. The only difference is that the piece was made a moment before you asked for it. This Christmas Day, Netflix does it again, with two NFL games.