How Shopify's Servers Take 489 Million Requests a Minute

0:00Ten Shoppers: one Rails app, one MySQL database

How would you build a system that survives a sudden burst? Planning gets you a system that scales, but it does not get you one that survives ten thousand shoppers pressing buy in the same minute, in one store. That was one of the challenges Shopify had to tackle. Last Black Friday, at the peak, Shopify's servers took four hundred eighty-nine million requests in a single minute. Every one of those was a phone, a laptop or an app asking Shopify for something. Here is how Shopify built for that. Do not start there, though. Start with one store, ten shoppers, and one box. Everything lives on that box: a Rails app that builds the pages, and a MySQL database that holds the records. Rails and MySQL are what Shopify runs, and they are what Shopify started on. Now watch what one order costs, slowly, because the whole video rests on it. A shopper opens a product page, and the app asks the database for the product, the price and the stock. Three reads out, three answers back, and then the shopper presses buy. The app opens one checkout record in the database. From here on it changes that same record at every step: the address, the shipping choice, the payment, the confirmation. So one order is a few reads first, then a chain of writes to that one record. Keep that shape in mind, because every wave after this is that same order, placed by more people at once. Now watch ten shoppers order at once. Ten small bursts of reads, ten records rewritten a few times each, and the box answers every one before anyone notices a wait. This is the whole system, and for one store it is enough. But Shopify is not one store. Shopify is millions of merchants, and they all share the same app. Before we can talk about the burst, the system has to grow the ordinary way. The next wave is not more shoppers, it is more stores.

2:30A Thousand Stores: load balancer, memcached, Redis

A thousand stores now run on the box. Every store's products sit in the same database, and every shopper's order is the same three reads and the same chain of writes. The first crack is the product page. Every shopper who opens one makes the app ask the database for the product, the price and the stock. Thousands of shoppers an hour, the same three questions, and one box answering every shopper. Pages slow down. Look at the two things working on the box: the app and the database. Split the app from the database, so the database keeps its server and the app gets its own. Now the app can be copied. Put a second server running the same Rails app beside the first, then a third, and these are the app servers. Something has to decide which app server a shopper goes to, and that is the load balancer. It sits in front and hands every request to whichever app server is free, and the shopper never sees it. Pages are fast again, but every product page still ends with the same three questions to the one database. The answers rarely change, because a price stays the same until the store owner edits it. So keep the answers close to the app, in memory, and Shopify uses memcached for this. The first shopper's questions go to the database, memcached keeps the answers, and the next thousand shoppers are answered by memcached. The database never hears from those shoppers. One more thing slows the box. Some work does not have to happen while the shopper waits. The app writes a note of the job to a queue, sends the page, and the job gets done afterwards, in the background. Shopify keeps that queue in Redis. Reads are solved, and that was the cheap half of scaling. Then the shoppers press buy.

4:43The Write Wall: MySQL sharding

Pressing buy is different. The app opens a checkout record and rewrites it: address, shipping, payment, confirmation, and each of those is a write. A write cannot be answered from memory. The checkout record is the truth of the order, and the truth lives in one place, the database. So watch where every write goes: from every store, through every app server, onto that one database server. Copying the app helped the reads. It does nothing for the writes, because more app servers only means more writes arriving at the same database. That server has a limit: so many writes a second, and no more. Past the limit, checkouts queue, and then checkouts fail. The first fix is the obvious one. Buy a bigger database server: more processors, more memory, faster disks. When it fills, buy a bigger one again, and Shopify did this too, until 2015. In 2015, Shopify reached the end of that road. There was no larger database server left to buy. The only move left is to split the database. The first slice of stores goes on one database server, the next slice on another, and so on. Each piece is called a shard, and a shopper's order goes to the shard that holds the store. Now writes spread across many servers, the way reads spread across app servers, and Shopify grew again. Then the cost showed up. Some jobs in Shopify touch every store, and the code walked every shard in turn. If any one shard went down, that whole job stopped for the whole platform. What Shopify gained in performance, it lost in resilience. One shard's failure was everyone's failure. Shopify's fix for that is the pod.

6:56Pods: one database per pod, twin data centres

Sharding split the writes, but it left one habit in the code: jobs that loop over every shard, one after another. If one shard is down, that job fails for every store on the platform. Shopify solved this by drawing a hard wall around each group of stores and its databases, so no job can ever touch more than one group. Shopify calls that group a pod. A pod is a set of stores with its own copy of all three, shared with nobody: its own MySQL, its own memcached, its own Redis. The rule that makes it work: nothing reaches across pods. The app servers and the job machines are shared, but any one job talks to one pod at a time. So the job that used to loop over every shard now runs inside one pod. If that pod's database is down, that pod's stores wait, and every other pod keeps running. A shopper's request still has to find its pod. Shopify put a program inside its load balancers called Sorting Hat. It matches the request against a list of rules, adds a tag naming the pod, and the app server reads that tag to pick which pod's database to use. One failure is now one pod's problem. Pods fill unevenly, though. Some stores grow, and the pod a big store sits in gets hot. Shopify measured it: the busiest pod was carrying almost four times the load of the quietest. A store has to move to a quieter pod, while it is open. Shopify built a mover. Watch the mover go through every table in the old pod, pick the records that belong to that one store, and write those records to the new pod. While the mover copies, it watches for writes that land mid-copy and carries those over too. When the copy is caught up, the mover waits for the requests already reading that store to finish. Then it blocks new writes to the store for a few seconds while the last changes drain across. Then it updates the rules Sorting Hat reads, and Sorting Hat sends the next shopper to the new pod. After the moves, the busiest pod carried only about twice the quietest. One more thing can fail: the building. Shopify gives every pod a pair of data centres, one active and one a recovery site. A second tool, the Pod Mover, can move a whole pod to its twin in a minute without dropping a request or a job. Today the whole thing runs across three cloud regions, three groups of data centres. Look at the finished machine: balancer, app servers, pods, twin sites. This system serves millions of merchants. It is a decent system that scales, built with planning. Now test it for a sudden burst.

10:20The Burst: a flash sale hits one shard

The machine so far survives growth. Growth arrives slowly, store by store, and every fix had months to land. A burst arrives in a minute. Take the finished machine and run a sale through it. One store opens a flash sale, and ten thousand shoppers press buy in the same minute. Walk the order again and watch where it lands. Every one of those shoppers opens a checkout record and starts rewriting it: address, shipping, payment. Thousands of chains of writes, all at once. Every one of those chains lands in the same place, because the store lives on one pod, and the pod has one database. The load balancer cannot help. It can spread shoppers across app servers, but every app server writes to the same pod's database. Memcached cannot help either, because a write is not a question with an answer kept in memory. The database fills with writes and slows. Every shopper in the sale gets a slow page. Then the database stops answering, and every shopper gets an error. The sale is lost in the minute it opened. This is not a thought experiment. In February 2016, Kylie Cosmetics opened a sale on Shopify, and the rush crashed not just her store, but every other store on her database shard. Today a pod keeps the neighbours safe. It does nothing for the store having the sale. The pod's wall keeps a fire in one room; it does not put the fire out. The system built with planning survives millions of stores, and one store's sale takes it down. Shopify's fix for this does not touch the app or the database at all. It sits in front of both.

12:27The Doorman: leaky-bucket throttle on nginx

The fix sits in the load balancer, before any request reaches an app server. Shopify calls this front layer its edge: the first Shopify machines a request touches. The balancers run a program called nginx. An add-on called OpenResty lets engineers write small programs for nginx in a language called Lua, and those programs run on every request. The fix is one of those programs, a throttle that limits how many shoppers get through, and the Rails app never changed. The throttle is a leaky bucket algorithm. Picture a bucket with room for a set number of shoppers. Each shopper who gets in takes one space, and when the bucket is full, the next shopper is turned away. Every five seconds the bucket is emptied, and a new batch gets in. That is the pace the pod can hold. A shopper who does not fit is not dropped. The balancer sends the shopper to a queue page that says the shopper is in line to reach the checkout. The page keeps asking the balancer whether there is room yet. A shopper who gets through is given a securely signed cookie, a small file in the browser. That cookie skips the throttle for the rest of the visit, so a shopper is never thrown back into the line halfway through paying. Shopify calls this back pressure: when the app cannot take more, it tells the shopper, kindly, to hold off. Run the sale again, with ten thousand pressing buy. Watch the bucket let a batch through every five seconds, while the pod's database takes writes at a rate it can hold. The rest wait on the page and drain in behind, and nobody gets an error. The first version had a flaw, though. The queue page told each shopper they were in line, but there was no line. Every page just asked the balancer again after a random wait, on its own. When a space opened, whichever page happened to ask next got it. So a shopper who arrived first could keep losing to shoppers who arrived much later. Some shoppers waited up to forty minutes, for a sale that lasted forty minutes. The fix was fairness, without a database keeping a list. The time a shopper first tried rides inside the signed cookie. The balancer looks at every arrival time it saw in the last five seconds, and sets a cut-off from the room it has. While the bucket has room, everyone goes through. When the bucket fills, the shoppers who arrived earliest go first. After that shipped, Shopify had no reports of long queues or unfair checkouts. Watch the sale flow through the bucket in order, earliest first, and the pod stays steady. The store survives its own success, and the line is fair. That is the burst handled at one store, and Black Friday is next.

15:51The Rehearsal: load testing at 150 %

Black Friday is the flash sale at every store at once, and Shopify does not wait for the day to find out whether the machine holds. Preparation starts in the spring, planning how many machines each cloud region needs. From early spring, Shopify runs Game Days. Engineers break the system on purpose: cutting network links, adding delay, clearing saved answers, using Toxiproxy, a tool Shopify built to fake a broken network. Then the engineers watch how the system answers. Through the year, Shopify replays last year's Black Friday at one hundred fifty percent of its load, on the same machines shoppers use, at night. A load tool plays scripted shoppers: browsing, adding to cart, checking out. On top of that steady load, it fires flash-sale bursts. From April to October there were five of these tests. The early tests broke checkout: core steps failed with errors, and the checkout queues backed up. Every test found a weak spot, and each was fixed and tested again. A later test reached one hundred forty-six million requests a minute. It also pushed more than eighty thousand checkouts a minute. It hit limits that triggered an unplanned failover, with traffic moved out of a region, and it showed delays bringing regions back online. That is what a test is for. The last test ran at two hundred million requests a minute, the top of the forecast. The real day should stay under that level ninety-nine times out of a hundred. It was the only test run during business hours, to feel like the real day. By November the failure has already happened, on purpose, five times. On the weekend itself, engineers watch every region on dashboards, an incident manager is on call day and night, and the system is tuned as the traffic comes in. Then Friday comes.

18:1312:01 PM: 489 million requests a minute

Black Friday, twelve oh one pm Eastern, the busiest minute of the weekend. In that minute, shoppers paid five point one million dollars. And at its peak, the traffic reached four hundred eighty-nine million requests in a single minute. Watch one order go through the finished machine. A shopper opens a product page, and the request reaches the edge first, the balancers at the front, then an app server. At the peak the edge took those four hundred eighty-nine million requests a minute, and the app servers behind it took one hundred seventeen million. The shopper presses buy. Sorting Hat stamps the request with the store's pod, and the app server writes the checkout record on that pod alone. Other shoppers in the same store press buy too, and the bucket lets those shoppers in at the pod's pace, earliest first, cookie in hand. If a pod runs hot, a store can move house while it sells. If a region blinks, its pods hop to the twin sites. Every one of those moves was rehearsed this year. Over the weekend, the databases answered trillions of questions and took trillions of writes, and eighty-one million people bought something. It started as one store on one box, and it still is. One app, copied into pods, behind one bucket at the front.