Scale Out a Backend: Many API Maker Servers Behind One Load Balancer

Intro

0:00 Many API Maker servers behind one address. A second server shares the calls, the WebSocket clients, the schedulers and the cache with the first one : no sticky sessions, no cron server, no message broker to run.

More servers, the usual way

0:14 Scaling out a backend usually means sticky sessions, so that each user stays on the server which knows them, one server for the cron jobs, so that no job runs twice, a message broker, so that the other servers hear of a change, and WebSocket clients which hear only the events of their own server. With API Maker, any server answers any call, each scheduler runs on one server of the cluster, a save reaches every server through Redis, and every client is told, whichever server holds it.

0:45 In this video : a second server, one address in front, a save on every server, WebSocket clients on both, the schedulers, a server which stops, the cache, and upgrades, and what to know.

A second server

0:58 The autumn sale of Calm Home is coming : more calls than one server should take. A second server joins : the same API Maker, with the same .env : the same secrets, the same MongoDB and the same Redis. Only its name and its address are its own. That is all a cluster of API Maker is : servers which share MongoDB and Redis. On a new machine, the installer does it with one answer : secondary.

1:27 On the Servers page, both servers are listed : video1 and video2, each with its threads. video2 : its own server id, started seconds ago, on the same version. The page asks every server through Redis, and each one says what it is.

One address in front

1:45 In front of them, one address : Caddy, with both servers. Each call goes to the next server. A call which finds its server down goes to the other one, with its body, and /ping says which servers answer. A token of the API user, through that address, then four calls of the custom API Sale Banner : with x-am-meta, each answer names the server which ran its code : one, then the other.

2:10 The token came from one server, and both take it. No sticky sessions. In Analytics, the Live tab, while the shop calls Sale Banner eight times a second : two servers, both responding. Each one takes its half of the calls. The page asks every server, wherever it is served from.

A save on every server

2:31 The sale grows, and the banner with it. Sale Banner, its code. Twenty percent, now thirty. Save, on the panel, which video1 serves. The next calls, through the one address : both servers answer with the new banner, at once. The save told every server, through Redis, to drop its copy of the code : no build, no restart, no deploy to each server.

WebSocket clients on both servers

2:57 The app of the shop shows its prices live : it keeps a WebSocket open, subscribed to the price changes of the products. Two clients connect through the one address : Caddy hands one to each server. A price changes, through the one address. Both clients are told, at the same moment : the server which saved publishes on Redis, and each server tells its own clients.

Schedulers, once per cluster

3:21 The schedulers of the shop, every ten seconds. The Server column says which server runs each one : both run some. Each server works out the owner of every scheduler from the same list of servers : they agree without talking. One of video2 : its runs, one every ten seconds, never two, because its server holds a key in Redis while it runs it.

A server stops

3:45 Now video2 stops : an upgrade, a crash, a machine which goes away. Its client is told that the server is going away, and connects again, through the one address : video1 takes it. The calls go on : video1 answers all of them. And the next price reaches both clients. The schedulers of video2 run on video1 within seconds : a server which stops gives its schedulers back, and those of a server which is killed are taken over once its keys run out, within twenty seconds.

4:17 When video2 comes back, it takes its share again.

One cache

4:21 The cache is shared too : it is in Redis. The lamp, read on video1 : from the database. The same read on video2 : from the cache. Its price changes on video2 : the cached answers of the table are dropped, for every server. Read on video1 again : from the database, with the new price.

Upgrades

4:42 Upgrades. Launchpad switches API Maker one server at a time, while its gateway sends the calls to the others. The Deploy API Maker page upgrades every server from one page, each one checked, and rolled back on its own : video forty seven. And your code needs no upgrade at all : a save is live on every server.

Good to know

5:04 Good to know. A server full of WebSocket clients answers one more with 503 and Retry-After : it keeps connections for the calls of its APIs. Count the servers by memory, a few kilobytes a client, and keep one spare for the clients of a server which stops. Smaller servers lose fewer clients at once : in the benchmarks, one of four which stopped dropped a quarter of them, all back in under a minute.

5:31 And work done by one server, like the scan of Index Maker each minute, runs on one of them, through Redis.

Recap

5:39 To sum up. Servers which share MongoDB, Redis and their secrets are one cluster. One address in front : any server answers any call, no sticky sessions. A save, a WebSocket event and the cache reach every server. Each scheduler runs on one server, and moves when it stops. A client of a server which stops comes back on another. Next : clusters drawn and deployed by Launchpad. API Maker.

6:09 Build faster, spend smarter, deliver better.