Schedulers on Every Server and Distributed Locks with Redis

Intro

0:00 Schedulers on every server, and locks. A job can run on every server of your installation, and in every runner of each, sharing the work. And one line of code keeps each piece of that work to one run at a time.

Work shared by hand

0:14 Spreading a job over many machines usually means one cron server, which does all the work, and stops when it stops, a queue service to share the work, with servers of its own, a lock library with expiry, retries and waiting, to wire and to test, and an invoice sent twice, the day two servers run the same job. With API Maker, one choice runs a scheduler on every server, or in every runner, each run knows its share of the work, getLock and releaseLock keep a piece of work to one run, or to a few, in all eight languages, answered by the runner itself.

0:51 In this video : where a scheduler runs, every server, and every runner, the work shared between the runs, getLock and releaseLock, a lock for three at a time, and how the locks stay fast.

Where a scheduler runs

1:04 A scheduler runs on one server : the servers share the schedulers, and when a server goes away, its schedulers move to another. That is right for a job which must happen once. But Calm Home packs its paid orders every ten seconds, and one run on one server is the slowest way to do it. Every server can run it, and every runner of each server, at the same moment.

Every server, every runner

1:30 Schedulers. pack_orders runs on one server, video1. Pick it, and Edit. Runs on : the help behind the question mark says what each choice does. Every server. Runs per server : once on each server, or once in each runner, as many runs as the server has processor cores. Once per runner. Update. The list says Every runner, and the two servers which run it. Open it again : the Schedule panel says it runs on every server, and each run of a moment says its server, and its runner.

2:07 Some ran on video2.

The work shared between runs

2:09 Every paid order was packed, by all the runs of a moment together, some on video1, some on video2, and no order twice. The code says how. g.req.scheduler tells each run where it stands : worker, from zero, among workers, all the runs of the moment, on every server. Each run keeps the orders whose number, divided by workers, leaves its own worker. Then it takes the lock of the order : a run of an earlier moment, still busy with it, holds it.

2:39 And the lock is given back in finally, whatever happens. g.req.scheduler also says firedAt, the moment of the runs, the same on every server, the server of the run, and how many servers run it, and its runner, and how many runs its server started.

getLock and releaseLock

2:58 getLock takes the name of a lock, and how to take it. ttlSeconds : how long a hold lasts at most, so a run which crashed never blocks the others. waitSeconds : how long to wait for it. tries and retryDelayMs : how often to ask. maxHolders : how many may hold it at once. throwIfNotAcquired : an error, rather than an answer. It answers acquired, and a token. releaseLock takes the lock, or its token, and getLock with the token keeps the hold longer.

3:33 Two custom APIs of the shop. Send Invoice, Edit, Code. It takes the lock of the order, and a call which does not get it says so, at once. The lock is given back when the invoice is sent. Close. Three calls at once, for the same order, through the one address of both servers. One call sends the invoice. The others are told another call is sending it, on whichever server they landed.

Three at a time

4:03 Charge Payment, Edit, Code. maxHolders, three : the payment service takes three calls at a time, the others wait up to twenty seconds for a place, then fail with a clear error. Close. Six payments at once. Three hold the lock at the same time. The other three wait, and each takes the place a payment gives back, at once.

How the locks stay fast

4:28 How it stays fast. The locks live in the Redis of the installation, and the agent of API Maker, in the runner of your code, talks to Redis itself : no trip to the process of the server, and the same for every system API of Redis. Taking a lock is one script of Redis : one round trip. A run which waits does not keep asking : a lock given back wakes it at once, on every server.

4:54 And a server without Redis keeps its locks in its own memory. getLock and releaseLock are in all eight languages : TypeScript, Python, PHP, C#, Go, Rust, Java and Kotlin, typed, with the same options, and the editor of the panel knows them.

Good to know

5:12 Good to know. The names of the locks belong to your account : another account never meets them. getLock and releaseLock are for code only : the System API over HTTP does not offer them. A run in every runner opens the runners not open yet, and runs the process initializers first. And with Run Native, a server starts one run.

Recap

5:36 To sum up. A scheduler runs on one server, on every server, or in every runner. g.req.scheduler gives each run its share. getLock and releaseLock keep a piece of work to one run, or to a few. Holds end by themselves, and waits wake at once. And the runner talks to Redis itself. Next : many API Maker servers behind one address. API Maker. Build faster, spend smarter, deliver better.