Capacity that adds up
Each 8-core server adds about 4,500 requests per second in our benchmark. Add or remove one in minutes.
Level 3 · ScaleArchitecture 05 of 13
Load-balanced API servers
Several identical API Maker servers behind Caddy, which balances the load, sharing one data tier.
Add capacity by adding servers. Every server runs API Maker with the same .env, so any of them answers any request. Schedulers run once, cache and events are shared through Redis, and one git pull updates them all.
See it live
Every request is drawn as it travels. Slow it down, pause, go step by step, or open the full canvas and zoom in.
api-maker/architectures/load-balancedLive- Servers
- 3/3up
- Workers
- 24CPU cores
- Capacity
- ~13,587req/s
- Cache hits
- 30%of reads
Any server answers any request. Caddy sends each request to the least busy server. They share the same data, so no sticky sessions are needed.
How it works
The same steps as the diagram, in more detail.
Any server answers any request.
Caddy runs on a small VPS of its own: it holds the certificates of api.example.com and ws.example.com and passes plain HTTP and WebSocket traffic, over the private network, to the least busy server. Every API Maker server has the same .env: the same API Maker DB, the same Redis, the same secrets. They all serve every API, and tokens signed by one are valid on the others, so no sticky sessions are needed. Three 8-core servers answer about 13,500 requests per second in our benchmark.
Cached once, served by every server.
Cached responses live in the shared Redis, not in one server: a response cached while server 1 answered is served to the next caller by whichever server gets the request. A write through any server resets the cache of the table for everyone, so no server returns stale data.
Schedulers run once, not once per server.
Each scheduler runs on one server of the fleet only: the servers take it through a lock in Redis. If that server stops, its lock expires and another server takes the schedulers over, without any configuration. Jobs run once, never twice, never zero times.
One git pull, every server updated.
A git pull on any server writes the new definitions into the shared API Maker DB. It then publishes a cluster event through Redis: every worker of every server reloads the changes and recycles its sandboxes. There is nothing to deploy server by server.
Add a server, get more capacity.
To grow, install API Maker on one more VPS with the same .env: systemd runs it and starts it at every boot. Add its private address to the upstreams of Caddy and reload Caddy, without downtime. The new server reads the same definitions and connects to the same Redis, so it serves every API from its first request. Remove a server the same way: the others take its share.
Restart servers one by one, without downtime.
The Analytics dashboard of the admin panel shows every server and worker of the fleet, with their traffic, latency, memory and CPU, and restarts any of them remotely. Restart servers one at a time: the health checks of Caddy on GET /ping take each one out of its pool until it is back, and the other servers keep serving the apps.
Why choose this architecture
No sticky sessions
Tokens, the shared Redis and the shared API Maker DB let any server answer any user and any request.
Schedulers run once
A lock in Redis gives each scheduler to one server of the fleet, and another one takes over if it stops.
One cache for all
Responses cached through one server are served by all of them, and a write resets them everywhere.
Deploy once
A git pull on any server reaches every worker of every server through a cluster event.
One view of the fleet
The Analytics dashboard shows traffic, latency and health of every server and worker, and restarts them remotely.
Server configuration
Plain VPS from any provider, or your own servers, with Ubuntu 22.04 LTS. Sizes are a starting point: measure and adjust.
Caddy load balancer
×1- CPU
- 2 vCPU
- Memory
- 2 GB
- Storage
- 20 GB SSD
Runs
- Caddy: HTTPS and WSS, certificates from Let's Encrypt
- Load balancing by least connections
- Health checks of the API servers on GET /ping
Ports
- 443 and 80: from the internet
- to 38246 and 38245 of the servers, over the private network
API servers
×3 or more- CPU
- 8 vCPU
- Memory
- 16 GB
- Storage
- 40 GB SSD
Runs
- API Maker, run by systemd, cpuCount AUTO: 8 workers
- Docker: a sandbox for every worker
- The admin panel on one of them, or all
Ports
- 38246 and 38245: from Caddy only
MongoDB server
×1- CPU
- 4 vCPU
- Memory
- 8 GB
- Storage
- 100 GB NVMe
Runs
- API Maker DB and logs, replica set rs0
Ports
- 27017: from the API servers only
Redis server
×1- CPU
- 2 vCPU
- Memory
- 8 GB
- Storage
- 20 GB SSD
Runs
- Redis internal: events, locks, WebSocket subscriptions
- Redis cache: responses of your APIs
Ports
- 6379 and 6390: from the API servers only
Database servers
1 per database- CPU
- 8 vCPU
- Memory
- 32 GB
- Storage
- 200 GB+ NVMe
Runs
- Your databases: any of the 8 supported types
Ports
- From the API servers only
am__serverName="api-2" # the only line that differs, shown in the dashboardsam__cpuCount="AUTO"# The same on every serveram__passJWT="***"am__passDBEncryptDecrypt="***"am__passCommunication="***"am__mongo_db_connection="mongodb://api_maker:***@10.0.0.20:27017/api_maker_db?authSource=admin&replicaSet=rs0"am__redisInternal='{"nodes": [{host: "10.0.0.30", port: 6379, pass: "***"}]}'am__redisExternal='{"nodes": [{host: "10.0.0.30", port: 6390, pass: "***"}]}'api.example.com { reverse_proxy 10.0.0.11:38246 10.0.0.12:38246 10.0.0.13:38246 { lb_policy least_conn health_uri /ping health_interval 5s }}ws.example.com { reverse_proxy 10.0.0.11:38245 10.0.0.12:38245 10.0.0.13:38245 { lb_policy least_conn health_uri /ping health_port 38246 # /ping answers on the HTTP port health_interval 5s }}Caddy gets and renews the certificates of both domains, and passes WebSockets through as they are. On a small fleet, the Caddy of the first API server can do this job: list the other servers on its reverse_proxy lines, as the install script explains.
Good to know
- Caddy and the data servers are still single: the High availability architecture doubles them.
- Every server needs the same .env. Give each one its own serverName so it is easy to spot in the dashboards.
- A restarted server drops the requests it was answering: restart one server at a time.
Questions
Do I need sticky sessions?
No. Authentication uses tokens, the cache and the WebSocket subscriptions live in the shared Redis, and the definitions in the shared API Maker DB, so any server can answer any request.
How do WebSockets work behind Caddy?
Caddy takes the secure WebSocket and passes it, as a plain WebSocket, to port 38245 of one server, where it stays. Events raised on any other server reach it through Redis, and API Maker keeps quiet connections alive with pings.
Why Caddy as the load balancer?
It is the same Caddy the install script puts in front of API Maker, which has no TLS of its own. It gets and renews the certificates by itself, passes WebSockets through without extra settings, balances by least connections and takes a failed server out of the pool with health checks on GET /ping.
How many servers can I add?
As many as your data tier can serve. Each server opens its own connections to MongoDB, Redis and your databases, so size those for the whole fleet, or make them clusters as the Resilient architectures show.