Capacity that adds up
Each 8-core server adds about 11,000 requests per second in our benchmark. Add or remove one in minutes.
Level 3 · ScaleArchitecture 05 of 13
Load-balanced API servers
Several identical API Maker servers behind Caddy, which balances the load, sharing one data tier.
Add capacity by adding servers. Every server runs API Maker with the same .env, so any of them answers any request. Schedulers run once, cache and events are shared through Redis, and one git pull updates them all.
See it live
Every request is drawn as it travels. Slow it down, pause, go step by step, or open the full canvas and zoom in.
api-maker/architectures/load-balancedLive- Servers
- 3/3up
- Threads
- 24CPU cores
- Capacity
- ~33,006req/s
- Cache hits
- 30%of reads
Any server answers any request. Caddy sends each request to the least busy server. They share the same data, so no sticky sessions are needed.
How it works
The same steps as the diagram, in more detail.
Any server answers any request.
Caddy runs on a small VPS of its own: it holds the certificates of api.example.com and ws.example.com and passes plain HTTP and WebSocket traffic, over the private network, to the least busy server. Every API Maker server has the same .env: the same API Maker DB, the same Redis, the same secrets. They all serve every API, and tokens signed by one are valid on the others, so no sticky sessions are needed. Three 8-core servers answer about 33,000 requests per second in our benchmark.
Cached once, served by every server.
Cached responses live in the shared Redis, not in one server: a response cached while server 1 answered is served to the next caller by whichever server gets the request. A write through any server resets the cache of the table for everyone, so no server returns stale data.
Schedulers run once, not once per server.
Each scheduler runs on one server of the fleet only: that server holds its key in Redis and renews it every few seconds. If the server stops, its schedulers move to the others, within a second when it stops cleanly and in about twenty seconds when it dies, without any configuration. A run happens once, never on two servers.
One git pull, every server updated.
A git pull on any server writes the new definitions into the shared API Maker DB. It then publishes a cluster event through Redis: every server reloads the changes and renews its sandboxes. There is nothing to deploy server by server.
Add a server, get more capacity.
To grow, install API Maker on one more VPS with the same .env: systemd runs it and starts it at every boot. Add its private address to the upstreams of Caddy and reload Caddy, without downtime. The new server reads the same definitions and connects to the same Redis, so it serves every API from its first request. Remove a server the same way: the others take its share.
Restart servers one by one, without downtime.
The Analytics dashboard of the admin panel shows every server of the fleet, with its traffic, latency, memory and CPU, and restarts any of them remotely. Restart servers one at a time: the health checks of Caddy on GET /ping take each one out of its pool until it is back, and the other servers keep serving the apps.
Why choose this architecture
No sticky sessions
Tokens, the shared Redis and the shared API Maker DB let any server answer any user and any request.
Schedulers run once
A key in Redis gives each scheduler to one server of the fleet, and another one takes over if it stops.
One cache for all
Responses cached through one server are served by all of them, and a write resets them everywhere.
Deploy once
A git pull on any server reaches every server of the fleet through a cluster event.
One view of the fleet
The Analytics dashboard shows traffic, latency and health of every server, and restarts them remotely.
On video
Scale Out a Backend: Many API Maker Servers Behind One Load Balancer
Two API Maker servers on the same MongoDB and Redis, behind Caddy: any server answers any call, a save is live on every server at once, WebSocket clients are told whichever server holds them, each scheduler runs once, and a server which stops hands over its clients and schedulers. Shown live, with two servers.
Every video of API Maker
Server configuration
Plain VPS from any provider, or your own servers, with Ubuntu 24.04 LTS, installed by API Maker Launchpad. Sizes are a starting point: measure and adjust.
Caddy load balancer
×1- CPU
- 2 vCPU
- Memory
- 2 GB
- Storage
- 20 GB SSD
Runs
- Caddy: HTTPS and WSS, certificates from Let's Encrypt
- Load balancing by least connections
- Health checks of the API servers on GET /ping
Ports
- 443 and 80: from the internet
- to 38246 of the servers, over the private network
API servers
×3 or more- CPU
- 8 vCPU
- Memory
- 16 GB
- Storage
- 40 GB SSD
Runs
- API Maker, run by systemd: one process, 8 threads on 8 cores
- Docker: the sandboxes of custom code
- The admin panel, served by API Maker on the primary server
Ports
- 38246 (APIs and WebSockets) and 4626 (admin panel): from Caddy only
MongoDB server
×1- CPU
- 4 vCPU
- Memory
- 8 GB
- Storage
- 100 GB NVMe
Runs
- API Maker DB and logs, replica set rs0
Ports
- 27017: from the API servers only
Redis server
×1- CPU
- 2 vCPU
- Memory
- 8 GB
- Storage
- 20 GB SSD
Runs
- Redis internal: events, locks, WebSocket subscriptions
- Redis cache: responses of your APIs
Ports
- 6379 and 6390: from the API servers only
Database servers
1 per database- CPU
- 8 vCPU
- Memory
- 32 GB
- Storage
- 200 GB+ NVMe
Runs
- Your databases: any of the 8 supported types
Ports
- From the API servers only
am__serverName="api-2" # a name per server, shown in the dashboards# The same on every serveram__passJWT="***"am__passDBEncryptDecrypt="***"am__passCommunication="***"am__mongo_db_connection="mongodb://api_maker:***@10.0.0.20:27017/api_maker_db?authSource=admin&replicaSet=rs0"am__redisInternal='{"nodes": [{host: "10.0.0.30", port: 6379, pass: "***"}]}'am__redisExternal='{"nodes": [{host: "10.0.0.30", port: 6390, pass: "***"}]}'The primary server, the first one installed, has two more lines, am__adminPanel__folder and am__adminPanel__port: API Maker serves the admin panel there.
api.example.com { reverse_proxy 10.0.0.11:38246 10.0.0.12:38246 10.0.0.13:38246 { lb_policy least_conn health_uri /ping health_interval 5s }}ws.example.com { # the WebSockets: the same port of API Maker reverse_proxy 10.0.0.11:38246 10.0.0.12:38246 10.0.0.13:38246 { lb_policy least_conn health_uri /ping health_interval 5s }}Caddy gets and renews the certificates of both domains, and passes WebSockets through as they are. On a small fleet, the Caddy of the first API server can do this job: list the other servers on its reverse_proxy lines.
Good to know
- Caddy and the data servers are still single: the High availability architecture doubles them.
- Every server needs the same .env. Give each one its own serverName so it is easy to spot in the dashboards.
- A server that restarts lets the requests it was answering finish, for up to 30 seconds, and closes its WebSockets: restart one server at a time.
Questions
Do I need sticky sessions?
No. Authentication uses tokens, the cache and the WebSocket subscriptions live in the shared Redis, and the definitions in the shared API Maker DB, so any server can answer any request.
Why Caddy as the load balancer?
It is the same Caddy API Maker Launchpad puts in front of API Maker, which has no TLS of its own. It gets and renews the certificates by itself, passes WebSockets through without extra settings, balances by least connections and takes a failed server out of the pool with health checks on GET /ping.
How many servers can I add?
As many as your data tier can serve. Each server opens its own connections to MongoDB, Redis and your databases, so size those for the whole fleet, or make them clusters as the Resilient architectures show.