Survives any one VPS
A load balancer, an API server, a database member: any single machine can stop while the apps keep working.
Level 4 · ResilientArchitecture 10 of 13
High availability
A spare for every tier: two Caddy load balancers on a floating IP, three API servers and replicated databases.
Any single VPS can stop and the apps keep working. The public IP floats between two Caddy load balancers, health checks steer around failed servers, and every database has a copy ready to take over. Upgrades roll through the servers one at a time.
See it live
Every request is drawn as it travels. Slow it down, pause, go step by step, or open the full canvas and zoom in.
api-maker/architectures/high-availabilityLive- Floating IP
- lb-1on
- Servers
- 3/3in the pool
- Database
- db-1primary
- Capacity
- ~8,277req/s
Two of everything, three where a vote is needed. A floating IP on two Caddy load balancers, three API servers, a replica set of three and a database with a standby.
How it works
The same steps as the diagram, in more detail.
Two of everything, three where a vote is needed.
Every tier has a spare. The public IP floats between two load balancers running Caddy with the same certificates, three API servers share the traffic, Redis runs as a cluster, API Maker DB runs on a replica set of three and your database has a standby. Any single VPS can stop, and the apps are still served.
The active load balancer fails, the standby takes the IP.
Both load balancers run Caddy and keepalived. The active one holds the floating IP and sends heartbeats to the standby over VRRP. When they stop, the standby claims the IP, through the API of your VPS provider where needed, and the apps reconnect to the same address. Its Caddy already holds the certificates, kept in a storage both share, so HTTPS works from the first request. DNS never changes.
A server fails, the others take its share.
Caddy, on both load balancers, calls GET /ping on every server every few seconds. When server 2 stops answering, Caddy takes it out of the pool: the requests it was answering fail, new ones go to the other servers. Its lock of the schedulers expires and server 1 takes them over, and its WebSocket clients reconnect through Caddy.
The database fails, its standby is promoted.
Your database streams every change to a standby, and a failover manager such as Patroni or repmgr watches the primary. When it stops, the standby is promoted and the virtual IP moves to it. API Maker opens new connections to the same address: only the queries running at the moment of the failure return an error.
Upgrade API Maker without downtime.
Upgrade one server at a time: take it out of the upstreams of Caddy on both load balancers, run the installer with the new version, which keeps the existing .env and has systemd start it, and put it back once GET /ping answers. The other servers carry the traffic meanwhile, and the definitions stay in API Maker DB: nothing to redeploy.
Why choose this architecture
One address, always
The floating IP moves to the standby load balancer within seconds. Apps and DNS never see a change.
Health checks built in
Every API Maker server answers GET /ping, which tells Caddy on the load balancers where to send traffic, and where not to.
Schedulers never stop
The server holding the scheduler lock can fail: another one takes the lock and the jobs run on time.
Upgrades in office hours
Roll a new API Maker version through the servers one at a time, without a maintenance window.
Every database has a copy
API Maker DB elects a new primary by itself, and your database fails over to its standby.
Server configuration
Plain VPS from any provider, or your own servers, with Ubuntu 22.04 LTS. Sizes are a starting point: measure and adjust.
Caddy load balancers
×2: active, standby- CPU
- 2 vCPU
- Memory
- 2 GB
- Storage
- 20 GB SSD
Runs
- Caddy: HTTPS, WSS and health checks, the same certificates on both
- keepalived: the floating IP
Ports
- 443 and 80: on the floating IP
- VRRP: between the two only
API servers
×3 or more- CPU
- 4 – 8 vCPU
- Memory
- 8 – 16 GB
- Storage
- 40 GB SSD
Runs
- API Maker, run by systemd, the same .env on every server
- The admin panel on each of them
Ports
- 38246 and 38245: from the load balancers only
Data servers
×3- CPU
- 4 vCPU
- Memory
- 16 GB
- Storage
- 200 GB NVMe
Runs
- A MongoDB member of rs0: API Maker DB and logs
- A Redis master, and the replica of another master
Ports
- 27017, 6379 – 6380, 16379 – 16380: from the API servers and each other
Three servers hold both clusters: when one stops, its Redis master has a replica on another server.
Database servers
×2: primary, standby- CPU
- 8 vCPU
- Memory
- 32 GB
- Storage
- 500 GB NVMe
Runs
- PostgreSQL with streaming replication
- Patroni or repmgr, and a virtual IP
Ports
- 5432 on the virtual IP: from the API servers only
api.example.com { reverse_proxy 10.0.1.11:38246 10.0.1.12:38246 10.0.1.13:38246 { lb_policy least_conn lb_try_duration 5s # a failed server: the request goes to another one health_uri /ping health_interval 2s }}ws.example.com { reverse_proxy 10.0.1.11:38245 10.0.1.12:38245 10.0.1.13:38245 { lb_policy least_conn health_uri /ping health_port 38246 # /ping answers on the HTTP port health_interval 2s }}The same Caddyfile on lb-1 and lb-2. Both need the certificates: give them a shared storage, or the DNS challenge of your DNS provider, so the standby serves HTTPS the moment it takes the floating IP.
vrrp_script caddy { script "pidof caddy" interval 2}vrrp_instance api { state MASTER # BACKUP on lb-2 priority 150 # 100 on lb-2 interface eth1 virtual_router_id 51 advert_int 1 unicast_src_ip 10.0.0.2 # lb-1 unicast_peer { 10.0.0.3 # lb-2 } virtual_ipaddress { 203.0.113.10 } track_script { caddy } notify_master /etc/keepalived/claim-ip.sh # asks your provider to route the IP here}VPS networks rarely carry multicast, hence the unicast peers.
# 1. On both load balancers: remove api-1 from the upstreams in the Caddyfile, thencaddy reload --config /etc/caddy/Caddyfile# 2. On api-1: install the new version, the existing .env is kept, systemd starts itcurl -fsSL https://apimaker.dev/v1/install.sh > install.sh && bash install.sh --version=3.2.1# 3. Once it answers, put it back in the Caddyfile and reload. Then api-2, then api-3curl -fs http://10.0.1.11:38246/pingcaddy reload --config /etc/caddy/Caddyfilecaddy reload applies the new Caddyfile without downtime.
Good to know
- Twice the machines of a simple setup, to patch and to watch.
- A floating IP moves within one location of one provider. To survive the loss of a data center, see Multi-cloud.
- Requests running on a server or database at the moment it fails return an error: let clients retry safe requests.
- Test your failovers on purpose, before a real failure tests them for you.
Questions
Why a floating IP and not DNS with two addresses?
A floating IP moves within seconds and every client follows it at once. DNS answers are cached by resolvers and devices, often for minutes, so users keep reaching a failed address. DNS is the right tool between regions, as the Geo-routing architecture shows, with a setup like this one in each region.
Does my VPS provider support floating IPs?
Most do, under names like floating, reserved, failover or additional IP. keepalived decides which load balancer holds it, and a small script calls the API of the provider when the IP has to be routed to the other one.
What happens to WebSocket clients when a server fails?
Their connections close. The clients reconnect through Caddy, land on another server and subscribe again. Plan for events missed while reconnecting, for example by reloading the data after a reconnect.