No single data node
Every Redis slot has a replica and API Maker DB lives on three machines: one failed node stops no API.
The framework for AI era
The framework for AI era
Level 4 · ResilientArchitecture 09 of 13
Redis Cluster, a MongoDB replica set and sharded data: a data tier with no single node to lose.
The data tier becomes clusters. Redis runs as masters with replicas, API Maker DB and its logs live on a replica set of three, and your own data can be sharded. When a node fails, a copy takes over and API Maker follows by itself.
Every request is drawn as it travels. Slow it down, pause, go step by step, or open the full canvas and zoom in.
api-maker/architectures/clustered-dataLiveRedis as a cluster: the keys shared by three masters. List several nodes in redisInternal or redisExternal and API Maker connects in cluster mode. Each key lives on one master.
The same steps as the diagram, in more detail.
When redisInternal or redisExternal list more than one node, API Maker connects to them as a Redis Cluster. The cluster splits its 16,384 hash slots between the masters, so cached responses, locks, auto increments and WebSocket subscriptions are spread over three machines. When memory runs short, add a master and move a share of the slots to it.
Each master streams its writes to a replica running on another VPS: redis-4 copies redis-1, redis-5 copies redis-2, redis-6 copies redis-3. Reads and writes stay on the masters, while the replicas keep a live copy of the same slots, ready to take over.
When a master stops answering for longer than the node timeout of the cluster, the other masters promote its replica. The Redis client of API Maker receives the new map of the slots and sends their keys to the new master. Only the keys of the failed master wait during the switch: the other masters serve theirs all along.
API Maker keeps its definitions and its logs in MongoDB, and needs a replica set anyway for its transactions. With three members on three VPS, every write lands on the primary and is copied to both secondaries. List the three members in the connection string: the driver finds the primary by itself.
When the primary stops, the two remaining members still form a majority and elect a new primary, typically within about twelve seconds with the default settings. The MongoDB driver of API Maker finds the new primary by itself and retries the writes that can be retried. Writes pause only during the election.
Repair or replace the failed machines and start them again. A Redis master that comes back joins as a replica of the node that replaced it, and a MongoDB member catches up on what it missed before it serves again. API Maker needs no restart and no change of settings at any point.
Every Redis slot has a replica and API Maker DB lives on three machines: one failed node stops no API.
Redis promotes a replica, MongoDB elects a new primary, and API Maker follows both without a restart.
Cached responses, locks and WebSocket subscriptions are spread over the masters. Add one when memory runs short.
The replica set API Maker needs for its transactions also keeps live copies of every definition and log.
Point an instance at the mongos routers of a sharded MongoDB, or at any cluster your database offers.
A list of nodes for Redis, a list of hosts for MongoDB. No code, no plugin, the same APIs.
Plain VPS from any provider, or your own servers, with Ubuntu 22.04 LTS. Sizes are a starting point: measure and adjust.
Runs
Ports
Runs
Ports
Runs
Ports
Never put a master and its own replica on the same VPS.
Runs
Ports
Keep an odd number of members, three or five, so a majority remains after a failure.
Runs
Ports
# API Maker DB and logs: the three members of the replica setam__mongo_db_connection="mongodb://api_maker:***@10.0.3.11:27017,10.0.3.12:27017,10.0.3.13:27017/api_maker_db?authSource=admin&replicaSet=rs0"am__logs__mongo_db_connection_logs="mongodb://api_maker:***@10.0.3.11:27017,10.0.3.12:27017,10.0.3.13:27017/api_maker_logs?authSource=admin&replicaSet=rs0"# More than one node: API Maker connects in cluster mode and finds the replicas by itselfam__redisInternal='{"nodes": [{host: "10.0.2.11", port: 6379, pass: "***"}, {host: "10.0.2.12", port: 6379, pass: "***"}, {host: "10.0.2.13", port: 6379, pass: "***"}]}'am__redisExternal='{"nodes": [{host: "10.0.2.11", port: 6379, pass: "***"}, {host: "10.0.2.12", port: 6379, pass: "***"}, {host: "10.0.2.13", port: 6379, pass: "***"}], redisValueExpireInSeconds: 7200}'redisInternal and redisExternal can list the same cluster, or a cluster each.
# redis.conf on each of the 6 nodescluster-enabled yescluster-config-file nodes.confcluster-node-timeout 5000 # ms before a silent master is replacedrequirepass ***masterauth ***# Once, from any node: 3 masters with 1 replica eachredis-cli -a *** --cluster create \ 10.0.2.11:6379 10.0.2.12:6379 10.0.2.13:6379 \ 10.0.2.14:6379 10.0.2.15:6379 10.0.2.16:6379 \ --cluster-replicas 1rs.initiate({ "_id": "rs0", "members": [ { "_id": 0, "host": "10.0.3.11:27017" }, { "_id": 1, "host": "10.0.3.12:27017" }, { "_id": 2, "host": "10.0.3.13:27017" } ]})Start each mongod with replSetName rs0 and a shared keyFile, then run this on one of them.
Run a Redis Cluster and list its nodes in redisInternal and redisExternal. API Maker connects to a single node when one is listed, and to the cluster when there are several. Even three masters with one replica each give every key a copy ready to take over.
They can. On busy setups, give each its own cluster: cached responses in redisExternal may be evicted when memory runs short, while redisInternal holds locks, events, auto increments and WebSocket subscriptions that must stay.
Whatever your database offers behind a connection string: MongoDB replica sets and sharded clusters through mongos, TiDB, Percona XtraDB and MariaDB Galera clusters, or a primary with standbys behind a virtual IP for PostgreSQL, MySQL, SQL Server and Oracle. API Maker generates the same APIs on each of them.
Requests that need the keys of the failed Redis master, and writes during a MongoDB election, wait or fail for the seconds the switch takes. Everything else is served as usual, and API Maker needs no restart before or after.