Automatic Database Backups to S3, Encrypted, Restored Table by Table

Intro

0:00 Automatic Backup. Your databases on schedules, into a bucket of S3 : encrypted, kept by your rules, and brought back table by table.

Backups by script

0:10 Backups are usually a cron job and a script on one server, the keys of the cloud pasted into it, old files removed by another script, or never, and a restore nobody ever tried. With API Maker, a plan per database, on the schedule you say, made by the database's own program and encrypted before it leaves, the old backups removed by your rules, and a restore compared with the database of now, table by table.

0:39 In this video : a storage of S3, checked, a plan, a backup and its log, a plan for every engine, the bucket, browsed, and a restore, table by table.

A storage of S3

0:50 On the dashboard, under Operations : Automatic Backup. Three tabs : the storages, the plans, and the log. No storage yet. New storage. The store : Amazon S3, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces, Akamai, Scaleway, Google Cloud Storage, MinIO, or any other store which speaks S3, like the one of Calm Home. A name, the address of the store, the bucket, and a folder for the backups. Then the keys : the secret key is sealed on the server as soon as it is saved, and never shown again.

1:32 Check lists the bucket, writes a small file, reads it, and removes it : what a backup does. All four work.

A plan

1:39 Save. Plans, New plan. Catalog every night. What : mongodb, one database, catalog. The form of the file and its options are the ones of Import and Export. When : every few hours, every day, some days of the week, once a month, or an expression of cron. Every day, at two in the morning, in its time zone. The dialog says the schedule in words, and its next five moments.

2:10 Where, and how long. The store of Calm Home, in a folder per instance and per plan. Grandfather-father-son keeps the seven newest, one a day for a week, one a week for a month, and one a month for a year : a year in about twenty-five files. The newest one always stays. Security and alerts. Encrypt : the archive is encrypted on the server before it leaves, with the password of the plan, sealed on the server.

2:39 Gently, on by default : the backup takes the least share of the server. And an address told when a backup fails : Slack, Teams or Discord take it. Save.

A backup and its log

2:50 The plan waits for its moment. Back up now runs it at once : the dump, encrypted, sent into the bucket in parts. The log : the backup, its size, and how long it took. Its details : the dump, the upload, the old backups removed, its key in the bucket and its SHA-256.

Every engine

3:11 A plan for every engine : MongoDB every night, PostgreSQL every six hours, the orders of MySQL, MariaDB on Sundays, SQL Server, and Oracle once a month. Each one backed up by the database's own program. The log keeps every backup, every restore, and every moment missed. Every server looks at the plans every twenty seconds. The first one which takes a moment runs it : one backup per moment, however many servers there are.

3:42 A moment no server was there for is noted missed, not run late. And the archive never sits in memory : it goes from the disk to the bucket, piece by piece.

The bucket

3:53 Storages, browse : the bucket as it is. A folder per instance, a folder per plan, and in it each backup, an encrypted archive. Its details, from the description saved beside it : the plan, the database, the program, the SHA-256, and each collection with its documents.

A restore, table by table

4:15 Now a restore. Last night, catalog was backed up. This morning, three reviews were deleted by mistake. Its backups, newest first. Restore : where it goes. Into its own database, or into a new one, to look at the backup without touching anything. Bring it back : the backup comes from the bucket, is decrypted, and its SHA-256 is checked. Then it is compared with the database of now : reviews changed, the others the same.

4:48 Nothing, then Replace for reviews. What reviews holds now goes : the name of the database confirms it. Restore. Counted after : reviews, ok. The log keeps who restored what.

Good habits

5:03 Encrypt the plans whose data leaves your network, and keep their passwords somewhere else. Give the keys of a storage the least rights : list the bucket, then write, read and remove in its folder. And try a restore now and then, into a new database : it touches nothing.

Recap

5:21 To sum up. Storages of S3, checked before they are used. Plans with a schedule, a time zone and rules of retention. Encrypted, gently, and an alert when one fails. Every engine, every backup in the log. And a restore, compared, table by table. Next : the API Testing page. API Maker. Build faster, spend smarter, deliver better.