Backups and disaster recovery for your application

A backup only counts once you have restored it. So decide per system how much data you can lose and how long you can be down, keep copies according to the 3-2-1 rule and test restores on a schedule. For a database, that usually means point-in-time recovery plus your own copy outside your cloud account.

RPO and RTO in plain words

RPO (recovery point objective) is how much data you can afford to lose, expressed in time. If you take one backup a night, you can lose up to 24 hours of work. RTO (recovery time objective) is how long it may take until everything runs again.

The product owner picks those numbers, and the team says what it costs to meet them. They differ per system, for example:

The smaller the numbers, the more expensive the setup. Write them down even if they are generous, so everyone knows what to expect when something goes wrong.

The 3-2-1 rule in the cloud

The US Cybersecurity and Infrastructure Security Agency (CISA) describes the 3-2-1 rule like this: keep three copies of any important file (one primary and two backups), on two different media types, and store one copy offsite.

In the cloud, the second storage type is a different service, for example object storage next to your database snapshots. The other location is a separate account or a different provider, with its own credentials.

That last part matters most. Whoever gets into your production account, through a leaked key or a faulty script, must not be able to delete your backups as well. Turn on versioning or a retention lock on the bucket if your storage supports it, so a backup can't be deleted for a set period. Protecting those keys is the subject of managing secrets and API keys safely.

Databases: dumps, snapshots and point-in-time recovery

Postgres gives you two kinds of backup. A pg_dump is a logical copy of a single moment, useful as an extra copy outside your account and for pulling back a single table. You cannot use a dump to replay changes made after that moment.

Point-in-time recovery uses a base backup plus the WAL files archived after it. With those you can restore the database to any moment since that base backup. Your RPO drops from a day to minutes or less, depending on how often you archive WAL.

With a managed database, the provider usually handles this. On Amazon RDS you set a retention period of 0 to 35 days (0 disables automated backups), and you can restore to any point within that period. Be careful when you delete the instance, though. If you don't choose "Retain automated backups", the automated backups are deleted along with the database. Manual snapshots do remain.

My advice is to keep your own copy next to the provider's backups, for example a daily pg_dump to storage in another account. That way you still have something if the account itself is the problem.

A backup nobody restores

In 2017 GitLab published an open postmortem of an incident on 31 January. An engineer accidentally removed data on the primary production database while meaning to wipe a replica. It then turned out the pg_dump backups were not working. They ran pg_dump 9.2 against PostgreSQL 9.6 and failed, and the emails about that error were rejected, so nobody knew.

GitLab eventually restored an LVM snapshot taken about six hours before the incident. The database changes from those hours were lost, and GitLab.com was down for about eighteen hours.

I take two lessons from that. A backup job that sends no errors may still be broken, so you want an alert when the success message is missing. And you only know whether a backup is usable once you have restored it.

Test restores on a schedule

Automate a small restore test. A nightly job restores the latest backup into a temporary database and checks a few things:

• whether the restore succeeds and how long it takes, compared with your RTO;
• whether the newest record is no older than your RPO allows;
• whether row counts in the main tables are roughly right;
• whether your application can start against that database.

Have the job send a signal when it succeeds, and have your monitoring alert when that signal does not arrive. That way you also notice when the job itself stops running.

On top of that, run a full exercise once or twice a year: restore a complete environment from backups, with someone in charge who did not build the system. That tests the runbook too. Anything that person has to ask about goes into the runbook afterwards.

More than the database

Go through what you need to rebuild the whole application:

• uploads and other files in object storage, with versioning enabled;
• secrets and configuration, with a copy of the vault or a documented way to recreate them;
• your infrastructure as code, so you can stand up an environment without clicking through a console;
• DNS settings and certificates;
• data in SaaS tools your processes depend on.

Backups cost storage, and without a retention policy they grow every month. Agree how long you keep daily, weekly and monthly copies, and include that in your monthly cost review. Backups usually contain personal data too. Encrypt them and cover them in your GDPR arrangements.

If your organisation falls under NIS2, this is not optional. The directive lists backup management and disaster recovery among its business continuity measures (Article 21). What else NIS2 means for your team is covered in NIS2 and your development team.

Sources

• CISA: Data Backup Options (PDF)
• PostgreSQL: continuous archiving and point-in-time recovery
• Amazon RDS: introduction to backups
• Amazon RDS: backup retention period
• GitLab: postmortem of database outage of January 31
• EUR-Lex: NIS2 Directive (EU) 2022/2555