Most organisations have backups, and the daily report from the backup software is green. But the important question is not “do we have backups?” It is “if we lose the server today, how many hours until it is back, and how complete will it be?” This article describes the routine I use to answer that question; the same approach used with automatic Veeam backups in the Golrokh Teb imaging centre project.
Why a “successful backup” is not enough
A “success” message only says the backup job finished without errors. It does not guarantee that:
- everything that matters was backed up; a server added later may not be in the job.
- the database was in a healthy, consistent state at backup time.
- the backup itself has not been corrupted or hit by ransomware.
- the restore finishes within the time the business can tolerate.
Step 1: write down what has to come back, and by when
For each critical service, agree two numbers with management:
- Maximum acceptable data loss (RPO): with nightly backups, up to one working day of data may be lost. If that is not acceptable, backups must run more often.
- Maximum downtime (RTO): how many hours until the service has to be back.
Without these two numbers you cannot tell whether a restore test passed.
Step 2: actually read the reports
- Do not ignore warnings; many problems show up as warnings first.
- Whenever a new server or virtual machine is added, check that it is in the backup job.
- Check free space and retention periodically.
Step 3: test a file restore
The simplest test: restore a few files and folders from different dates to a separate location and open them. Pick files people actually use, not test files.
Step 4: test booting a virtual machine
Restore a full virtual machine into an isolated network (not connected to the production network, so it does not clash with the original) and check:
- Does the operating system boot?
- Do the services start, and can you sign in?
In Veeam, SureBackup automates this test: it runs the machine from the backup in an isolated environment and checks that it is healthy.
Step 5: test the application and database
A server that boots means the operating system is fine, not necessarily the data. Restore critical databases (for example the SQL Server behind the finance software or the reception system) and:
- check database integrity; in SQL Server with
DBCC CHECKDB. - ask a key user of that application to look at the latest records and confirm the data reaches the expected date.
Step 6: measure the restore time
Time the restore from start to finish and compare it with the RTO. If a full restore takes 12 hours and the business can tolerate 4, the problem is the design, not the test.
Step 7: keep one copy out of ransomware’s reach
- The 3-2-1 rule: three copies, on two kinds of media, one outside the organisation. The details are in Backups with Veeam Backup: the 3-2-1 rule.
- Keep at least one copy immutable or disconnected from the network.
- Keep the backup admin account separate from domain accounts; ransomware that reaches the domain should not be able to delete the backups.
Step 8: record the result and write a runbook
After each test, one page is enough: date, what was restored, how long it took, what went wrong and what needs fixing. Also write a step-by-step restore procedure, so someone else can do it on the day of a crisis.
Suggested schedule
- Daily: check reports and warnings
- Monthly: restore a few files and one database
- Quarterly: fully boot critical virtual machines in an isolated environment and time it
- After every major change: test again, for example after a server migration or a change of backup software
If you are not sure your servers’ backups can really be restored, request a review from the HPE servers and virtualization service.
Photo: NOIRLab/NSF/AURA/T. Slovinský, licensed under CC BY 4.0, via Wikimedia Commons.jpg); cropped.
