Context / need
The platform hosts data that does not belong to me: client websites in production, documents of a client firm, application data. A dying disk, an update gone wrong, a human mistake: the question is when it will happen. And on that day, having backups is not enough. You need to be able to restore.
Constraints
- Third-party data. Loss is not negotiable, and the off-site copy must be encrypted since it leaves the infrastructure.
- The day you restore is a bad day: the procedure must already be written, and followable by a tired human.
- Automatic. A manual backup is a backup that will eventually not be made.
Options considered
- Snapshots or disk images only: simple, but opaque. Restoring one precise service or database becomes a heavy operation, and application consistency is not guaranteed.
- Application-level backups per service: clean dumps (databases, data, configuration), granular, testable, readable.
- The choice: application-level first, disk-level as a complement.
Decision & why
3-2-1 strategy: several copies, on distinct media, one of them encrypted off-site. Each service gets its application-level backup (database dump, data, configuration) with automatic rotation: the footprint stays bounded and restore points spread over time.
And above all, a restore runbook: for each type of data, the exact procedure, in order, with the known pitfalls. Written cold, to be executed under pressure.
Where the rollout stands. The strategy runs today on the client case platform: local versions and an encrypted off-site copy. On the server side, generalization is in progress; together with timed restore tests, it makes up the Proven backups roadmap project.
Accepted tradeoff
Full restores are not yet tested regularly and with tooling: until they are, the recovery time remains an estimate (roadmap project). Application-level granularity also requires a bit of per-service maintenance, the price of readability.
Outcome
- Reliable: on the deployed scope, losing a file, a folder or the hardware has a documented way out, with staged restore points; the server is brought under the same contract as the project progresses.
- Secure: the copy that leaves the infrastructure is encrypted, and backups follow the same access hygiene as everything else (secrets).
What the restore runbook contains
Without copying the document, which stays internal, its structure:
- Per scenario. Loss of a service, corruption of a database, loss of the whole server: each scenario has its own procedure.
- Explicit prerequisites: where the copies live, how to decrypt the off-site copy, which secrets are needed and where they live.
- Recovery order: what to restore first (data, then services, then the edge) and how to verify each step before moving to the next.
- Known pitfalls, noted from real restores and drills.