Disaster Recovery Objectives, Ownership, RPO & RTO
Recovery mechanisms, technical ownership, implementation state, and objective gaps for network configuration, workloads, secrets, DNS, UniFi control, and whole-site failure.
Disaster recovery is asynchronous and service-specific. Local copies protect routine failure, planned cross-site copies protect site loss, and rebuildable configuration reduces dependence on any single appliance.
No Time-Based RPO or RTO Is Committed
The architecture requires local PBS backups and cross-site replication but does not define backup cadence, retention, replication schedule, restore-test frequency, recovery-point objective, or recovery-time objective. Values below report the available mechanism and current gap; they are not service guarantees.
Objective Register
| Capability | Recovery mechanism | Technical owner | Human owner | RPO | RTO | State |
|---|---|---|---|---|---|---|
| OPNsense configuration | Export /conf/config.xml after A2, A4, A7, A8, and later major milestones | Local OPNsense edge | Not named | Last captured milestone; no time target | Not committed | Required during Site A campaign |
| UniFi control | Existing device config plus UOS autobackup capability | sa-uos-01 | Not named | Not committed | Not committed | Backup/DR scheme not designed |
| Proxmox workloads | Local Proxmox Backup Server (PBS) jobs | PBS-A / PBS-B | Not named | Not committed | Not committed | Local-first design; policy incomplete |
| Whole-site workloads | Planned cross-site PBS copies over VLAN 90 and WireGuard | PBS-A ↔ PBS-B | Not named | Last successful replication; schedule not set | Not committed | Depends on parked Site B |
| OpenBao data | Planned Raft snapshots shipped over the backup path | Local OpenBao instance | Not named | Last successful snapshot; schedule not set | Not committed | Snapshot timer and shipping deferred |
| OpenBao unseal | Current Site A Shamir; future cross-site transit with manual cold-start break-glass | Local OpenBao instance and peer | Recovery-key custodian not named | Not applicable to stored data | Not committed | Site A Shamir active; transit future |
| DNS zones | sa-dns-01 primary with three secondary copies | Technitium DNS service | Not named | Last successful AXFR; cadence not committed | Not committed | Site B replicas future |
| Infrastructure definition | Git-tracked Pulumi, Ansible, runbooks, and generated configuration inputs | Repository and per-subproject tooling | Not named | Last committed and exported state | Not committed | Active reconstruction source |
Technical owner identifies the system or layer that performs recovery. The repository does not assign a named person or on-call role, so human accountability remains an explicit gap.
Failure Classes and Recovery Sources
| Failure class | Preferred recovery source | Cross-site dependency | Main limitation |
|---|---|---|---|
| One VM or service | Local configuration, local data, local PBS | None for ordinary local restore | Restore procedure and timing are service-specific |
| One host | Surviving local cluster capacity and local PBS | None when another local copy exists | sa-edge-01 and sa-stor-01 concentrate multiple services |
| Local storage loss | Local redundancy first; PBS restore second | Cross-site PBS copy for site-level loss | PBS-A is co-located with Site A storage |
| One site unavailable | Other site's last replicated PBS and OpenBao snapshot copies | Required and not yet deployed | Site B is parked |
| WireGuard unavailable | Local service copies and local PBS | Replication and cross-site management pause | New DNS changes and cross-site unseal paths wait |
| Both sites cold | Password-manager recovery material, tier-0 secrets, source repository, hardware rebuild | None | Future transit-unseal deadlock requires manual seal migration |
Replication Is Not Synchronous Storage
Ceph, ZFS, Proxmox quorum, and OpenBao Raft do not span sites. Cross-site recovery accepts a non-zero data-loss window bounded by the last successful asynchronous copy.
Recovery Precedence
- Restore physical power, local switching, Proxmox management, and the site E200 network bridges.
- Restore OPNsense and validate gateways, policy, DNS forwarding, and required WireGuard paths.
- Unseal OpenBao using the current Shamir path or the future transit break-glass procedure.
- Restore local storage, DNS, UOS, and PBS; confirm that authoritative data and backup datastores are readable.
- Restore core service VMs before Kubernetes/OpenShift and application workloads.
- Validate service behavior, then re-establish cross-site replication and record the new recovery point.
The order follows service dependencies. It does not replace the platform-specific restore procedure.
Recovery Mechanisms by Domain
Network and Control Plane
OPNsense configuration exports are mandatory after major migration milestones. The Gateway Max remains a manual Site A rollback spare through cutover disposition, not automatic failover. UOS loss does not stop already adopted devices from forwarding, but the controller backup and rebuild policy remains undecided.
Workloads and Storage
PBS-A and PBS-B back up their local sites and replicate across WireGuard so neither site is the sole recovery copy. The design does not yet define retention, prune/garbage-collection cadence, synchronization direction, bandwidth window, credentials, encryption, or recurring restore evidence.
Secrets
Each site has an independent single-node OpenBao Raft store. Planned snapshots travel to the other site. Tier-0 bootstrap secrets and recovery keys remain outside OpenBao so a sealed or lost instance can be rebuilt. Current Site A restarts require three-of-five Shamir unseal; future simultaneous-site recovery starts by manually returning one instance to Shamir.
DNS
sa-dns-01 is the writable primary. Secondary servers answer from the last transferred copy when the primary or WireGuard path is unavailable. Writes and new Site B registrations wait for primary reachability; this is stale-but-functional recovery, not active multi-primary DNS.
Decisions Still Required
| Decision | Required output |
|---|---|
| Service tiers | Classify each service by business and infrastructure criticality |
| Ownership | Name the accountable operator and recovery-key custodian for each tier |
| RPO | Set maximum acceptable data loss per service |
| RTO | Set maximum acceptable restoration time per service |
| Backup policy | Define cadence, retention, pruning, garbage collection, encryption, and replication direction |
| Restore evidence | Define test frequency, sample workloads, pass criteria, and where results are recorded |
| Spare strategy | Identify replacement hardware for edge, storage, switching, NIC, and boot-media failures |
| Communications | Define how recovery status and decisions are recorded during an incident |
Until these values are approved, recovery planning must use factual phrases such as “last successful export,” “last successful snapshot,” and “last successful PBS replication,” never an implied hourly or daily guarantee.
Security Trust Zones & Permitted Flows
Target default-deny policy for WAN, management, services, workloads, DMZ, monitoring, backup, trusted, IoT, guest, storage, and inter-site traffic.
Capacity, Power, Thermal & Expansion Envelopes
Known compute, memory, storage, network, switch-port, and PoE limits, plus the electrical and thermal measurements required before expansion.