Blast Radius & Recovery Matrix

Blast radius is bounded by site, layer, and attachment point. A link failure differs from a switch failure, a controller failure differs from a forwarding failure, and a routed-service failure differs from loss of local Layer 2.

Local Failures Stay Local

No Proxmox cluster, Ceph pool, OpenBao Raft cluster, Corosync network, or Layer 2 segment spans both sites. A local infrastructure failure may remove an entire site's service, but it cannot directly change quorum or storage health at the other site.

Failure and Recovery Matrix

Failure domain	Direct blast radius	What remains available	Recovery path	Exposure
One ISP handoff	Internet and WireGuard at that site	Local switching, cluster, storage, and same-site services	Restore the handoff; Site A can use its documented manual UCG rollback posture during the campaign	Single ISP termination is an accepted risk
One Site A DAC	One RSTP path	Remaining two-switch path and attached endpoints	RSTP reconverges; repair the failed DAC before another fabric failure	Link redundancy exists
`sa-sw-02` or `sa-sw-03`	Endpoints attached only to that switch	Other switches and their direct triangle link	Restore or replace the failed switch and reapply its known profile	Endpoint loss is switch-local
`sa-sw-01`	Its endpoints plus the only OPNsense fabric attachment	Same-switch Layer 2 on surviving switches	Restore or replace `sa-sw-01`; no alternate router attachment is pre-approved	Fabric has link redundancy, not gateway-uplink redundancy
`sa-fw-01` or `sa-edge-01`	Site A WAN, gateways, DHCP, NAT, WireGuard, DNS forwarding, inter-VLAN traffic; edge-host UOS/Bao if the host fails	Same-VLAN Layer 2 and existing adopted-device forwarding when switches remain powered	Restore the pinned edge VM/host from code and exports; use manual UCG rollback only within its documented posture	Site A edge is a concentrated single point of failure
`sa-uos-01`	Adoption, telemetry, configuration changes	Existing switch and AP forwarding	Restore or rebuild UOS; controller backup policy is not yet designed	Control loss does not equal forwarding loss
`sa-bao-01`	New secret reads and config pushes that use Bao	OPNsense runtime traffic; consumers with encrypted fallback; already configured services	Current three-of-five Shamir unseal or rebuild/restore from planned snapshot path	Single-node Raft; snapshot automation deferred
`sa-dns-01`	Authoritative writes, DDNS updates, primary replication source	Reachable secondary resolvers serve the last zone copy	Restore the primary and resume AXFR/NOTIFY	Stale-but-functional replicas; no committed RTO
`sa-stor-01`	Site A ZFS, PBS-A, primary DNS placement, monitoring, databases, hosted core VMs	Other Site A hosts and the entire Site B storage domain	Repair/replace storage, then restore from an independent PBS copy when available	PBS-A co-location creates correlated local loss
One Site A ThinkPad	Its local workloads and attached paths	Other `sa-pve` nodes according to local quorum and workload placement	Restore or reschedule workloads; physical intervention may be required	No IPMI on ThinkPads
One Site B Ceph node	Its OSDs and any MON/MGR or workload roles	Remaining Ceph members and Site A storage	Follow Ceph health, recovery, and replacement procedures	Capacity and performance degrade; exact effect follows the failed roles
WireGuard tunnel	Cross-site management, DNS changes, PBS copies, transit unseal, later Site B UniFi control	Both sites' local routing, clusters, storage, switching, local DNS copies, local PBS jobs	Restore the tunnel, then resume and verify replication	Future OpenBao restart may need break-glass while peer is unreachable
One PBS instance	Local backup and restore service at that site	Production workloads and the other PBS instance	Restore PBS and datastore; recover from the other site's last copy when deployed	Cadence, retention, and restore-test policy are open
Entire site	Every system physically at that location	The other site's local infrastructure	Rebuild from source, exports, recovery keys, and last cross-site copies	Cross-site DR is designed but Site B-dependent mechanisms are not yet live
Both sites cold	All site-local services; future mutual transit unseal cannot start automatically	Offline source, password-manager material, hardware, and tier-0 secrets	Seal-migrate one OpenBao instance to Shamir, unseal it, bring up its peer, then restore remaining services in dependency order	Documented manual recovery; no committed time objective

Concentrated Failure Domains

Concentration point	Capabilities sharing the failure domain	Design response
`sa-edge-01`	OPNsense, OpenBao, UOS, edge networking	Keep workloads light, preserve exports and break-glass credentials, retain manual rollback
`sa-sw-01`	RSTP root, edge trunk, several host links	RSTP protects DAC paths only; replacement configuration and physical access remain necessary
`sa-stor-01`	ZFS, PBS-A, primary DNS placement, monitoring, databases, core VMs	Mirror vdevs protect drive failure; cross-site PBS is the site-loss path
WireGuard	Cross-site DNS replication, PBS copies, future transit unseal, later UniFi L3 adoption	Local services continue; every cross-site consumer needs a stale-copy, pause, or break-glass mode
`sa-dns-01`	Writable forward and reverse zones	Secondary servers retain read availability from the last AXFR

Recovery Patterns

Redundant Path

RSTP can route around one Site A inter-switch link. It does not replace a switch, the only OPNsense attachment on sa-sw-01, a power feed, or the edge host.

Rebuild from Definition

Pulumi, Ansible, pinned configuration, port maps, and runbooks recreate infrastructure. Rebuild still depends on current exports, tier-0 credentials, recovery keys, and healthy replacement hardware.

Restore from Copy

ZFS mirrors and Ceph replication address local device failure. PBS and OpenBao snapshots address service or site recovery. These mechanisms protect different layers and are not interchangeable.

Operate Degraded

UOS-down forwarding, DNS secondaries, local PBS during a tunnel outage, and encrypted secret fallbacks allow limited operation while the failed control plane is repaired.

Recovery Decision Boundary

Routine recovery restores the approved architecture. Moving an OPNsense VM to another host, routing VLAN 25/60/65, attaching an XG10 to the UCG/XG6 island, stretching storage or quorum, or publishing management services changes the architecture and is not an incident shortcut.

Use disaster recovery objectives for recovery commitments and gaps, service dependencies for restoration order, and physical & logical topology for packet-path detail.

On this page