The Two-Site Model
Why AORXI uses two independent site-local failure domains, what the sites share, and why Site A and Site B intentionally use different network and storage designs.
The two sites cooperate at the service layer but remain independent at the infrastructure layer. A WAN outage can interrupt cross-site replication or management without changing local Proxmox quorum, Ceph health, Layer 2 forwarding, or gateway ownership.
The Boundary Model
| Boundary | Site A | Site B | Cross-site behavior |
|---|---|---|---|
| Address space | 10.10.0.0/16 | 10.20.0.0/16 | Prefixes route through WireGuard |
| WAN router | sa-fw-01 | sb-fw-01 | Firewalls peer on 10.255.0.0/24 |
| Proxmox | sa-pve | sb-pve | Separate clusters and quorum |
| Corosync | 10.10.25.0/24 | 10.20.25.0/24 | Never routed or tunneled |
| Primary storage | ZFS on sa-stor-01 | Ceph across five compute nodes | No shared or stretched storage fabric |
| Backup | PBS-A | PBS-B | Planned asynchronous replication on VLAN 90 |
| Secrets | sa-bao-01 | sb-bao-01 | Independent Raft stores; planned transit unseal |
| Layer 2 | Three-switch UniFi RSTP fabric | Netgear core plus UniFi access | No VLAN or broadcast domain crosses the WAN |
A Routed Relationship, Never a Stretched Datacenter
No Proxmox cluster, Corosync network, Ceph pool, OpenBao Raft cluster, broadcast domain, or client VLAN spans both sites. The only inter-site network is routed WireGuard.
Why the Sites Are Independent
WAN Faults Do Not Become Quorum Faults
Corosync expects stable, LAN-like latency. Keeping sa-pve and sb-pve separate prevents an ISP or tunnel interruption from changing cluster membership or quorum at the other site.
Storage Latency Stays Local
Site B Ceph uses dedicated public and cluster networks inside Site B. Stretching writes across the WAN would make latency and tunnel availability part of every storage operation. Cross-site recovery therefore uses PBS replication instead of synchronous Ceph replication.
Network Ownership Stays Obvious
Each OPNsense instance owns one site. Each switch fabric carries local VLANs only. A route across WireGuard is visible and policy-controlled; a hidden Layer 2 dependency across the WAN is not permitted.
Recovery Remains Possible When a Service Is Unavailable
OpenBao instances are independent and retain break-glass recovery. PBS copies are asynchronous. The design avoids a circular requirement where Site A must be healthy to boot Site B, or vice versa.
Deliberate Asymmetry
The sites share principles, not identical hardware.
| Design choice | Site A | Site B | Reason |
|---|---|---|---|
| Switching | sa-sw-01/02/03 UniFi L2 RSTP triangle | sb-sw-01 Netgear L2 core | Uses the hardware and attachment model already assigned to each site |
| Client edge | OPNsense routes trusted, IoT, and guest networks directly | USG Pro remains behind OPNsense on VLAN 253 | Site B preserves its existing user network and accepted double NAT |
| Storage | Local ZFS mirrors | Distributed Ceph, replication size 3 | Site A has the storage-heavy 5049A-T; Site B has five OSD-capable nodes |
| Compute shape | Two ThinkPad workers plus edge and storage hosts | Five Supermicro compute/storage nodes plus edge | Site B is the worker-heavy site |
| UniFi control | sa-uos-01 manages Site A locally | sb-sw-02 is adopted later over WireGuard | One controller is sufficient; no second Site B controller is planned |
Do Not Normalize Away the Differences
Site A removes the Gateway Max from its target data path and retires VLAN 253. Site B retains the USG Pro behind OPNsense on VLAN 253. A rule or migration applied uniformly to both sites can break the intentional Site B compatibility boundary.
What the Tunnel Is For
WireGuard joins sa-fw-01 (10.255.0.1) and sb-fw-01 (10.255.0.2). It carries explicitly allowed routed flows after Site B is online:
| Flow | Direction | Purpose |
|---|---|---|
| Site prefixes | 10.10.0.0/16 ↔ 10.20.0.0/16 | Controlled management and service reachability |
| DNS AXFR / NOTIFY | sa-dns-01 → sa-dns-02, sb-dns-01, sb-dns-02 | Replicate core.aorxi.io and reverse zones |
| PBS replication | PBS-A ↔ PBS-B | Cross-site backup copies and disaster recovery |
| OpenBao transit | sa-bao-01 ↔ sb-bao-01 | Planned cross-site transit auto-unseal, not shared storage |
| UniFi adoption | sa-uos-01 → sb-sw-02 | Later L3 adoption of Site B UniFi access gear |
WireGuard does not carry Corosync, Ceph cluster traffic, Layer 2 frames, or an always-required boot dependency without a documented break-glass path. WireGuard contains the routing and policy detail.
Failure-Domain Consequences
| Failure | Local effect | Other-site effect |
|---|---|---|
| One ISP or WireGuard path | Cross-site services and replication pause | Local cluster, storage, switching, and gateways continue |
| One OPNsense VM or edge host | Routed gateways, WAN, DHCP, and inter-zone policy fail at that site | Other site remains locally routed |
| Site A storage host | Site A ZFS-backed services and PBS-A are affected | Site B Ceph remains independent; replicated backups depend on last completed sync |
| Site B Ceph quorum or OSD failure | Site B storage degrades according to Ceph health | Site A ZFS remains independent |
| One OpenBao instance | Local secret service uses its recovery/fallback posture | Other OpenBao instance retains its own data |
Continue with Site roles & systems to see what belongs inside each boundary, then Physical & logical topology to follow traffic through it.
Architecture Overview
A guided mental model of the two-site private cloud: current state, site responsibilities, control boundaries, and the best path through the architecture documentation.
Site Roles & Systems
What Site A and Site B are responsible for, how physical hosts and service workloads support those outcomes, and where workloads should be placed.