The Two-Site Model

Two independent Proxmox clusters connected by WireGuard: the rationale for splitting across two physical sites, the per-site topology pattern, and the hard rules governing inter-site connectivity.

Two physically separate sites each run a fully independent Proxmox cluster, an OPNsense VM as the WAN router and firewall, and a 10 Gb Netgear core switch — connected to each other by a WireGuard routed tunnel and nothing else. This page covers the rationale for that split and the hard constraints that define the model.

Why Two Independent Sites

The two-site design separates infrastructure concerns by role and supports cross-site disaster recovery without the risks of a stretched cluster or stretched storage:

Site A (10.10.0.0/16) handles management, DNS, monitoring, ZFS storage, and Proxmox Backup Server (PBS).
Site B (10.20.0.0/16) handles Ceph block storage, Kubernetes/OpenShift worker nodes, and high-core distributed compute.

Splitting roles means that heavy Ceph I/O and Kubernetes workloads at Site B do not compete with management services at Site A. PBS-A can replicate to PBS-B for cross-site disaster recovery without requiring Ceph to span the WAN.

Per-Site Topology

Every site follows the same pattern:

ISP → OPNsense → 10Gb Core Switch → Proxmox / K8s / Storage / PBS
                                  → UniFi Router WAN → Existing Wi-Fi/users

At Site A, the OPNsense VM is sa-fw-01, running on sa-edge-01 (SYS-E200-8D). The 10 Gb core switch is sa-sw-01 (Netgear XS716T). At Site B, the OPNsense VM is sb-fw-01, running on sb-edge-01 (SYS-E200-8D). The 10 Gb core switch is sb-sw-01 (Netgear XS748T). See Architecture Overview for the full device and role table.

Hard Constraints

These rules define what "two-site model" means in practice. None are optional.

One Proxmox cluster per site — no cross-WAN clusters

sa-pve runs at Site A; sb-pve runs at Site B. Never stretch a Proxmox cluster across WAN or WireGuard. Corosync requires LAN-like latency; a WAN/VPN cluster risks quorum instability.

No stretched Ceph

Site B Ceph stays local. Do not stretch Ceph across sites. Use PBS replication for cross-site disaster recovery — not stretched Ceph.

No stretched L2 between sites

Inter-site connectivity is routed only (WireGuard 10.255.0.0/24). Never bridge L2 across sites.

OPNsense owns infrastructure routing — Proxmox/OpenShift must not sit behind UniFi

OPNsense handles WAN, VLAN gateways, firewall policy, DHCP, and WireGuard at each site. Proxmox and OpenShift must connect to the OPNsense-side network, not behind UniFi.

OPNsense VM stays pinned to its local E200 — no HA migration

sa-fw-01 is pinned to sa-edge-01; sb-fw-01 is pinned to sb-edge-01. HA migration of the firewall VM must not be enabled.

Inter-Site Connectivity

The two OPNsense VMs form the only link between sites. WireGuard runs on the 10.255.0.0/24 transit subnet: OPNsense-A is 10.255.0.1, OPNsense-B is 10.255.0.2. Each site's full supernet is routed across the tunnel:

Route direction	Prefix
Site A → Site B	`10.20.0.0/16` via WireGuard
Site B → Site A	`10.10.0.0/16` via WireGuard

No L2 extends between sites. Every inter-site packet is routed at L3 by OPNsense. See WireGuard for tunnel configuration.

UniFi Retention and Double-NAT

Double-NAT is intentional

Existing UniFi routers remain in place so current Wi-Fi and user devices continue to work without disruption. UniFi sits behind OPNsense (double-NAT). This is a deliberate trade-off, not a misconfiguration.

The UniFi WAN interface connects to OPNsense via the UniFi WAN transit VLAN (VLAN 253, 10.x0.253.0/24). OPNsense is the gateway (.1); the UniFi router WAN interface takes .2 — for example, 10.10.253.1 (OPNsense-A) and 10.10.253.2 (UniFi WAN) at Site A. Proxmox and OpenShift infrastructure never go behind UniFi; only existing user and Wi-Fi devices do.

E200 Workload Guidance

The SYS-E200-8D nodes (sa-edge-01, sb-edge-01) join their local Proxmox cluster but must not be loaded with workloads that compete with OPNsense.

Suitable E200 workloads

OPNsense VM, DNS helper, UniFi controller, WireGuard helper, small reverse proxy, monitoring agent.

Avoid on E200

Heavy databases, Ceph OSDs, storage-heavy VMs, heavy Kubernetes workers, or any workload that competes with OPNsense for CPU or memory.

Architecture Overview — full device table, build sequence, and all hard rules
Site Roles — per-site hardware and workload breakdown
Topology — detailed network topology diagrams
WireGuard — tunnel configuration and inter-site routing
IP Addressing — supernets, VLAN subnets, and the addressing convention