Physical & Logical Topology
How WAN, Layer 2 transport, routed VLANs, storage planes, service traffic, and failure boundaries fit together across Site A and Site B.
The topology is easiest to understand as a set of traffic paths. OPNsense handles every routed transition, switches carry local Ethernet, storage stays on directly attached no-gateway networks, and WireGuard joins only the two routed site prefixes.
The Overview Map Shows the Approved Architecture
Site A's three-switch fabric is an approved target, not the current production path. The Gateway Max/XG6 bootstrap island remains separate and never connects to an XG10. Site B remains parked.
Follow Four Representative Paths
Internet to a Site A Client
During staging only, the WAN side becomes ONT → UCG Max → sa-fw-01. The downstream target path and all 10.10.x addressing remain unchanged, so Phase A8 changes one WAN cable rather than rebuilding the LAN.
Site A VM to Storage
VLAN 60 has no .1, OPNsense interface, SVI, default gateway, or static route. Storage consumers attach directly to the same Layer 2 segment.
Site B Ceph Write
Both networks stay inside Site B and neither has a gateway. Traffic never routes from VLAN 60 to VLAN 65 and never traverses WireGuard.
Cross-Site DNS Replication
This is an application flow over routed prefixes. The VLAN 30 broadcast domain does not cross the tunnel.
Network Planes
| Plane | Networks | Who terminates it | Where it can travel |
|---|---|---|---|
| WAN edge | ISP-assigned WANs | Local OPNsense VM | Internet and local firewall only |
| Management | VLANs 10 and 20 | Network devices, IPMI, Proxmox hosts, OPNsense | Routed locally under policy; cross-site only when explicitly allowed |
| Cluster heartbeat | VLAN 25 | Proxmox hosts | Site-local Layer 2 only; no gateway |
| Services and workloads | VLANs 30, 40, 50, 70, 80 | VMs, K8s/OpenShift, DMZ, monitoring | Routed by local OPNsense |
| Storage | VLAN 60; Site B VLAN 65 | Storage clients and servers | Site-local Layer 2 only; no gateway |
| Backup | VLAN 90 | Proxmox hosts and PBS | Routed locally; planned replication over WireGuard |
| Client access | VLANs 100, 110, 120 | Clients and APs | Routed and filtered by local OPNsense |
| Site B compatibility | VLAN 253 | sb-fw-01 and USG Pro WAN | Site B only |
| Native sink | Site A VLAN 999 | Trunks and unused ports only | No subnet, gateway, DHCP, or client workload |
| Inter-site transit | 10.255.0.0/24 | sa-fw-01 and sb-fw-01 | Routed firewall-to-firewall only |
Site A Attachment Model
sa-fw-01 reaches sa-sw-01 through an explicit 802.1Q trunk. The three XG10 switches form a DAC triangle with RSTP priorities 4096, 8192, and 12288. Host links are distributed across all three switches; sa-ap-01 lands on sa-sw-03 port 6.
The triangle provides link-path redundancy, not gateway, chassis, or power redundancy. sa-sw-01 remains the only target attachment from the fabric to OPNsense.
The Bootstrap and Target Switching Domains Never Bridge
The XG10 fabric never connects to the Gateway Max or XG6, including during adoption. XG6 ports 5 and 6 remain disconnected. Target switches are adopted only through final VLAN 10 on the isolated OPNsense-backed plane.
Site B Attachment Model
sb-fw-01 trunks routed Site B VLANs to the Netgear sb-sw-01 core. Each of the five compute nodes uses dedicated paths for management, Corosync, Ceph public, Ceph cluster, backups, and workload bridges. sb-sw-02 handles access, IPMI, AP, and PoE/client functions under its restricted VLAN policy.
The USG Pro WAN remains 10.20.253.2 behind OPNsense 10.20.253.1. Infrastructure connects to sb-sw-01 and does not sit behind the USG NAT boundary.
Failure Boundaries
| Failure | What continues | What stops or degrades |
|---|---|---|
| One Site A inter-switch DAC | RSTP uses the remaining path | Brief reconvergence; no loss of attached endpoint unless its switch also fails |
| One Site A switch | Surviving switches retain their direct triangle link | Endpoints unique to the failed switch are lost |
sa-sw-01 | Same-switch Layer 2 on surviving devices | OPNsense attachment is lost; routed gateways and WAN become unreachable from the fabric |
sa-fw-01 or sa-edge-01 | Same-VLAN Layer 2 forwarding | Site A gateways, DHCP, WAN, WireGuard, DNS forwarding, and inter-VLAN traffic |
sa-uos-01 | Existing switch and AP forwarding | Adoption, telemetry, and configuration changes |
| WireGuard or either ISP | Each site-local cluster, storage plane, and switch fabric | Cross-site management, AXFR, PBS replication, transit unseal |
| Site B Ceph node | Remaining Ceph members follow local quorum and replication behavior | Capacity/performance degrade according to failed OSD/MON roles; Site A storage is unaffected |
The full system map adds every named host and service with searchable IPs and direct relationships. Exact cables and switch profiles remain in the Site A port map and Site B port map.
Site Roles & Systems
What Site A and Site B are responsible for, how physical hosts and service workloads support those outcomes, and where workloads should be placed.
Full Two-Site System Map
Interactive estate-wide diagram of every named host, switch, edge VM, service workload, and logical platform across both sites, with IPs, networks, purpose, placement, and connections.