Design Principles & Hard Rules

Nine non-negotiable rules govern every architecture decision in this homelab. Violating any of them creates instability or unrecoverable failure modes: understand these before touching anything.

Hard Architecture Rules

The rules below are locked. Do not propose alternatives that violate them.

Proxmox Cluster Boundaries

One Cluster per Site: No Cross-WAN Proxmox Clusters

One Proxmox cluster per site: sa-pve at Site A, sb-pve at Site B. Never stretch a Proxmox cluster across WAN or WireGuard. Corosync requires LAN-like latency; a WAN/VPN cluster risks quorum instability and split-brain scenarios.

Ceph Locality

No Stretched Ceph

Site B Ceph stays local. Do not stretch Ceph across sites. Use Proxmox Backup Server (PBS) replication for cross-site disaster recovery.

Infrastructure Routing

OPNsense Remains the Stateful Edge

OPNsense owns WAN, NAT, WireGuard, DMZ, routed VLAN gateways, DHCP, and stateful zone policy at Site A. The three UniFi switches remain Layer 2 only. Site B retains its separate OPNsense/VLAN-253 design.

OPNsense VM Stays Pinned: No HA Migration

sa-fw-01 is pinned to sa-edge-01; sb-fw-01 is pinned to sb-edge-01. Never enable HA migration for the OPNsense VM. Migrating the firewall VM severs the network it is managing.

Proxmox and OpenShift Must Not Sit Behind UniFi

Proxmox hosts and OpenShift nodes must connect to the OPNsense-side network (VLAN 20 and above). Placing them behind UniFi routing removes OPNsense control and firewall policy from the infrastructure path.

XG10 Switches Never Connect to UCG/XG6

sa-sw-01, sa-sw-02, and sa-sw-03 are commissioned only on the isolated OPNsense-backed target plane. They never connect to UCG Max or XG6, including for adoption; XG6 p5/p6 remain disconnected.

UniFi Continuity

Routing Differs by Site

Site A's approved target removes the Gateway Max and double NAT while all three UniFi switches remain Layer 2 only. Site B retains the USG Pro behind OPNsense with intentional double NAT. The Site A target is not yet deployed.

Network Isolation

Never Expose IPMI to the Internet

All Supermicro IPMI interfaces are isolated on VLAN 10 (Network Mgmt / IPMI). IPMI must never be routed to or reachable from the internet.

No L2 Stretch Between Sites

Inter-site connectivity is routed only, via WireGuard (10.255.0.0/24). OPNsense-A is 10.255.0.1; OPNsense-B is 10.255.0.2. Never bridge L2 between sites.

Guiding Principles

Broader operational principles that shape build decisions without rising to the level of hard constraints.

E200 Workload Limits

The SYS-E200-8D nodes (sa-edge-01, sb-edge-01) join their local Proxmox cluster but are reserved for lightweight workloads so OPNsense is never resource-starved.

Suitable E200 Workloads

OPNsense VM, DNS helper, UniFi controller, WireGuard helper, small reverse proxy, monitoring agent.

Avoid on E200

Heavy databases, Ceph OSDs, storage-heavy VMs, heavy Kubernetes workers, or any workload that competes with OPNsense for CPU or memory.

Build Sequencing

Build one site at a time. Use the XG6 as a separate bootstrap island, bring OPNsense VLAN 10 online, then build and adopt the XG10s only on the final-addressed target tree behind OPNsense. Keep XG6 p5/p6 disconnected. Move hosts directly from the XG6 service lead to their final ports; do not bridge bootstrap and target. Do not form a Proxmox cluster until all nodes in the site have their final management IPs and /etc/hosts entries are correct. Export OPNsense and switch configs after every major milestone.

Subnet Ownership

Site A owns 10.10.0.0/16; Site B owns 10.20.0.0/16. These ranges must not overlap and must not leak between sites at L2. All inter-site traffic routes through WireGuard at L3.

Architecture Overview: topology diagram and full site role table
Two-Site Model: detailed per-site breakdown
Migration Phases: phased OPNsense insertion sequence
Build Phases: ordered build checklist

On this page