Design Principles & Hard Rules
Non-negotiable architecture rules and guiding principles every operator must understand before making any change to this homelab.
Eight non-negotiable rules govern every architecture decision in this homelab. Violating any of them creates instability or unrecoverable failure modes — understand these before touching anything.
Hard Architecture Rules
The rules below are locked. Do not propose alternatives that violate them.
Proxmox Cluster Boundaries
One cluster per site — no cross-WAN Proxmox clusters
One Proxmox cluster per site: sa-pve at Site A, sb-pve at Site B. Never stretch a Proxmox cluster across WAN or WireGuard. Corosync requires LAN-like latency; a WAN/VPN cluster risks quorum instability and split-brain scenarios.
Ceph Locality
No stretched Ceph
Site B Ceph stays local. Do not stretch Ceph across sites. Use Proxmox Backup Server (PBS) replication for cross-site disaster recovery.
Infrastructure Routing
OPNsense owns infrastructure routing
OPNsense handles WAN, VLAN gateways, firewall policy, DHCP, WireGuard site-to-site, and inter-VLAN routing at each site. UniFi routing devices are bootstrap and fallback only.
OPNsense VM stays pinned — no HA migration
sa-fw-01 is pinned to sa-edge-01; sb-fw-01 is pinned to sb-edge-01. Never enable HA migration for the OPNsense VM. Migrating the firewall VM severs the network it is managing.
Proxmox and OpenShift must not sit behind UniFi
Proxmox hosts and OpenShift nodes must connect to the OPNsense-side network (VLAN 20 and above). Placing them behind UniFi routing removes OPNsense control and firewall policy from the infrastructure path.
UniFi Continuity
UniFi is retained — double-NAT is intentional
Existing UniFi routers remain in place so current Wi-Fi and user devices continue to work without disruption. UniFi sits behind OPNsense (double-NAT). This is a deliberate trade-off, not a misconfiguration.
Network Isolation
Never expose IPMI to the internet
All Supermicro IPMI interfaces are isolated on VLAN 10 (Network Mgmt / IPMI). IPMI must never be routed to or reachable from the internet.
No L2 stretch between sites
Inter-site connectivity is routed only, via WireGuard (10.255.0.0/24). OPNsense-A is 10.255.0.1; OPNsense-B is 10.255.0.2. Never bridge L2 between sites.
Guiding Principles
Broader operational principles that shape build decisions without rising to the level of hard constraints.
E200 Workload Limits
The SYS-E200-8D nodes (sa-edge-01, sb-edge-01) join their local Proxmox cluster but are reserved for lightweight workloads so OPNsense is never resource-starved.
Suitable E200 workloads
OPNsense VM, DNS helper, UniFi controller, WireGuard helper, small reverse proxy, monitoring agent.
Avoid on E200
Heavy databases, Ceph OSDs, storage-heavy VMs, heavy Kubernetes workers, or any workload that competes with OPNsense for CPU or memory.
Build Sequencing
Build one site at a time. Start with a flat bootstrap network, bring OPNsense online, then introduce VLANs once every node has its final 10.x.20.x management IP. Do not form a Proxmox cluster until all nodes in the site have their final management IPs and /etc/hosts entries are correct. Export OPNsense and switch configs after every major milestone.
Subnet Ownership
Site A owns 10.10.0.0/16; Site B owns 10.20.0.0/16. These ranges must not overlap and must not leak between sites at L2. All inter-site traffic routes through WireGuard at L3.
Related Pages
- Architecture Overview — topology diagram and full site role table
- Two-Site Model — detailed per-site breakdown
- Migration Phases — phased OPNsense insertion sequence
- Build Phases — ordered build checklist