Migration Phases
Detailed walkthrough of the six OPNsense migration phases: inserting OPNsense upstream of UniFi at each site, building the WireGuard site-to-site VPN, migrating Proxmox to final management IPs, and standing up clusters, storage, and Kubernetes.
Six ordered phases bring OPNsense into production at both sites, establish the inter-site WireGuard VPN, then migrate Proxmox hosts to their final management addresses before standing up clusters, storage, Proxmox Backup Server (PBS), and Kubernetes. Export the OPNsense config after every major milestone.
Proxmox and OpenShift must not sit behind UniFi
OPNsense is the infrastructure router and firewall at each site. Proxmox, OpenShift, PBS, and storage must connect to the OPNsense-side network. Placing them behind UniFi is a hard architecture violation.
Phase Overview
| Phase | Scope | Summary |
|---|---|---|
| 1 | Site A | Insert OPNsense upstream of UniFi |
| 2 | Site B | Insert OPNsense upstream of UniFi |
| 3 | Both | WireGuard site-to-site VPN |
| 4 | Both | Move Proxmox to final VLAN 20 IPs |
| 5 | Both | Build Proxmox clusters (sa-pve, sb-pve) |
| 6 | Both | Storage, PBS, Kubernetes |
Phase numbering differs between sources
vault/05-opnsense-unifi-migration.md uses a five-phase sequence that begins with "Prepare OPNsense offline" as Phase 1 and ends with "Optional UniFi LAN routing" as Phase 5. It does not enumerate Phases 4–6 above. The six-phase sequence in this document follows CLAUDE.md and is authoritative; vault/05 detail is incorporated for the phases it covers. The "Prepare offline" work from vault/05 Phase 1 appears as a pre-phase section below. Vault/05 Phase 5 (optional UniFi LAN cross-routing) is documented under Firewall Baseline.
Pre-Phase: Prepare OPNsense Offline
Complete all configuration offline before inserting OPNsense at either site to minimise the production cutover window.
For each site (sa-fw-01 at Site A, sb-fw-01 at Site B):
- Install OPNsense on the appliance or VM.
- Configure WAN.
- Configure LAN trunk and VLAN interfaces.
- Configure DHCP and static reservations as needed.
- Configure the DNS resolver.
- Configure outbound NAT.
- Prepare WireGuard — do not depend on it yet.
- Test each VLAN with a laptop before the production cutover.
Test every VLAN before cutting over
Verify each VLAN's routing, DHCP, and DNS offline. A misconfigured VLAN scope is far faster to fix on the bench than during a live ISP cutover.
Phase 1 — Insert OPNsense at Site A
Topology before Phase 1:
ISP → UniFi Gateway Max → existing networkTopology after Phase 1:
ISP
|
OPNsense (sa-fw-01 on sa-edge-01)
|
OPNsense-side infrastructure / core network
|——— Proxmox / storage / backups
|
UniFi Gateway Max WAN (10.10.253.2/24, GW 10.10.253.1)
|
Existing UniFi LAN / Wi-Fi / user devicesSteps:
- Move the ISP handoff cable from the UniFi Gateway Max WAN port to the OPNsense WAN port.
- Connect the OPNsense LAN/transit/core port to the UniFi Gateway Max WAN port.
- Set the UniFi Gateway Max WAN to static
10.10.253.2/24, gateway10.10.253.1(OPNsense side of VLAN 253). - Confirm UniFi LAN, Wi-Fi, and user devices still have internet access.
- Connect Proxmox hosts to the OPNsense-side switch and VLANs.
- Confirm Proxmox internet access, DNS, and local routing.
Export the OPNsense config when Phase 1 is stable.
UniFi WAN Transit — Site A
| Address | Role |
|---|---|
10.10.253.0/24 | UniFi WAN transit — VLAN 253 |
10.10.253.1 | OPNsense transit gateway (sa-fw-01) |
10.10.253.2 | UniFi Gateway Max WAN |
NAT Model
User device → UniFi NAT → OPNsense NAT → Internet (double NAT — intentional)
Proxmox → OPNsense NAT → Internet (single hop, no UniFi)Double NAT is intentional
Existing UniFi routers remain in place to preserve Wi-Fi and user continuity. User devices traverse two NAT hops. This is a deliberate trade-off accepted by the project, not a misconfiguration.
Phase 2 — Insert OPNsense at Site B
Repeat the same procedure as Phase 1 using Site B addressing (10.20.x.x).
Topology after Phase 2:
ISP
|
OPNsense (sb-fw-01 on sb-edge-01)
|
OPNsense-side infrastructure / core network
|——— Proxmox / Ceph / Kubernetes
|
USG Pro WAN (10.20.253.2/24, GW 10.20.253.1)
|
Existing UniFi LAN / Wi-Fi / user devicesUniFi WAN Transit — Site B
| Address | Role |
|---|---|
10.20.253.0/24 | UniFi WAN transit — VLAN 253 |
10.20.253.1 | OPNsense transit gateway (sb-fw-01) |
10.20.253.2 | USG Pro WAN |
Steps mirror Phase 1 with Site B addressing. Export the OPNsense config when Phase 2 is stable.
Phase 3 — WireGuard Site-to-Site VPN
Build a routed WireGuard tunnel between sa-fw-01 and sb-fw-01. The WireGuard transit network is 10.255.0.0/24 (sa-fw-01 takes .1, sb-fw-01 takes .2).
- Configure WireGuard on
sa-fw-01: peer =sb-fw-01public key, endpoint = Site B WAN IP, allowed IPs =10.20.0.0/16and10.255.0.0/24. - Configure WireGuard on
sb-fw-01: peer =sa-fw-01public key, endpoint = Site A WAN IP, allowed IPs =10.10.0.0/16and10.255.0.0/24. - Route
10.10.0.0/16 ↔ 10.20.0.0/16through the tunnel. - Disable NAT for internal site-to-site traffic.
- Add firewall rules for management, Proxmox, backups, and selected services.
- Test ping, SSH, DNS, PBS, and selected app reachability across sites.
Export the OPNsense config from both sites when Phase 3 is stable.
Terminate the VPN on OPNsense — not UniFi
The WireGuard site-to-site tunnel must terminate on sa-fw-01 and sb-fw-01. Do not configure site-to-site VPN on the UniFi routers.
Phase 4 — Move Proxmox to Final VLAN 20 IPs
Migrate all Proxmox hosts from bootstrap IPs to their final VLAN 20 management addresses before clustering. Target subnets: 10.10.20.0/24 at Site A, 10.20.20.0/24 at Site B.
Do not create either Proxmox cluster until every node in the site has its final 10.x.20.x IP and /etc/hosts is correct.
SSH drops on network restart
Changing a Proxmox host's management IP terminates the SSH session. Keep an IPMI console open on each host throughout the IP migration so you retain out-of-band access if the network config does not apply cleanly.
See Proxmox Clusters for the full per-host IP table.
Phase 5 — Build Proxmox Clusters
Create the Proxmox clusters after every node in the site has its final VLAN 20 IP and /etc/hosts is correct.
- Site A — cluster name
sa-pve: create onsa-stor-01, then joinsa-edge-01,sa-cmp-01,sa-cmp-02. - Site B — cluster name
sb-pve: create onsb-cmp-01, then joinsb-edge-01,sb-cmp-02,sb-cmp-03,sb-cmp-04,sb-cmp-05.
One cluster per site — no cross-WAN clusters
Never stretch a Proxmox cluster across the WireGuard VPN. Corosync requires LAN-level latency; a WAN-stretched cluster risks quorum instability and split-brain. sa-pve stays at Site A; sb-pve stays at Site B.
Phase 6 — Storage, PBS, and Kubernetes
Stand up storage, PBS, and Kubernetes after both clusters are stable.
- Site A: ZFS mirror vdevs on
sa-stor-01; PBS-A management IP10.10.30.20, backup-data IP10.10.90.40on VLAN 90. - Site B: Ceph across all five nodes via
pveceph; PBS-B management IP10.20.30.20, backup-data IP10.20.90.40on VLAN 90. - Kubernetes/OpenShift nodes use VLAN 40 (
10.x0.40.0/22).
No stretched Ceph
Site B Ceph stays local. Do not stretch Ceph across the WireGuard VPN. Use PBS cross-site replication for disaster recovery.
See Proxmox Clusters and Kubernetes Planning for full detail.
Firewall Baseline
WAN Rules
- Block all inbound by default.
- Allow only WireGuard or explicitly required services inbound.
- Do not blindly forward all ports to the UniFi router.
OPNsense Admin Access
Restrict OPNsense admin UI (HTTPS) and SSH to VLAN 10 (Network Mgmt / IPMI) and VLAN 20 (Proxmox Management). Block access from guest, IoT, and unknown networks.
Proxmox Access Rules
| Direction | Protocol/Port | Action |
|---|---|---|
| VLAN 20 → Proxmox | TCP 8006 | Allow (web UI) |
| VLAN 20 → Proxmox | TCP 22 | Allow (SSH) |
| Proxmox → internet | various | Allow (DNS, NTP, repo updates) |
| Proxmox → PBS / storage | various | Allow |
| Guest / IoT → Proxmox | any | Block |
| WAN → Proxmox | any | Block unless via VPN |
| UniFi user LAN → Proxmox | any | Block unless explicitly needed |
OpenShift Access Rules
Allow only:
- Admin networks to the OpenShift API.
- Required ingress and load-balancer services to selected clients or networks.
- Cluster nodes to DNS, NTP, internet, and update endpoints.
Optional: UniFi LAN Cross-Routing
If OPNsense-side infrastructure needs to initiate or receive connections from existing UniFi LANs (not required for core operations):
- Add OPNsense static routes to the UniFi LAN subnets.
- Add UniFi static routes back to OPNsense and VPN networks.
- Add matching firewall rules on both sides.
- Verify source IP visibility — UniFi NAT behaviour may mask the original client IP.
Optional — not required for core infrastructure
Proxmox, PBS, Ceph, and Kubernetes do not require cross-routing into the UniFi user LAN. Only add this if a specific use case demands it.
Config Export Reminder
Export OPNsense config after every milestone
Navigate to System → Configuration → Backups and download the config after completing each phase. This enables rapid recovery without reconfiguring from scratch if a later change corrupts the firewall state.
Related Pages
- OPNsense Overview — topology decision and NAT model
- UniFi Handoff — detailed OPNsense/UniFi/Proxmox handoff runbook
- VLAN Reference — full VLAN table with subnets and gateways
- Proxmox Clusters — cluster setup, node IPs, Corosync rules
- Kubernetes Planning — K8s node networks, VIPs, pod CIDRs
- Build Phases — full ordered build sequence across both sites
Firewall / OPNsense
OPNsense's role as the infrastructure router and firewall at each site, VM placement constraints, the intentional double-NAT UniFi topology, and the six-phase migration approach.
OPNsense / UniFi / Proxmox Handoff
How OPNsense fronts the demoted UniFi routers at each site: VLAN 253 transit wiring, the double-NAT model for user devices, Proxmox network zone placement, firewall zone rules, and optional cross-zone routing.