Switching & Cabling
Switch tier roles, VLAN carry rules, and cabling policy for Site A and Site B. Netgear 10 Gb core switches handle server and infrastructure traffic; access switches handle IPMI, APs, and user devices.
Two switching tiers run at each site: a Netgear 10 Gb core switch for Proxmox, Corosync, storage, Kubernetes, and VM traffic; and one or more access switches for IPMI, management offload, APs, and user devices. This section documents which switch belongs to which tier, which VLANs each uplink carries (and must never carry), and the physical cabling constraints.
Switch Tiers
Core Switches — sa-sw-01 / sb-sw-01
sa-sw-01 (Netgear XS716T, 16-port) at Site A and sb-sw-01 (Netgear XS748T, 48-port) at Site B are the 10 Gb core switches. Both operate L2 only — no routing, no DHCP. They terminate the OPNsense LAN trunk and carry all server-facing infrastructure VLANs: Proxmox management, Corosync heartbeat, VM Services, Kubernetes, storage, backup, monitoring, and DMZ. Site B's sb-sw-01 additionally carries VLAN 65 (Ceph cluster) on dedicated per-node ports.
No routing or DHCP on core switches
sa-sw-01 and sb-sw-01 are strictly L2. OPNsense (sa-fw-01 / sb-fw-01) owns all inter-VLAN routing and DHCP. Never configure a Layer 3 gateway or DHCP scope on either Netgear core switch.
Access Switches
Access switches carry only management and user-side VLANs. They do not see server, storage, Corosync, Kubernetes, or infrastructure VLANs.
| Switch | Site | Hardware | Primary Role |
|---|---|---|---|
sa-sw-02 | Site A | Access switch (4× 10G RJ45 PoE++ 802.3bt, 2× 10G SFP+) | Proxmox mgmt + IPMI for sa-edge-01 and sa-stor-01 |
sa-sw-03 | Site A | Access switch (4× 10G RJ45 PoE++ 802.3bt, 2× 10G SFP+) | Proxmox mgmt for sa-cmp-01, sa-cmp-02; sa-ap-01 (WiFi 7 AP, PoE++) |
sb-sw-02 | Site B | UniFi USW 24 PoE | IPMI for all Site B nodes; APs and clients |
sa-sw-02 and sa-sw-03 uplink to sa-sw-01 via 10G SFP+ DAC cables (combo slots 15F and 16F), freeing the RJ45 PoE++ ports for sa-ap-01 and a future second AP. sb-sw-02 uplinks to sb-sw-01 port 39 over copper.
VLAN Carry Rules
Access-switch uplinks
All three access switches carry the same restricted VLAN set on their uplinks:
| VLAN | Name |
|---|---|
| 10 | Network Mgmt / IPMI |
| 20 | Proxmox Management |
| 100 | Lab / Trusted Client |
| 110 | IoT |
| 120 | Guest WiFi |
Access-switch uplinks must never carry server or infrastructure VLANs
VLANs 25 30 40 50 60 65 70 80 90 must not appear on sa-sw-02, sa-sw-03, or sb-sw-02 uplinks. Corosync, VM Services, Kubernetes, storage, Ceph, DMZ, monitoring, and backup/replication traffic stays on the core switch only.
Core-switch port assignments
The core switch connects directly to each host NIC with per-purpose access or trunk ports. Full per-port VLAN assignments are in the per-site port maps linked below.
Cabling Policy
All host-to-switch links are single copper connections — no LACP. Cabling is Cat6a throughout both sites.
Site A: Proxmox mgmt and IPMI offloaded from the core switch
At Site A, Proxmox management (VLAN 20) and IPMI (VLAN 10) for all four nodes run through sa-sw-02 / sa-sw-03, not sa-sw-01. This freed four Netgear ports, allowing sa-edge-01 to have a dedicated Corosync link (VLAN 25) on sa-sw-01 port 2. sa-stor-01 (one onboard 1GbE, Aquantia AQC107 onboard 10G avoided for critical traffic) and the ThinkPads (no spare NIC) carry Corosync tagged alongside other VLANs on their existing data trunks. At Site B, node management connects directly to sb-sw-01.
In This Section
- Site A Port Map — full
sa-sw-0116-port table,sa-sw-02/sa-sw-03access-port allocation, and per-NIC VLAN assignments for all Site A nodes - Site B Port Map — full
sb-sw-0148-port layout with per-node NIC wiring andsb-sw-02access wiring - Switch Operations — Netgear factory reset, firmware update procedure, fan noise, and UniFi client/network deletion gotchas
Interactive tools
The vault/interactive/port-wiring-guide.html and vault/interactive/switch-faceplate-wiring.html tools provide phase-by-phase visual cabling maps and per-port NIC/VLAN highlighting for both sites. See Build Phases for the activation sequence.
E200 Fan Quieting
Runbook to quiet the Supermicro SYS-E200-8D 40 mm chassis fan on sa-edge-01 and sb-edge-01 using IPMI fan mode control, BMC threshold tuning, and an optional Noctua fan swap.
Site A Port Map
Authoritative per-port wiring reference for sa-sw-01 (Netgear XS716T core), sa-sw-02, and sa-sw-03 (access), including VLAN assignments, dedicated Corosync arrangement, SFP+ DAC uplinks, and sa-ap-01 PoE++ port.