Site B Port Map
Authoritative switch port map for Site B: sb-sw-01 (Netgear XS748T 48-port 10 Gb core) and sb-sw-02 (UniFi USW 24 PoE access), with per-node NIC-to-VLAN wiring for all five compute nodes and the edge device.
sb-sw-01 (Netgear XS748T, 48-port 10 Gb core) carries all infrastructure VLANs for Site B; sb-sw-02 (UniFi USW 24 PoE) handles IPMI, APs, and client access. This page records every port assignment on sb-sw-01, per-node NIC wiring for all six Site B hosts, and the known device assignments on sb-sw-02.
L2 core only — no routing on sb-sw-01
sb-sw-01 is L2 only: no routing, no DHCP. All inter-VLAN routing is handled by OPNsense on sb-edge-01. sb-sw-02 carries only VLANs 10, 20, 100, 110, and 120 — never VLANs 25, 30, 40, 50, 60, 65, 70, 80, or 90.
Per-Node NIC Wiring
Standard compute nodes — sb-cmp-01 through sb-cmp-05
All five compute nodes follow this identical NIC assignment. Each node carries one Intel X710-T4 (4 × 10GBASE-T) for tagged data trunks. The FN8TP onboard SFP+ ports are unused because sb-sw-01 has only four SFP+ combo slots.
| NIC | Connects to | VLANs / Role |
|---|---|---|
| Onboard 1G #1 | sb-sw-01 (Access) | 20 Proxmox Management |
| Onboard 1G #2 | sb-sw-01 (Access) | 25 no GW Corosync heartbeat — dedicated, site-local |
| Onboard 10GBASE-T | sb-sw-01 (Trunk) | 65 no GW Ceph cluster / OSD replication — site-local |
| X710-T4 port 1 | sb-sw-01 (Trunk) | 30 VM Services |
| X710-T4 port 2 | sb-sw-01 (Trunk) | 40 50 Kubernetes Nodes + K8s LB / VIPs |
| X710-T4 port 3 | sb-sw-01 (Trunk) | 60 Storage / Ceph public |
| X710-T4 port 4 | sb-sw-01 (Trunk) | 90 Backup / Replication |
| Dedicated IPMI | sb-sw-02 (Access) | 10 Network Mgmt / IPMI |
Ceph network split — decided 2026-06-06
Ceph public traffic (clients and VMs to OSDs) runs on VLAN 60 via the X710-T4. Ceph cluster traffic (OSD replication, backfill, heartbeat) runs on VLAN 65 via each node's onboard 10GBASE-T — isolating recovery traffic from the client path. VLAN 65 has no gateway and is site-local only. See VLAN Reference.
sb-edge-01 NIC Wiring
sb-edge-01 (Supermicro SYS-E200-8D, running OPNsense VM sb-fw-01) differs from the compute node pattern: onboard 1G #1 is the WAN uplink to the ISP, so management and Corosync shift to the remaining interfaces.
| NIC | Connects to | VLANs / Role |
|---|---|---|
| Onboard 1G #1 | Site B ISP handoff (WAN) | OPNsense WAN — 1 Gbps; not on any switch |
| Onboard 1G #2 | sb-sw-01 port 2 (Access) | 20 Proxmox Management |
| Onboard 10G #1 | sb-sw-01 port 3 (Access) | 25 no GW Corosync heartbeat |
| Onboard 10G #2 | sb-sw-01 port 1 (Trunk) | All VLANs — OPNsense LAN, VLAN-aware Linux bridge |
| Dedicated IPMI | sb-sw-02 (Access) | 10 Network Mgmt / IPMI |
sb-sw-01 — Netgear XS748T Core Switch
48-port 10 Gb switch. Uses 41 of 48 ports: 3 for sb-edge-01, 7 per compute node (5 nodes = 35 ports), plus 3 for the sb-sw-02 uplink, demoted USG Pro WAN, and emergency/spare. All IPMI connections land on sb-sw-02, not on sb-sw-01.
| Port | Device / NIC | Mode | VLANs |
|---|---|---|---|
| 1 | sb-edge-01 OPNsense LAN (onboard 10G #2) | Trunk | Tagged 10 20 25 30 40 50 60 65 70 80 90 100 110 120 253 |
| 2 | sb-edge-01 Proxmox mgmt (onboard 1G #2) | Access | Untagged 20 |
| 3 | sb-edge-01 Corosync (onboard 10G #1) | Access | Untagged 25 |
| 4 | sb-cmp-01 mgmt (onboard 1G #1) | Access | Untagged 20 |
| 5 | sb-cmp-01 Corosync (onboard 1G #2) | Access | Untagged 25 |
| 6 | sb-cmp-01 Ceph cluster (onboard 10GBASE-T) | Trunk | Tagged 65 |
| 7 | sb-cmp-01 X710-T4 port 1 — VM Services | Trunk | Tagged 30 |
| 8 | sb-cmp-01 X710-T4 port 2 — K8s | Trunk | Tagged 40 50 |
| 9 | sb-cmp-01 X710-T4 port 3 — Ceph public | Trunk | Tagged 60 |
| 10 | sb-cmp-01 X710-T4 port 4 — backup | Trunk | Tagged 90 |
| 11 | sb-cmp-02 mgmt (onboard 1G #1) | Access | Untagged 20 |
| 12 | sb-cmp-02 Corosync (onboard 1G #2) | Access | Untagged 25 |
| 13 | sb-cmp-02 Ceph cluster (onboard 10GBASE-T) | Trunk | Tagged 65 |
| 14 | sb-cmp-02 X710-T4 port 1 — VM Services | Trunk | Tagged 30 |
| 15 | sb-cmp-02 X710-T4 port 2 — K8s | Trunk | Tagged 40 50 |
| 16 | sb-cmp-02 X710-T4 port 3 — Ceph public | Trunk | Tagged 60 |
| 17 | sb-cmp-02 X710-T4 port 4 — backup | Trunk | Tagged 90 |
| 18 | sb-cmp-03 mgmt (onboard 1G #1) | Access | Untagged 20 |
| 19 | sb-cmp-03 Corosync (onboard 1G #2) | Access | Untagged 25 |
| 20 | sb-cmp-03 Ceph cluster (onboard 10GBASE-T) | Trunk | Tagged 65 |
| 21 | sb-cmp-03 X710-T4 port 1 — VM Services | Trunk | Tagged 30 |
| 22 | sb-cmp-03 X710-T4 port 2 — K8s | Trunk | Tagged 40 50 |
| 23 | sb-cmp-03 X710-T4 port 3 — Ceph public | Trunk | Tagged 60 |
| 24 | sb-cmp-03 X710-T4 port 4 — backup | Trunk | Tagged 90 |
| 25 | sb-cmp-04 mgmt (onboard 1G #1) | Access | Untagged 20 |
| 26 | sb-cmp-04 Corosync (onboard 1G #2) | Access | Untagged 25 |
| 27 | sb-cmp-04 Ceph cluster (onboard 10GBASE-T) | Trunk | Tagged 65 |
| 28 | sb-cmp-04 X710-T4 port 1 — VM Services | Trunk | Tagged 30 |
| 29 | sb-cmp-04 X710-T4 port 2 — K8s | Trunk | Tagged 40 50 |
| 30 | sb-cmp-04 X710-T4 port 3 — Ceph public | Trunk | Tagged 60 |
| 31 | sb-cmp-04 X710-T4 port 4 — backup | Trunk | Tagged 90 |
| 32 | sb-cmp-05 mgmt (onboard 1G #1) | Access | Untagged 20 |
| 33 | sb-cmp-05 Corosync (onboard 1G #2) | Access | Untagged 25 |
| 34 | sb-cmp-05 Ceph cluster (onboard 10GBASE-T) | Trunk | Tagged 65 |
| 35 | sb-cmp-05 X710-T4 port 1 — VM Services | Trunk | Tagged 30 |
| 36 | sb-cmp-05 X710-T4 port 2 — K8s | Trunk | Tagged 40 50 |
| 37 | sb-cmp-05 X710-T4 port 3 — Ceph public | Trunk | Tagged 60 |
| 38 | sb-cmp-05 X710-T4 port 4 — backup | Trunk | Tagged 90 |
| 39 | sb-sw-02 (UniFi USW 24 PoE) uplink | Trunk | Tagged 10 20 100 110 120 |
| 40 | USG Pro WAN (demoted — VLAN 253 transit) | Access | Untagged 253 |
| 41 | Emergency admin / spare | Access | Untagged 10 or 20 |
USG Pro WAN port — discrepancy resolved 2026-07-03
Port 40 is the demoted USG Pro WAN connection, per the vault/14 port table. An earlier discrepancy (the vault/14 WAN/Edge prose said port 29 — actually sb-cmp-04 X710-T4 port 2) was corrected in the vault on 2026-07-03; table and prose now agree on port 40.
sb-sw-02 — UniFi USW 24 PoE Access Switch
sb-sw-02 handles IPMI / BMC for all six Supermicro nodes, PoE for APs, and client/lab access. It uplinks to sb-sw-01 port 39. The USG Pro (demoted) provides existing Wi-Fi users and client LAN; its WAN interface connects to sb-sw-01 port 40 (VLAN 253), and its LAN side continues to serve users under double NAT — this is intentional.
The vault does not document specific port numbers within sb-sw-02. The table below records known device assignments by VLAN role.
| Device | Mode | VLAN | Purpose |
|---|---|---|---|
sb-edge-01 IPMI | Access | 10 | Network Mgmt / IPMI |
sb-cmp-01 IPMI | Access | 10 | Network Mgmt / IPMI |
sb-cmp-02 IPMI | Access | 10 | Network Mgmt / IPMI |
sb-cmp-03 IPMI | Access | 10 | Network Mgmt / IPMI |
sb-cmp-04 IPMI | Access | 10 | Network Mgmt / IPMI |
sb-cmp-05 IPMI | Access | 10 | Network Mgmt / IPMI |
Uplink → sb-sw-01 port 39 | Trunk | 10 20 100 110 120 | Core switch uplink |
| APs / client access ports | Access / Trunk | 100 110 120 | Lab / Trusted Client, IoT, Guest WiFi |
IPMI must not be exposed to the internet
All BMC / IPMI interfaces connect only to sb-sw-02 on VLAN 10 (Network Mgmt / IPMI). This VLAN is unreachable from the internet and must never be routed across the WireGuard tunnel.
VLAN 253 — UniFi WAN Transit
VLAN 253 is tagged on sb-edge-01's OPNsense LAN trunk (port 1) so OPNsense can provide the WAN gateway (10.20.253.1) to the demoted USG Pro. The USG Pro WAN port lands on sb-sw-01 port 40, configured as an untagged VLAN 253 access port. The USG Pro LAN side continues to serve existing Wi-Fi users and clients under double NAT — this is intentional.
The cutover from the current USG Pro WAN arrangement to OPNsense-as-upstream is a physical cable move at Site B Phase 2: relocate the single ISP handoff cable from the USG Pro to sb-edge-01's onboard 1G #1, then feed the USG Pro its WAN from OPNsense via VLAN 253.
Related Pages
- Site A Port Map — equivalent reference for
sa-sw-01,sa-sw-02, andsa-sw-03 - Switch Operations — VLAN provisioning and switch management procedures
- VLAN Reference — all 15 VLANs with subnets and gateway assignments
- Per-Site Inventory — NIC models, RAM, and storage per host
- IP Tables — per-host IP addresses for all VLANs
Site A Port Map
Authoritative per-port wiring reference for sa-sw-01 (Netgear XS716T core), sa-sw-02, and sa-sw-03 (access), including VLAN assignments, dedicated Corosync arrangement, SFP+ DAC uplinks, and sa-ap-01 PoE++ port.
Switch Operations
Operational guide for the Netgear XS716T/XS748T core switches and UniFi access switches: roles, management addresses, VLAN assignments, jumbo frame policy, and common operational gotchas.