Packet-Flow Troubleshooting
Layered diagnosis for representative client, management, storage, DNS, denied-zone, and future cross-site packet paths.
Trace one failed packet story from source to destination. The method separates wrong-phase addressing, access/trunk errors, gateways, firewall policy, NAT, DNS, reverse paths, and service failures without opening the network as a test.
Diagnose Before Changing
Capture the failing tuple and current state before editing a switch profile or
OPNsense rule. Never add Allow All, a convenience gateway on VLAN 25/60, or a
broad /16 pass to discover whether policy is the cause.
Define One Packet Story
Record the source device, port or SSID, VLAN and IP/CIDR; destination hostname, IP and expected VLAN; protocol and destination port; expected policy result; current phase; and the last hop where the packet is observed.
If the expected result is not in the Routing and Firewall Policy Matrix, stop. The missing policy decision is the problem; do not debug an undefined allow.
Isolation Order
| Layer | Question | Discriminating evidence |
|---|---|---|
| 1 | Is the endpoint on the intended port or SSID? | Cable/AP association, link state, switch port identity |
| 2 | Did it enter the intended VLAN? | Access/native/tagged profile, client address, neighbor table |
| 3 | Is local addressing correct? | IP/CIDR, duplicate detection, gateway, DNS options, lease owner |
| 4 | Does same-VLAN traffic work? | Peer/neighbor reachability without OPNsense |
| 5 | Does the local gateway work? | Routed VLAN .1; no gateway expected on 25/60 |
| 6 | Did OPNsense receive and permit it? | Ingress-interface firewall log/state and packet capture |
| 7 | Did it leave through the expected egress? | Egress capture, route, NAT state, or WireGuard interface |
| 8 | Did the destination receive and answer? | Destination listener/capture and reverse route |
Advance only after the current layer is proven. A service working from its Proxmox host does not prove that a laptop has a return path.
Representative Site A Flows
| Flow | Expected path | First checks |
|---|---|---|
| Trusted client → internet | VLAN 100 port/SSID → UniFi fabric → OPNsense VLAN 100 → staged UCG WAN through A7 or direct ONT after A8 | Lease/router/DNS, VLAN 100 profile, ingress rule, outbound NAT, phase-correct WAN lease |
| Named admin → Proxmox | VLAN 100 admin alias → OPNsense → VLAN 20 host | Alias membership, TCP service rule, ingress log, host listener and return via 10.10.20.1 |
| Host → storage | Host VLAN 60 interface → UniFi fabric → storage VLAN 60 interface | Direct IP/CIDR, tagged/access carriage, neighbor resolution; OPNsense must not appear |
| Client → DNS during A4-A9 | Client VLAN → local OPNsense interface | DHCP DNS option, local resolver listener, client-to-gateway path |
| Client → DNS after A10 | Client VLAN → OPNsense → 10.10.30.10/.11 | DHCP option cutover, VLAN 30 reachability, TCP/UDP 53 rule, Technitium A/PTR answer |
| Guest/IoT → management | Client VLAN → OPNsense block | Correct source VLAN and expected deny log |
| Site A → Site B service | Source VLAN → sa-fw-01 → WireGuard → sb-fw-01 → destination | Later campaign only: peer, summary routes, two-sided policy, preserved source, return route |
Capture Points
Use the smallest set that proves where the packet stops:
- source endpoint;
- OPNsense ingress interface for routed flows;
- OPNsense egress or WireGuard interface after ingress is proven;
- destination interface or host; and
- the reverse direction when the request arrives but the response does not.
A focused standard Unix-like capture is:
tcpdump -ni <interface> host <peer-ip> and port <service-port>Use OPNsense packet capture and live firewall logs on the exact ingress interface. No log entry often means the packet never reached that interface; it does not prove a firewall block.
Symptom Map
| Symptom | Most likely boundary | Next proof |
|---|---|---|
Final 10.10.x address fails before its gate | Phase mismatch | Check the address valid for the current phase |
| Same-VLAN peer and gateway both fail | Link, access VLAN, address, or mask | Port profile, IP/CIDR, neighbor resolution |
| Same-VLAN works but routed destinations fail | Gateway, OPNsense trunk/interface, or rule | Local .1, ingress capture, explicit LAN-vNIC tag list |
| One VLAN fails across the fabric | Missing tag or wrong access profile | Compare every hop with the Site A port map |
| IP works but hostname fails | DNS option, resolver, zone, or record | Query the expected resolver; test A and PTR separately |
| Cross-VLAN works but internet fails | WAN route/lease or outbound NAT | WAN state, default route, NAT state, A7/A8 upstream |
| Request arrives but no reply returns | Destination service or reverse path | Listener, host firewall, destination gateway, reverse capture |
| Host reaches a VM but laptop cannot | VM default gateway or client route | Compare route tables and return via the correct .1 |
| Only remote-site traffic fails | WireGuard peer, summary route, or two-sided policy | Transit peer, both routes, capture before/after tunnel |
| Intermittent loss after fabric change | RSTP or physical link | Root/alternate state, link errors, one-DAC validation |
Evidence-Backed Conclusions
End with one classification: source attachment defect, Layer 2 carriage defect, addressing/IPAM defect, routing/policy defect, NAT/WAN defect, DNS defect, destination/reverse-path defect, expected deny, or undefined policy.
Record the capture points and smallest corrective change. After a fix, rerun the original probe and its adjacent negative test from the Network Validation Runbook.