How It All Fits Together

AORXI is designed as two independent private-cloud sites joined by routed WireGuard. Each site keeps its own WAN edge, Layer 2 fabric, Proxmox quorum, storage, and local recovery path. Site A is the active build; Site B remains a parked design.

Read the State Before the Target

As of 2026-07-26, the working environment is still the Site A bootstrap island. The final architecture is approved and documented, but most of it is not yet deployment evidence.

The End-to-End Shape

The diagram shows the approved end state. Site A reaches it without rebuilding the LAN during the Internet handoff: the entire final LAN is commissioned behind OPNsense while the UCG temporarily supplies the firewall's WAN.

Site A: Bootstrap, Commissioning, Then Final WAN

The WAN-only A8 handoff happens after every host, service, IPMI interface, AP, SSID, and client passes on its final network. Rollback restores the UCG upstream path; it does not readdress the LAN or move endpoints back to the flat network. After the stability window, the UCG Max and XG6 are powered down and retained as labeled rollback equipment.

During commissioning, the XG6 remains exclusively on the UCG side. Its p4 copper lead services one host or BMC at a time; p5/p6 stay disconnected. No XG10 ever connects to UCG Max or XG6. A3 adopts the switches on final VLAN 10 and starts the target as a two-DAC tree: sw1 p11 ↔ sw2 p11 and sw1 p12 ↔ sw3 p12. A6 adds sw2 p12 ↔ sw3 p11 only after RSTP profiles and priorities are verified.

The final fabric is an RSTP triangle: sa-sw-01 p11 ↔ sa-sw-02 p11, sa-sw-02 p12 ↔ sa-sw-03 p11, and sa-sw-03 p12 ↔ sa-sw-01 p12. These three 10 Gb SFP+ links provide switch-link redundancy only. They do not provide stacking, MLAG, LACP, router HA, or a replacement for sa-fw-01. RSTP priorities are 4096, 8192, and 12288 for sa-sw-01, sa-sw-02, and sa-sw-03 respectively. sa-ap-01 terminates on sa-sw-03 p6 with 10 Gb RJ45 and PoE++.

Site B: Separate Edge and Retained User Branch

Site B keeps the USG Pro user-side branch and its intentional double NAT on VLAN 253. Infrastructure stays directly behind OPNsense and the Netgear L2 core; the UniFi access switch does not carry the forbidden infrastructure VLANs. This is approved design, not deployed state.

Between Sites: Routed Services, Never a Shared Failure Domain

The planned WireGuard transit is 10.255.0.0/24: Site A uses 10.255.0.1, Site B uses 10.255.0.2, and OPNsense routes 10.10.0.0/16 ↔ 10.20.0.0/16. It will carry selected management, DNS replication, PBS replication, UniFi adoption traffic, monitoring, and OpenBao unseal traffic.

What Owns Each Layer

Explicit ownership prevents two systems from trying to control the same resource.

Physical Hosts and Site Roles

Site A concentrates management and durable services. Site B concentrates distributed storage and worker-heavy compute.

The E200s reserve capacity for routing and light control services. DNS helpers, the UniFi controller, OpenBao, a small reverse proxy, and monitoring agents fit; heavy databases, Ceph OSDs, storage-heavy VMs, and heavy Kubernetes workers do not. sa-edge-01 remains a deliberate single point of failure for Site A routing and its colocated control services.

The Network Planes

Both sites reuse the same VLAN IDs where the function is shared. Site A uses 10.10.x.0; Site B uses 10.20.x.0. The exceptions are the Site B-only VLAN 253 legacy transit and the Site A-only VLAN 999 native sink.

On every routed /24, .1 is OPNsense, .2–.9 is network infrastructure, .10–.39 is physical hosts, .40–.49 is infrastructure VMs, .50–.199 is DHCP or static services, and .200–.254 is VIP space. Proxmox hosts hold L3 addresses only on infrastructure VLANs they terminate: 10, 20, 25, 60, 65, and 90. Guest and VM VLANs are bridged without a host IP.

Edge Services and Startup Dependencies

sa-edge-01 carries the minimum services needed to establish and manage the Site A control plane.

The controller is not in the forwarding path: adopted switches and APs continue to forward if UOS is down, while configuration and telemetry pause. OpenBao is also outside the runtime packet path; a sealed instance blocks normal config lookups, not an already-applied OPNsense configuration. OPNsense must boot before OpenBao because the future cross-site tunnel becomes the transit-unseal path.

Provisioning, Configuration, and Secrets

The repository separates object creation from in-system configuration.

Ansible does not manage a competing host firewall. One manager per layer avoids ruleset drift and lockout.

Secrets follow a two-tier bootstrap boundary:

Consumers read OpenBao first and fall back to encrypted Ansible vault files or .env.local. Unset BAO_ADDR selects pure fallback mode. Tier 0 never moves into OpenBao: the secrets needed to rebuild and unseal OpenBao cannot depend on OpenBao itself. Each site eventually runs an independent single-node Raft instance; snapshots, not Raft quorum, cross sites.

Storage, Backup, DNS, and Certificates

Storage and Recovery

• Site A: sa-stor-01 targets 8–12 Samsung SM863 1.92 TB SSDs in ZFS mirror vdevs. PBS-A uses 10.10.30.20 for management and 10.10.90.40 for backup data.
• Site B: five compute nodes target 20–30 local Ceph OSDs with replication size 3. VLAN 60 is the client-facing public network; VLAN 65 is isolated OSD replication/backfill. Ceph is pinned to the Proxmox tentacle no-subscription repository and excluded from unattended upgrades.
• Cross-site DR: each site backs up locally first. PBS replication and OpenBao Raft snapshots later cross WireGuard on the routed backup path.

Naming and Certificates

OPNsense Unbound provides the initial resolver. The A10 target adds four Technitium DNS VMs on VLAN 30: sa-dns-01 (10.10.30.10) is primary, sa-dns-02 (10.10.30.11) is the local secondary, and sb-dns-01/02 (10.20.30.10/.11) are read-only secondaries over WireGuard. Site B serves its last replicated zones locally if the tunnel fails.

The internal zone is core.aorxi.io; the public domain is aorxi.io on Cloudflare. Certificates use Let's Encrypt with Cloudflare DNS-01 for *.core.aorxi.io and *.aorxi.io. No private CA is required.

Kubernetes and OpenShift Fit Above the Foundation

Kubernetes/OpenShift comes after stable routing, storage, backups, and DNS. VLAN 40 carries node networks (10.10.40.0/22, 10.20.40.0/22); VLAN 50 carries API, ingress, and LoadBalancer VIPs. The planned stack is Cilium in overlay mode, MetalLB, cert-manager, external-dns, ArgoCD, and ingress-nginx or Traefik.

Site A OpenShift machine networks begin at 10.10.128.0/22, then 10.10.132.0/22 and 10.10.136.0/22, with 10.10.144.0/20 reserved for expansion. They deliberately avoid trusted-client VLAN 100 (10.10.100.0/22). Site B retains 10.20.100.0/22, 10.20.104.0/22, 10.20.108.0/22, and 10.20.112.0/21 expansion.

Pod and service CIDRs remain separate from physical LAN and VPN ranges. Examples include pod blocks 10.128.0.0/14, 10.132.0.0/14, and service blocks such as 172.30.0.0/16 within 172.16.0.0/12. 172.32.x.x is not RFC 1918 and must not be used.

The Active Site A Campaign

The A0–A11 plan is the execution spine. Each gate produces cable and port evidence, previews/check-mode output, positive and negative reachability tests, and fresh configuration exports where relevant.

Design Invariants and Accepted Limits

Known tradeoffs remain visible: sa-edge-01 is a routing/control-plane single point of failure; Site A Corosync shares links on some hosts while Site B uses dedicated links; the ThinkPads have no IPMI; sa-bao-01 needs manual Shamir unseal until Site B exists; and the UCG creates accepted temporary double NAT during Site A commissioning. None of these weak points changes the ownership or failure-domain rules above.

Continue into the Details