sa-edge-01 Bootstrap Runbook

This is the per-node version of Initial site bootstrap for sa-edge-01. It starts with bare metal, installs and hardens Proxmox, brings up sa-fw-01, sa-uos-01, and sa-bao-01, then moves consumers to bao-first secrets. The runbook ends when every Phase 0 gate passes. Switch adoption and the production cutover follow in Build phases and Migration phases.

The Node at a Glance

The bridge model was settled on 2026-07-03. On-host verification on 2026-07-05 confirmed the stable custom names nic0 through nic3 and their role comments in /etc/network/interfaces.

During bootstrap, nic0 reaches the UCG only through XG6 p4, nic3 reaches the staged OPNsense WAN through XG6 p3, and nic1/nic2 remain disconnected until the target fabric phases. IPMI uses p4 for first contact, then disconnects until its direct A5 move to sw2 p5.

vmbr1 through vmbr3 are managed by the HostBridges component in platform/ (implemented and applied on 2026-07-05). vmbr0 is installer-owned and never automated.

VM vNIC map: sa-fw-01 net0 (WAN) = vmbr1, net1 (LAN/mgmt) = vmbr2; sa-uos-01 and sa-bao-01 bridge via vmbr0.

Stage 0: Workstation Prerequisites (Tier 0)

Everything needed to (re)build the node lives on the workstation + password manager before the node exists (the tier-0 boundary from Secrets: OpenBao).

1. Clone the repo; cp .env.local.example .env.local && chmod 600 .env.local.
2. Fill tier-0 values: PULUMI_CONFIG_PASSPHRASE, ANSIBLE_VAULT_PASSWORD. The Proxmox API credential exists only after Stage 2: circle back. On a full-site rebuild leave the BAO_* knobs empty (pure fallback mode until Stage 7).
3. ssh-add the operator key: the Pulumi Proxmox provider authenticates over SSH via the agent only and ignores ~/.ssh/config; an empty agent fails cryptically mid-apply.
4. Run make install to install dependencies for every subproject.
5. On the first Pulumi run from a workstation, run uv run pulumi stack init site-a once inside each provision project. The local file backend does not create stacks automatically.

Gate 0: make help prints targets; ./scripts/vault-pass.sh prints, not errors; ssh-add -L non-empty.

Stage 1: Hardware + IPMI First Contact

1. Physical install per the hardware allocation: 512 GB M.2 boot drive, RDIMMs, no add-in NIC (the four onboard ports cover all roles).
2. Label every jack against the final combined port wiring and faceplate map, but cable only the links required by the current phase. All links below use Cat6a copper without LACP.

When unsure which physical jack is which NIC, ethtool -p <iface> 60 blinks the port LED. 3. Connect IPMI to XG6 p4 before any other setup. Change ADMIN/ADMIN (or the sticker password) immediately, then set static address 192.168.0.10/23 with gateway 192.168.0.1. Record the test, disconnect IPMI, and attach nic0 to p4 for the continuing edge bootstrap. Never expose IPMI to the internet. For macOS iKVM problems, see IPMI / KVM.

Gate 1: The IPMI web UI is reachable at 192.168.0.10 from a temporary-LAN client with rotated credentials, the result is recorded, and the BMC is then disconnected from p4. The client must be on the same flat /23; no routing is involved, and the interface must never be reachable from the internet. If it does not answer, check the client's mask first because a /24-masked host cannot reach the 192.168.0.x half.

Stage 2: Proxmox VE on Bare Metal

1. Install PVE to the M.2 drive through IPMI virtual media. Use FQDN sa-edge-01.core.aorxi.io, IP 192.168.1.10/23, and gateway/DNS 192.168.0.1. The installer binds 1G #1 to vmbr0; keep this management bridge.
2. Create the Pulumi API credential first in the PVE UI (Datacenter → Permissions → API Tokens, privilege separation OFF) → PROXMOX_VE_API_TOKEN in .env.local. Step 3 cannot run without it (step order fixed 2026-07-05).
3. Create vmbr1 through vmbr3 with the Pulumi-managed HostBridges component in platform/ (implemented and applied on 2026-07-05). The aorxi:bridges stack configuration maps vmbr1 to nic3 for WAN, vmbr2 to nic2 for the VLAN-aware LAN trunk, and vmbr3 to nic1 for Corosync. The component refuses to manage installer-owned vmbr0.

# config is already armed (bridges.enabledNodes: [sa-edge-01], committed 2026-07-05).
# On a REBUILD over existing stack state, refresh first:
#   cd platform && uv run pulumi stack select site-a && uv run pulumi refresh --yes
make platform-preview STACK=site-a   # expect the 3 Bridge creates (+ component wrapper);
                                     # any update/replace/delete on OTHER resources = stop
make platform-up STACK=site-a        # PVE console reachable; someone near the hardware

Gate 2: PVE UI on https://192.168.1.10:8006; root SSH works with the agent key; vmbr0-vmbr3 exist and vmbr2 is VLAN-aware.

Stage 3: Baseline + Hardening (Host OS Only)

The baseline/ Ansible project owns the host OS; Pulumi owns everything API-side.

make baseline-ping                        # SSH reachability
make baseline-check  LIMIT=sa-edge-01     # dry-run harden — read the diff
make baseline-apply  LIMIT=sa-edge-01     # baseline-pve.yml
make baseline-harden LIMIT=sa-edge-01     # harden-pve.yml (STIG-mapped)

After the apply, confirm that the node reboots cleanly and root SSH still works with keys. Keep PermitRootLogin without-password because Ansible and Pulumi require it. Run make platform-preview STACK=site-a to exercise the provider against a stack that declares resources; the OPNsense stack remains inert until Stage 4 and could otherwise produce a false pass. A repeated baseline-check must report 0 changes, and node_exporter must listen on the management IP rather than 0.0.0.0.

Gate 3: clean 0-change re-run; PVE UI up; pulumi preview connects.

Stage 4: sa-fw-01: OPNsense, Staged on Vmbr1

A pinned FreeBSD cloud image, cloud-init seed config.xml, and in-place opnsense-bootstrap conversion produce an API-ready VM on the seed management IP. Under the vmbr1-direct staging path (amended 2026-07-04), the ONT stays on the Gateway Max while OPNsense receives a temporary WAN from the temporary LAN through the jack the ONT will later occupy. Staging causes no user downtime.

1. Staging cable: plug onboard 10G #1 (nic3, the vmbr1 jack) into XG6 p3 on the UCG bootstrap LAN. On 2026-07-05, nic3 linked at 1 Gbps to the Gateway Max-side LAN.
2. Set the stack config. Release 26.1 and the FreeBSD 14.3 ZFS image are a matched pair and must not drift independently. wanBridge defaults to vmbr1.

ROOT_HASH="$(make -s opnsense-root-hash)"   # prompts twice; hash goes into the shell var
make opnsense-provision-set STACK=site-a \
     OPNSENSE_MGMT_IP=192.168.1.41 \
     OPNSENSE_ROOT_HASH="$ROOT_HASH"
make opnsense-provision-preview STACK=site-a   # expect creates, NO replaces
make opnsense-provision-up STACK=site-a        # waits for the guest: ~10-20 min total
# image = nuageinit-patched, internally-compressed qcow2 from the self-hosted mirror
# (github.com/maksudsharif/homelab-images, release freebsd-14.3-cloudinit —
# provenance + patch rationale in its notes); no staging, no decompression, no console

The stack must already exist from Stage 0.5 (pulumi stack init site-a). The target accepts raw bcrypt $ sequences after the 2026-07-05 quoting fix. Verify the values with make opnsense-provision-config STACK=site-a.

3. Wait out the conversion. cloud-init writes the seed (hostname sa-fw-01, WAN=DHCP, LAN static 192.168.1.41, API key, root SSH) then runs opnsense-bootstrap: packages download over the staged WAN, then the VM reboots into OPNsense. Watch from the PVE console.
4. Sync the minted API creds into the Ansible fallback (added 2026-07-05: every fresh provision mints NEW API creds; the config plays read them bao-first with vault-credentials.yml fallback, and while bao is down the fallback is the only source): copy pulumi stack output sa-fw-01-apiKey and pulumi stack output sa-fw-01-apiSecret --show-secrets into vault_opnsense_api_key / vault_opnsense_api_secret (plus vault_opnsense_root_password = the chosen root password) via ansible-vault edit opnsense/config/inventory/group_vars/opnsense/vault-credentials.yml. Optional: pin opnsense.apiKey/opnsense.apiSecretStored for stable re-applies.
5. Inspect the historical staging app config; do not treat it as the target. The existing role still models outbound NAT for 10.10.253.0/24 plus a baseline pass rule. For the clean-slate plan, run check mode only until that role is replaced by the reviewed OPNsense VLAN/DHCP/policy implementation:

make opnsense-config-check
# Do not run opnsense-config-apply for Phase A4 until the final model is implemented.

6. Export the config to OpenBao KV v2 homelab/opnsense/site-a/exports under a dated key (destination decided 2026-07-03). Never commit config.xml to git; it holds the bcrypt hashes and the API secret.

Gate 4: OPNsense API answers on 192.168.1.41; opnsense-config-apply re-run = 0 changes; from the OPNsense shell: WAN holds a temp-LAN (192.168.0.0/23) lease, ping 1.1.1.1 and drill example.com succeed (OPNsense ships drill, not dig).

Stage 5: sa-uos-01: UniFi OS Server Controller

See UOS controller for the design. UOS is pinned to 5.1.19 because request shapes are version-specific. Its static temporary address is 192.168.1.40; its final VLAN 10 address is 10.10.10.40.

make unifi-provision-set STACK=site-a
make unifi-provision-preview STACK=site-a
make unifi-provision-up STACK=site-a

make unifi-config-check
make unifi-config-setup              # first-run owner/console via the UOS API

Gate 5: UOS UI on https://192.168.1.40:11443; unifi-config-check clean. Device adoption is deliberately not part of bootstrap.

Stage 6: sa-bao-01: OpenBao VM + Init Ceremony

See Secrets: OpenBao for the design. Follow openbao/provision/docs/runbooks/openbao-initial-setup.md with OpenBao pinned to 2.5.4. Open the password manager before starting because the initialization ceremony displays its secrets exactly once.

1. Provision (the VM boots sealed + uninitialized by design; allow 5-10 minutes):

make openbao-provision-set STACK=site-a
make openbao-provision-preview STACK=site-a
make openbao-provision-up STACK=site-a

2. Find the lease and reserve it: on sa-edge-01: qm list, then qm guest cmd <vmid> network-get-interfaces; add a DHCP reservation for the MAC. A rebuild gets a new MAC: re-reserve after any destroy/up.
3. Verify: ssh bao-admin@<ip> (prompt reads bao-admin@sa-bao-01); cloud-init status --long = done; bao status = Initialized: false, Sealed: true; bao version = 2.5.4.
4. Init ceremony: store all six secrets in the password manager before anything else:

bao operator init -key-shares=5 -key-threshold=3
bao operator unseal      # ×3, different keys
bao login                # root token

5. Hygiene: bao audit list shows the declarative file device; write the admin policy, enable userpass, mount KV v2, smoke-test (exact commands in the canonical runbook).
6. Reboot/unseal loop: after sudo reboot, bao status shows sealed: bao operator unseal ×3. This is the operating reality until Site B brings transit auto-unseal: after any reboot, sa-bao-01 comes back sealed.
7. VM start order on the E200 is Pulumi-managed: OPNsense order=1,up=90,down=180, bao order=2,up=60,down=180, and UOS order=3,up=0,down=180. Proxmox applies up after the current VM starts, so OPNsense's 90-second window delays bao. Verify with qm config; these delays are not health probes. Operational law: systemctl restart openbao seals the server; use reload for config changes.

Gate 6: bao status = initialized + unsealed; start order set on both VMs.

Stage 7: Secrets Go bao-first

(First executed 2026-07-01 for the original sa-bao-01; that instance was destroyed 2026-07-04, so this stage is unconditionally pending: run it in full after Stage 6. The old BAO_* values in .env.local are dead; every value below gets replaced.)

1. .env.local: set BAO_ADDR=http://<bao-lease>:8200 and, for this run only, BAO_ADMIN_TOKEN.
2. make openbao-config-setup ARGS='-e generate_secret_ids=true': KV v2 homelab mount, per-scope policies, AppRoles; prints role_ids + one-time secret_ids → paste into .env.local. Not retrievable later.
3. make openbao-config-seed: copies the vault-credentials values + Proxmox creds into KV. Idempotent; re-run after rotations.
4. Remove BAO_ADMIN_TOKEN from .env.local.

Gate 7 (two-sided): make opnsense-config-check succeeds with the vault-credentials file renamed away (bao path) and with BAO_ADDR unset (break-glass fallback). Restore both afterwards.

Final Gate: Phase-0 Complete for sa-edge-01

Next: reconcile A0 evidence, add the required VM trunk support, and begin A3 switch adoption. Continue with Build phases and Migration phases.

Rebuild / Rollback Notes

• The whole edge VM stack is rebuildable from the repo (*-provision-destroy → -up). Rebuilding sa-bao-01 creates a new bao, so reserve its new MAC, initialize it, and repeat Stage 7. Rebuilding sa-fw-01 requires the Stage 4 configuration to be applied and exported again.
• If a node-side manual qm destroy ever bypasses Pulumi, run pulumi refresh before the next up.

Related Pages

• Initial Site Bootstrap: the site-wide sequence this expands
• Current Build State: where the build stands today
• Site A Port Map: the locked cabling this runbook wires to