Makes the incus hosting-provider deployment machine-agnostic and reproducible end to end, and converges it live on the bare-metal incus host 192.168.51.101: a hosting-provider admin can follow the documented bootstrap and have all 16 hosting-provider containers come up from the base LXC image onto real per-node configs, with the secret mesh (SPIRE, OpenBao, PKI), internal DNS, mail, and the admin UIs all working.
Fresh-provider bootstrap
Machine-agnostic base-image build/upload (tf/incus/image) and a staged BOOTSTRAP=1 local-then-migrate tofu backend, with durable local state, a pinned TF_HTTP_ADDRESS override, and an auto-opened remote-deployer migrate tunnel (BOOTSTRAP_TUNNEL_SSH).
Deploy transport: bridge-local substituter plus delta closure push via incus file push (not exec-stdin streaming), with realistic import timeouts.
Bootstrap peer DNS: peers resolve each other via the incus bridge dnsmasq (instance FQDNs from DHCP leases plus <served domain> -> <instance FQDN> CNAMEs derived from the group's ssl requests), before bind/NetBox exist; the host resolver joins the same chain.
Container-shaped node neutralization: bootloader/disk/VM build products dropped on LXC-deployed nodes, eth0 via networkd DHCP, systemd-machine-id-commit suppressed so the first switch is not killed mid-flight.
Bare-metal host integration
The hypervisor host itself attests to SPIRE via its own persistent swtpm 2.0 (host-swtpm / host-register-ek), so hosts with a TPM 1.2 or no TPM still get an SVID and the gap #38 token producer runs.
Host networking reconciled to the live LACP bond (802.3ad).
The incus web UI is served through a PKI-certified host nginx front on :443 (the daemon's :8443 self-signed cert stays stable for deploy-runner TOFU pinning), with OIDC against authelia (public client, JWT-profile access tokens, portless callback).
Per-instance incus metrics are scraped from the daemon's /1.0/metrics through the same nginx front (dedicated metrics-type client certificate minted and enrolled by the host) into the otel-collector -> OpenObserve pipeline.
The windmill token reaches the api node via an OpenBao broker instead of panel colocation; the incus host mints one-time per-runner trust tokens into its incus-deploy vars KV, and each deploy runner self-enrolls its incus client certificate from runner-local state.
Secret-mesh convergence
Live bring-up surfaced and fixed a long tail of structural gaps, including: cert-auth roles keyed on network.name (not app attr keys) and registered for every consumer and generation node including the openbao node itself; SPIRE trust-bundle export on a timer plus self-healing of agents, cached PKI CAs, and cert-auth trusted CAs across CA and EK rotations; OpenBao agents/logins coupling VAULT_CACERT to the https:// scheme; dynamic postgres DSN leases owned by a persistent role, with schema grants and consumer restarts on lease re-render; vars uploads ordered after token login with KV-backed secrets no longer rotating every reboot; the terraform-backend state store decoupled from OpenBao (its kmsKey routed via the on-machine provider) so it never sits downstream of the secret system it repairs.
Services and admin surfaces
Outbound mail through an upstream smarthost with per-sender submission accounts, served on smtp.<domain>; failed DSNs surface as OpenObserve alerts. (The upstream account seed is a temporary reuse of git@fediversity.eu, flagged in-tree as structural follow-up.)
Admin UIs reachable over the documented SSH SOCKS access path: lldap behind authelia forward-auth, OpenBao UI with authelia-group-mapped policies and read-only introspection for admins, NetBox OIDC (internal CA trust, client_secret_post, gunicorn otel instrumentation), pgweb across application databases, redis-commander/garage-webui, and an eager per-group valkey namespace.
fediversity-admins group provisioned in lldap and enrolled for the seeded administrator; docs cover private access, cert regimes, first-login seeding, and bootstrap recovery.
Verification
Live end-to-end on 192.168.51.101: full BOOTSTRAP=1 ladder converged (base image -> per-node configs -> local->HTTP state migrate), secret mesh healthy, panel demo deploy green, admin UIs and metrics verified.
CI: treefmt, deployment-tf-incus-hosts, and the service checks are green on earlier batches; apps-tf-incus and a heavy sweep on the rebased tip are delegated to CI.
Disclaimer: I used a coding agent in the creation of this patch.
Closes #954.
Makes the incus hosting-provider deployment machine-agnostic and reproducible end to end, and converges it live on the bare-metal incus host `192.168.51.101`: a hosting-provider admin can follow the documented bootstrap and have all 16 hosting-provider containers come up from the base LXC image onto real per-node configs, with the secret mesh (SPIRE, OpenBao, PKI), internal DNS, mail, and the admin UIs all working.
## Fresh-provider bootstrap
- Machine-agnostic base-image build/upload (`tf/incus/image`) and a staged `BOOTSTRAP=1` local-then-migrate tofu backend, with durable local state, a pinned `TF_HTTP_ADDRESS` override, and an auto-opened remote-deployer migrate tunnel (`BOOTSTRAP_TUNNEL_SSH`).
- Deploy transport: bridge-local substituter plus delta closure push via `incus file push` (not exec-stdin streaming), with realistic import timeouts.
- Bootstrap peer DNS: peers resolve each other via the incus bridge dnsmasq (instance FQDNs from DHCP leases plus `<served domain> -> <instance FQDN>` CNAMEs derived from the group's ssl requests), before bind/NetBox exist; the host resolver joins the same chain.
- Container-shaped node neutralization: bootloader/disk/VM build products dropped on LXC-deployed nodes, `eth0` via networkd DHCP, `systemd-machine-id-commit` suppressed so the first switch is not killed mid-flight.
## Bare-metal host integration
- The hypervisor host itself attests to SPIRE via its own persistent swtpm 2.0 (`host-swtpm` / `host-register-ek`), so hosts with a TPM 1.2 or no TPM still get an SVID and the gap #38 token producer runs.
- Host networking reconciled to the live LACP bond (802.3ad).
- The incus web UI is served through a PKI-certified host nginx front on `:443` (the daemon's `:8443` self-signed cert stays stable for deploy-runner TOFU pinning), with OIDC against authelia (public client, JWT-profile access tokens, portless callback).
- Per-instance incus metrics are scraped from the daemon's `/1.0/metrics` through the same nginx front (dedicated metrics-type client certificate minted and enrolled by the host) into the otel-collector -> OpenObserve pipeline.
## Cross-node deploy credentials (gap #38)
- The windmill token reaches the api node via an OpenBao broker instead of panel colocation; the incus host mints one-time per-runner trust tokens into its `incus-deploy` vars KV, and each deploy runner self-enrolls its incus client certificate from runner-local state.
## Secret-mesh convergence
Live bring-up surfaced and fixed a long tail of structural gaps, including: cert-auth roles keyed on `network.name` (not app attr keys) and registered for every consumer and generation node including the openbao node itself; SPIRE trust-bundle export on a timer plus self-healing of agents, cached PKI CAs, and cert-auth trusted CAs across CA and EK rotations; OpenBao agents/logins coupling `VAULT_CACERT` to the `https://` scheme; dynamic postgres DSN leases owned by a persistent role, with schema grants and consumer restarts on lease re-render; vars uploads ordered after token login with KV-backed secrets no longer rotating every reboot; the terraform-backend state store decoupled from OpenBao (its `kmsKey` routed via the on-machine provider) so it never sits downstream of the secret system it repairs.
## Services and admin surfaces
- Outbound mail through an upstream smarthost with per-sender submission accounts, served on `smtp.<domain>`; failed DSNs surface as OpenObserve alerts. (The upstream account seed is a temporary reuse of `git@fediversity.eu`, flagged in-tree as structural follow-up.)
- Admin UIs reachable over the documented SSH SOCKS access path: lldap behind authelia forward-auth, OpenBao UI with authelia-group-mapped policies and read-only introspection for admins, NetBox OIDC (internal CA trust, `client_secret_post`, gunicorn otel instrumentation), pgweb across application databases, redis-commander/garage-webui, and an eager per-group valkey namespace.
- `fediversity-admins` group provisioned in lldap and enrolled for the seeded administrator; docs cover private access, cert regimes, first-login seeding, and bootstrap recovery.
## Verification
- Live end-to-end on `192.168.51.101`: full `BOOTSTRAP=1` ladder converged (base image -> per-node configs -> local->HTTP state migrate), secret mesh healthy, panel demo deploy green, admin UIs and metrics verified.
- CI: `treefmt`, `deployment-tf-incus-hosts`, and the service checks are green on earlier batches; `apps-tf-incus` and a heavy sweep on the rebased tip are delegated to CI.
Disclaimer: I used a coding agent in the creation of this patch.
Replace the placeholder incus machine in the single-node `hypervisor` group
with the real bare-metal host at `192.168.51.101`, and add the per-host
hardware file it needs so `nix run .#hypervisor-ssh-incus` (bootstrap.md
step 0) deploys to it over SSH.
The host is already provisioned (single hardware-RAID disk, existing GPT
filesystems), so disk handling is non-destructive: disko is disabled and the
existing `/` and `/boot` filesystems are pinned by UUID rather than
repartitioned.
Networking runs over a single live 25G NIC (`net0`, MAC-pinned): the host has
two 25G ports but only one is on a forwarding switch port, so an active-backup
bond black-holes traffic. The single-interface config drives the live NIC
directly; re-introduce a bond only once the second switch port is patched.
Assisted-by: Claude:claude-opus-4-8
Close the two step-1 gaps that made the documented bootstrap flow
localhost-only and non-reproducible across machines:
- Give `dev-tf-incus-image` the same `INCUS_HOST` env seam the hosts
graph has, and route the image through `bootstrap-apply.sh` (local
persisted state keyed by `STATE_KEY`, optional migrate) instead of a
throwaway `:8888` HTTP backend. Drop the alias-pre-delete hack now that
tofu tracks/replaces the image via persisted state.
- Auto-derive the migrate target in the wrapper: in the Cycle-1
`BOOTSTRAP=1` branch, when the operator has not pinned
`TF_HTTP_ADDRESS`, read the just-applied `terraform-backend` node's
bridge IP from `tofu output ipv4` and point the migrate at
`http://<node-bridge-ip>:8080/state/...`. Removes the bogus
`localhost:8080` default that meant the migrate never happened on a
remote deployer. Steady-state/test paths never enter this branch.
- Serve the `terraform-backend` migrate lane as plaintext `:8080` on the
incus bridge interface only (`eth0`, firewall-scoped), an internal
one-shot bootstrap lane; nginx+TLS on `:443` stays the public path.
Test and live now share one migrate path.
- Doc: `notes/bootstrap.md` records that `-tf-incus-all` brings up the
whole node roster (not just terraform-backend + netbox), documents the
auto-derive, and notes the `INCUS_HOST` remote flow closes the
`api/README.md` "PR #1013" TODO.
Assisted-by: Claude:claude-opus-4-8
Streaming the closure NAR through `incus exec` stdin deadlocks under
concurrent deploys: the exec control websocket multiplexes stdin/stdout/
stderr with no real stdin flow-control, so N channels mutually starve the
daemon's shared per-conn buffer budget (cf. `kubernetes/kubernetes` #89899,
`aiohttp` #1845). Observed 0/16 nodes converged at `-parallelism=10`, all
senders wedged in `anon_pipe_write` with flat 0 bytes imported.
Replace the stream with `incus file push` (HTTP file-transfer API) of a
single temp NAR, then `nix-store --import` from that local file in the
container -- no cross-network backpressure loop, so the deadlock precondition
is gone and the default parallelism (10) is safe again.
`FED_PARALLELISM` stays as a caller-supplied per-host safety valve (wins over
the Nix-baked `BOOTSTRAP_PARALLELISM`) for genuinely resource-starved hosts.
Document that step 1 and the base-image upload need `nix run --impure`: the
`INCUS_HOST`/`BOOTSTRAP` seams are read with `builtins.getEnv`, which returns
`""` under `pure-eval`, silently falling back to `localhost` / disabling the
local-then-migrate path.
Assisted-by: Claude:claude-opus-4-8
An incus deploy pushes each node's real config into an LXC container, which
has no bootloader and no disko-managed disks. The reused roster config targets
a physical/VM host: the shared `setups/nixos` base enables `systemd-boot` plus
a `/boot` vfat mount (`boot.nix`) and disko partitioning (`disks.nix`). Inside
the container `switch-to-configuration` therefore failed at `Failed to install
bootloader` (`efiSysMountPoint '/boot' is not a mounted partition`) and the
node never activated its real config -- it stayed on the base image.
Fold `lxc-container.nix` into the pushed config and neutralize the host
boot/disk options (`systemd-boot.enable = mkForce false`, `fileSystems =
mkForce {}`) whenever `isIncusHosts`, exactly as the `deployment-tf-incus-hosts`
test's `shared.nix` already does. The gate keeps the ssh / plain-tf deploy of
the same node (to a real host) unchanged.
The image build's `nodeConfiguration` reuses this same per-node config, so its
previously-separate `lxc-container.nix` + `mkForce` block is now redundant and
is removed to avoid a double-definition collision.
Assisted-by: Claude:claude-opus-4-8
The BOOTSTRAP local-apply cycle wrote tofu state to a file inside the run dir,
which lives under the `env/` tree that `materialize_workdir` wipes with
`rm -rf` at the start of every cycle. State was only copied up to the durable
`$persist_state` (workdir root) *after* the whole apply finished. An apply
interrupted mid-flight (SIGINT / timeout / crash during `tofu apply`) therefore
lost all its state: the next run's `env/` wipe destroyed it, the run re-seeded
empty, and it planned a full CREATE against instances that already existed on
the host -- failing with `instance already exists` and orphaning them.
Point the `local` backend directly at the absolute `$persist_state` (via the
`zzz_bootstrap_backend_override.tf` override, and matching `write_local_override`
for the migrate path). tofu persists local state after every applied resource,
so state is now durable the instant each resource finishes and an interrupted
apply stays resumable. The now-redundant post-apply `cp` steps are removed, and
the migrate cleanup drops the extra `rm -f "$localstate"` since `$localstate`
IS `$persist_state` (already renamed to the `.migrated` marker).
Assisted-by: Claude:claude-opus-4-8
On a fresh provider the spire-server is not reachable yet (the hypervisor
node's `serverAddress` is a placeholder FQDN with no DNS, or the server
container is not up). The two first-boot trust-bundle oneshots were
`requiredBy` a service `wantedBy multi-user.target` and ordered `before` it,
so a failed fetch wedged `multi-user.target` in `activating` for the whole
`bundleFetchTimeout` window and the node never finished booting.
Make both fail-soft: `wantedBy` (not `requiredBy`) the consuming service, and
bound `TimeoutStartSec` to the script's own retry deadline (else systemd's
default 90s kills the oneshot before its deadline). The consumers already
no-op until the bundle appears, so the node boots and converges once a later
deploy re-runs the fetch against a reachable server.
Also route the `terraform-backend` node's `generateFiles` want (its
state-encryption `kmsKey`) through the `vars`/openbao provider. Like `valkey`
and `panel`, `terraform-backend` declares `contracts.generateFiles.want`
directly in its module rather than via the resource DSL or the SSO-consumer
path, so it was never handed the `consumerModule` (routing + agent) and its
`kmsKey.path` threw "unrouted" when the node's `toplevel` closure was forced.
Assisted-by: Claude:claude-opus-4-8
The `isIncusHosts` neutralization (`ce594e2d2`) folds `lxc-container.nix`
into each incus node so `switch-to-configuration` works in an LXC. That
makes the node config container-shaped, where `boot.initrd.enable`
defaults off and `config.system.build.initialRamdisk` is absent.
`checks/common/deployerNode.nix` preloads every `system.extraSystems`
machine's closure and collects `machine.system.build.vm.inputDerivation`.
Its `or { }` guard only skips a missing attribute, but `vm` is always
declared (`build-vm.nix` defaults it to
`config.virtualisation.vmVariant.system.build.vm`); evaluating it pulls
in `qemu-vm.nix`, whose `virtualisation.directBoot.initrd` default
references the absent `initialRamdisk`, tripping `attribute
'initialRamdisk' missing`. This broke `apps-tf-incus` eval (whose
deployer uses `extraSystems`); `deployment-tf-incus-hosts` was unaffected
because it preloads via `extraDependencies` drvPaths without forcing full
config eval.
Force `system.build.vm` to a plain attrset on the neutralized nodes so
the guard skips it without ever evaluating `vmVariant`. A container is
never booted as a QEMU VM, so the VM product is meaningless there.
Assisted-by: Claude:claude-opus-4-8
The deploy-time config re-eval (`core/effects/common/write-config.nix`)
imports the caller setup with the baked, JSON-serialized `args`, from
which `groups` was stripped (it holds functions like each group's `fqdn`
and cannot be serialized). Without re-injecting it, the caller falls back
to `core`'s frontend-free `./setups/groups`, whose `hosting-provider`
group does not declare `applications.api`/`applications.panel` -- yet the
baked `default-configuration` DATA references them (the build folded the
front-ends in), faulting `The option 'default-configuration.applications.api'
does not exist` when a hosting-provider node is deployed with the front-ends.
`boundary.deploymentPlan` now threads the front-end-composed `groups`
(reached by name off `composedSurface`, the same fullest-leaf the root's
deploy family uses) into the re-eval, so it declares the same options the
build baked. `composedGroups` is `null` on a core-only checkout (no
front-ends), leaving the frontend-free default unchanged there; an explicit
`groups` already in `args` still wins.
`panel/default.nix` exposes `composedGroups` on its `lib` (mirroring
`api.lib.composedGroups`) so `composedSurface`->`.lib.composedGroups`
resolves for the fullest leaf.
Assisted-by: Claude:claude-opus-4-8
The incus deploy transport (`core/effects/incus/run.sh`) pushed each node's
FULL closure over the deployer->host->container path. The deployer->host WAN
uplink caps at ~4 MB/s and does not scale with `-parallelism`, so 16 nodes x
~1 GiB serialized took hours (a live 16-node deploy stalled 7.5h then dropped).
Every incus node is created from one shared base LXC image built on the
hosting-provider host, so that host's store already holds ~86% of any node's
closure, and host->container over `incusbr0` runs ~52 MB/s. Exploit both:
- Serve the host's own store to its containers as a bridge-local Nix binary
cache (`services.nix-serve` bound to the bridge gateway `10.0.100.1:5000` in
`applications/hosting-provider/incus.nix`). `net.ipv4.ip_nonlocal_bind` lets
it bind before incus brings up the bridge. Unsigned + bridge-internal.
- Point each incus node at that cache
(`nix.settings.substituters = http://10.0.100.1:5000`, `require-sigs = false`,
in the `isIncusHosts` block of `setups/hosts-common.nix`). The base image
reuses this node config, so the setting is present from the first switch.
- In `run.sh`, first let the container substitute the closure from the host
over the bridge (`nix-store -r`), then delta-export only the paths the host
ALSO lacks (freshly built on the deployer) as the NAR pushed over the slow
leg. On an unchanged re-deploy the delta is empty and the push is skipped.
Only the config-specific delta now crosses the WAN uplink; the shared base is
fetched at LAN speed.
Assisted-by: Claude:claude-opus-4-8
On the first switch the container still runs the base LXC image, whose
`nix.conf` predates the `nix.settings.substituters` entry the real node
config carries -- so relying on the container config to point `nix-store
-r` at `http://10.0.100.1:5000` only works from the second switch on. Pass
the bridge cache explicitly (`--extra-substituters` +
`--extra-trusted-substituters`, `--no-require-sigs` for the unsigned
bridge-internal store) so bridge substitution fires on the very first
switch too, keeping the full closure off the slow deployer->host WAN leg.
Verified live: with `cache.nixos.org` excluded, a container fetched
`openssl-3.6.3` and its deps `from 'http://10.0.100.1:5000'`.
Assisted-by: Claude:claude-opus-4-8
Real hosting-provider nodes deployed into incus containers inherit
`networking.useDHCP = true` from `setups/nixos/networking.nix`, which
starts `dhcpcd`. Inside a container namespace eth0 has no `Kind`
attribute, so `lxc-container.nix`'s `99-lxc-veth-default-dhcp` rule
(matching `Kind = "veth"`, only visible from the host side of the veth
pair) never matches and `dhcpcd` logs `no valid interfaces found`. The
node then gets no IPv4 on `incusbr0` and cannot reach any peer -- DNS,
spire, openbao, or the bootstrap migrate target -- so a fresh-provider
deploy converges but never completes its local->HTTP state migrate.
Switch these nodes to `systemd-networkd` with a name-based DHCP match on
eth0, exactly as the `deployment-tf-incus-hosts` and
`integrations-spire-incus` test `shared.nix` already do. Scoped to the
incus deploy methods (the `isIncusHosts` block), so an ssh / plain-tf
deploy of the same node to a real host keeps the shared `useDHCP`
default.
Assisted-by: Claude:claude-opus-4-8
The networkd fix in `a1d80c61c` requested DHCP on `eth0` via a separate
`systemd.network.networks."10-eth0-dhcp"` unit matching `eth0`. But the
`network` resource writes each node's NetBox-allocated routed address into
`networking.interfaces.eth0.ipv4.addresses`, which
`network-interfaces-systemd.nix` renders as a `40-eth0` `.network` unit also
matching `eth0`. systemd-networkd applies only the first matching `.network`
file by lexical order, so the `10-` unit shadowed the generated `40-eth0` one:
the node got the bridge DHCP lease but dropped the NetBox address (caught by the
`apps-tf-incus` "pixelfed eth0 carries the NetBox v4/v6 addresses" subtest).
Request DHCP per-interface via `networking.interfaces.eth0.useDHCP = true`
instead, which folds `DHCP=yes` into the same `40-eth0` unit that carries the
static address, so eth0 gets both from one unit. The global `useDHCP` fallback
would not work either: the same renderer forces DHCP off on any interface that
already has static addresses.
Assisted-by: Claude:claude-opus-4-8
On a fresh provider the incus-deployed nodes get `10.0.100.x` bridge leases but
nothing serves their peer FQDNs (`spire-placeholder.abundos.eu`, `tf-backend...`):
the group's `machines` addresses are `0.0.0.0` placeholders, the containers'
real leases aren't known upfront, and both the `bind` node and NetBox are
themselves being brought up by this same bootstrap. So the trust-bundle fetch
NXDOMAINs, the openbao/tf-backend chain never starts, `:8080` never binds, and
the local->HTTP state migrate never completes.
The one peer-address source that exists at bootstrap time is the incus bridge
dnsmasq (`incusbr0`, `10.0.100.1`): it hands out the leases, so it already knows
every container's address the moment the lease is granted. Serve the peer FQDNs
from those leases:
- Set the incus instance name to the node's `network.name` (the FQDN label the
graph resolves peers by), via a distinct `networkName`/`network_name` field so
the `for_each`/state keys stay on the instance name -- no tofu state rename.
- Give `incusbr0` `dns.domain = abundos.eu` so its dnsmasq answers
`<network.name>.abundos.eu` from the lease. Safe during bootstrap because the
real `abundos.eu` authority (the group's `bind` node) is not yet up; in steady
state bind serves the disjoint real (non-placeholder) records.
- Point the incus nodes' resolver at the bridge (`10.0.100.1`) FIRST (ordered
ahead of the `bind` node), so peer lookups hit the bridge dnsmasq during
bootstrap and fall back to bind once it is up.
All gated on the incus methods (`isIncusHosts`) / the incus-hosts effect, so
ssh / plain-tf / operator deploys are untouched.
Assisted-by: Claude:claude-opus-4-8
`setups/nixos/networking.nix` disables the NixOS firewall module
(`firewall.enable = false`) and loads a static Procolix
`nftables-ruleset.nft` whose `inet filter input` chain is `policy drop`
and opens only 22/80/443/161/5666. That makes every app module's
`networking.firewall.allowedTCPPorts` / `openFirewall` dead code, so all
cross-node hosting-provider calls (spire `:8081`/`:8082`, openbao
`:8200`, tf-backend `:8080`, netbox, postgres, valkey) are dropped -- the
openbao/tf-backend chain never comes up and the local->HTTP bootstrap
state migrate never completes.
On the incus methods (`isIncusHosts`), take over
`networking.nftables.ruleset`: replay the shared static file, then merge
one `iifname "eth0" ip saddr 10.0.100.0/24 accept` rule into the same
`inet filter input` chain. Bridge peers are trusted co-tenants of the one
deployment, so a subnet-accept is machine-agnostic -- no per-app port
list in the shared file. The WAN-facing `policy drop` is untouched, and
ssh / plain-tf / real-host deploys keep the unmodified static ruleset.
Assisted-by: Claude:claude-opus-4-8
incus mounts `/etc/machine-id` as a read-only tmpfs. Upstream guards
`systemd-machine-id-commit.service` with
`ConditionPathIsMountPoint = /etc/machine-id`, meant to fire the commit
only when a writable bind mount is present. The incus `ro` mount
satisfies that mountpoint condition but cannot be written, so the commit
blocks instead of being skipped; systemd escalates `SIGTERM` -> `SIGKILL`
over ~6 min and, because the unit was pulled into the switch
transaction, kills the `switch-to-configuration switch` transient with
it. The activation aborts and `/run/current-system` stays pinned to the
base LXC image, so the node's real config (spire-server, the
openbao/tf-backend chain) never lands and the local->HTTP state migrate
never completes.
The container's machine-id is incus-managed and needs no commit, so
suppress the unit. It is an upstream systemd unit (`upstreamSystemUnits`),
NOT a NixOS-generated `systemd.services.<name>`, so it is dropped via
`suppressedSystemUnits`. Gated on the incus methods, so ssh / plain-tf /
real-host deploys keep the upstream commit behaviour.
Assisted-by: Claude:claude-opus-4-8
On incus-deployed nodes SPIRE TPM node-attestation failed with `stat
/dev/tpm1: no such file or directory`, wedging the whole fresh-provider
bootstrap: the agent never got an SVID, so `vars-openbao-login` hung, no vars
were uploaded to OpenBao, the openbao/tf-backend PKI chain never came up,
`terraform-backend` never bound `:8080`, and the local->HTTP state migrate
never completed.
Two compounding causes, both fixed here:
- `/sys` is not namespaced in an incus container, so `/sys/class/tpm/` leaks
the HOST enumeration (the host's hardware TPM plus every container's swtpm).
The `spire-tpm-plugin` (go-attestation `attest.OpenTPM(&OpenConfig{})`, which
exposes no device-path config) picks the first non-1.2 entry and opens
`/dev/<name>` -- landing on `/dev/tpm1`, which has no devnode in the
container (only the injected `/dev/tpm0` exists). The picked index is also
non-deterministic across container-start order. Overlay `/sys/class/tpm` with
a tmpfs holding a single synthetic `tpm0` entry (no `caps` => TPM 2.0), so the
scan resolves to the container's real `/dev/tpm0`. Gated on the incus methods
(`isIncusHosts`); ssh / real-host deploys have a proper per-host `/sys`.
- A SPIRE-node redeploy resets its ephemeral EK allow-list drop-dir
(`/var/lib/spire-server/host-ek-hashes`), but `register_ek`'s triggers keyed
only on `instance_name` did not change, so registration never re-fired and
the allow-list stayed empty -- no node could attest. Thread the SPIRE node's
(and each node's own) `nixos_id` rebuild id into the `register_ek` triggers so
a redeploy re-pushes every node's EK. Adds a `nixos_id` output to the incus
host module.
Assisted-by: Claude:claude-opus-4-8
With the `/sys/class/tpm` mask in place the SPIRE tpm attestor now targets the
container's real `/dev/tpm0`, but the agent then failed with `open /dev/tpm0:
permission denied`. The `spire-agent.nix` tmpfiles `z` rule that sets the devnode
`tss`-group-readable does not take effect on incus: the daemon injects
`/dev/tpm0` AFTER `systemd-tmpfiles-setup-dev` has run, so the rule finds no
path and the device keeps its kernel default `0600 root:root`. Re-apply the
group/mode in the `mask-leaked-host-tpms` oneshot, which is ordered before
`spire-agent.service` and runs once the device exists. Verified live: with
`root:tss 0660` the agent reads the EK and attestation proceeds to the server.
Assisted-by: Claude:claude-opus-4-8
The #633 deploy-time EK auto-registration mechanism (`register-ek.sh` +
`null_resource.register_ek`, gated on the effect's `spireNode` option) was only
ever exercised by the `integrations-spire-incus` test, which passes its own
`spireNode`. The production `hosting-provider-tf-incus-all` deploy never set it,
so `spireNode` stayed null, `register_ek` had zero instances, and the SPIRE
server's EK allow-list (`/var/lib/spire-server/host-ek-hashes`) was never
populated. Every incus-deployed node then failed TPM attestation with
`nodeattestor(tpm): could not validate EK`, so no node got an SVID -- wedging
the openbao/tf-backend bootstrap and the local->HTTP state migrate.
Default `spireNode` on an incus-hosts deploy to the group's own spire node over
the `incus-exec` transport (the deployer reaches every container through the
same incus daemon, so no SSH into the spire node is needed). The spire node's
incus instance name is its `network.name`, matching `main.tf`'s
`coalesce(network_name, key)`. Only defaulted when the caller did not set
`spireNode` and the group has a spire node, preserving the prior
no-registration behavior elsewhere.
Assisted-by: Claude:claude-opus-4-8
Drop the stale `net0` MAC-pinned rename left behind by 42cc38131d: it
renamed `ens2f1np1`, the very port the bond enslaves, which clashes with
enslaving it. The switch is now configured for an `802.3ad` (LACP)
link-aggregation group across both `ens2f0np0`/`ens2f1np1`, so both ports
forward and the WAN IP/gateway ride the `bond0` master (assigned by the
group's `network` resource via `interface = "bond0"`). This mirrors the
host's proven-working local `configuration.nix` in our data-model form,
deployable via `nix run .#hypervisor-ssh-incus`.
Assisted-by: Claude:claude-opus-4-8
The transient `switch-to-configuration` unit was always scheduled with
`RuntimeMaxSec=180`. That cap guards a container-specific failure (the nspawn
post-reexec D-Bus hang, nixpkgs #478109/#533429), but on a bare-metal host it is
a footgun: an activation that legitimately reconfigures the host's own uplink
(e.g. tearing down and re-enslaving a `bond0`) can exceed 180s while the network
is briefly down, and the cap would SIGKILL the switch mid-activation -- after the
interface teardown but before the re-enslave -- stranding the host offline.
Detect the target's virtualization on the target itself
(`systemd-detect-virt --quiet --container` exits 0 inside a container) and apply
the cap only there; bare-metal switches run uncapped. Live-validated against the
incus hypervisor host (192.168.51.101): the switch ran with
`RuntimeMaxUSec=infinity` and completed without stranding.
Assisted-by: Claude:claude-opus-4-8
Deploy-time EK registration (`register-ek.sh`) restarts `spire-register` once
per node, so a full hosting-provider bring-up fires ~16 restarts in a tight
window. That trips systemd's default `StartLimitBurst=5` per 10s, refusing the
later restarts with "start ... attempted too often" and failing the
`register_ek` provisioner. Each run already reconciles the whole drop-dir
(idempotent, order-independent), so extra restarts are redundant, not harmful.
Set `StartLimitIntervalSec=0` to disable the rate-limit.
Assisted-by: Claude:claude-opus-4-8
`register_ek` ran `for_each = var.nodes`, so the SPIRE node registered its own
EK on itself. But the SPIRE node runs the spire-SERVER (the attestor), not a
spire-agent workload: its `cfg.spiffe` is empty, so it gets no
`spire-tpm-plugin` and thus no `get_tpm_pubhash` on `PATH`. The registration
therefore failed with `exit status 127` (`Command not found`), aborting the
whole `BOOTSTRAP=1` apply before the local->HTTP state migrate could run.
Self-registration is a no-op by design anyway: the SPIRE node has no agent
presenting an EK to allow-list. Drop `local.spire_node_key` from the
`register_ek` `for_each` so the node is skipped.
Assisted-by: Claude:claude-opus-4-8
The fresh-provider state migrate (`BOOTSTRAP=1`) auto-derives its target from
the `terraform-backend` node's address. It read `tofu output ipv4`
(`incus_instance.ipv4_address`), which since NetBox IPAM (#833) surfaces the
node's ROUTED public address -- unreachable from the host/incus bridge. The
migrate is a host-internal plaintext hop over the bridge, so it must target the
node's incus-bridge DHCP lease (`10.0.100.x`) instead. `eth0` carries both
addresses; the provider exposes only one scalar `ipv4_address`, and it is the
wrong one for this purpose.
Retain BOTH addresses as first-class TF outputs rather than swapping one for the
other: steady-state public reads still want the routed `ipv4`, the migrate wants
the bridge lease. Add a `data.external.bridge_ip` program (`bridge-ip.sh`) that
reads each node's lease from `incus network list-leases` -- the authoritative
bridge-address source, keyed on the incus instance name -- exposed as a new
`bridge_ipv4` output. The migrate derivation prefers `bridge_ipv4` and falls
back to `ipv4` when it is absent (ssh/proxmox effects) or empty (no lease yet),
so non-incus and pre-lease paths are unaffected.
Assisted-by: Claude:claude-opus-4-8
The fresh-provider migrate (#336/#882) auto-derives the reachable
incus-bridge lease as its target whenever the baked `httpBackend.address`
resolves to `localhost`/`127.0.0.1`. That collapses a real localhost SSH
tunnel (a remote deployer forwarding the backend node's bridge `:8080`
through the incus host) with the built-in localhost default: the tunnel
would be overridden by the derived bridge IP, which is unreachable from the
workstation.
Thread an explicit-pin signal so derivation keys on whether the operator
actually set `TF_HTTP_ADDRESS`, not on the host string:
- add a `pinned` option to the `httpBackend` submodule (internal signal,
excluded from the `TF_HTTP_*` `value` map);
- set `httpBackend.pinned = tfHttpAddress != ""` in `env-config.nix`;
- bake `BOOTSTRAP_HTTP_PINNED` into the `bootstrapApply` `runtimeEnv`;
- gate the bridge-IP auto-derivation in `bootstrap-apply.sh` on
`BOOTSTRAP_HTTP_PINNED != 1` (still requiring a localhost host), so a
genuinely-unset default derives but a pinned `localhost:8080` tunnel is
honored verbatim.
Document the remote-deployer SSH-tunnel migrate procedure in
`notes/bootstrap.md` (parameterized on `$TF_VAR_host` + the backend node's
bridge lease), and update the migrate section to reflect the bridge-lease
target.
Assisted-by: Claude:claude-opus-4-8
The effect appends the `-<group>` suffix (`-hosting-provider`) to
`httpBackend.address` itself (`hosts-common.nix`). A pinned
`TF_HTTP_ADDRESS` that already includes the suffix doubles the state
key (`...example-hosting-provider-hosting-provider`), which misses the
existing local state and fails the migrate against an empty key.
Document pinning the base address only.
Assisted-by: Claude:claude-opus-4-8
During the fresh-provider incus bootstrap, cross-node consumers (lldap
uploading its bind password, netbox/windmill/authelia reading secrets)
must reach the group's openbao node to authenticate and store vars. Two
gaps blocked that:
- `vars-backend.generateFiles.address` defaulted to `http://127.0.0.1:8200`,
pointing every consumer at its OWN loopback, so `vars-openbao-login` got
connection-refused on a cross-node node. Point it at the openbao node's
SERVED domain (`openbao.<group domain>`) over its portless nginx `:443`
face, and extend `openbaoCentralTrustDomains` so the consumer trusts that
vhost's `openbao-pki` CA. Probing the served domain (not the instance
FQDN) is what routes to `openbao-pki`, so the CA cert lands in the trust
store.
- The incus bridge dnsmasq answers each node's INSTANCE FQDN
(`<network.name>.<domain>`) from its DHCP lease, but a consumer dials the
SERVED domain (`openbao.abundos.eu`, the peer's PKI cert CN), which the
bridge did not resolve. Add `<served> -> <instance FQDN>` CNAMEs to the
`incusbr0` preseed via a new `fediversity.incus.bridgeCnames` option, so
the served name resolves to the lease and TLS verifies. The setup derives
the CNAMEs from the hosting-provider group's static machine data + the
`<app>.<domain>` service-name convention (openbao/windmill/netbox; a
same-name node like monitoring or an off-apex served domain like
tf-backend's is skipped), so it needs no per-provider literal.
Eval-verified on the hosting-provider path (`varsOpenbaoAddress =
https://openbao.abundos.eu`, `hasPkiTrustUnit = true`) and the hypervisor
incus node (`raw.dnsmasq` renders the three CNAMEs). Proven live on the
incus host: the served domains resolve to their leases, and a consumer
dialing `https://openbao.abundos.eu` with the PKI CA gets HTTP 200.
Assisted-by: Claude:claude-opus-4-8
Document the incus-bridge peer name resolution in the fresh-provider
ladder: the bridge dnsmasq serves each node's instance FQDN from its DHCP
lease and adds `<served> -> <instance FQDN>` CNAMEs
(`fediversity.incus.bridgeCnames`) so a consumer dialing an internal PKI
node by its served/cert domain reaches its lease and TLS verifies, before
the group's own `bind`/NetBox records exist.
Assisted-by: Claude:claude-opus-4-8
Two flake-show / spire-incus regressions surfaced once the incus app module
started declaring a `fediversity.incus` option:
- The deploy-time SPIRE EK-registration `spireNode` was added to the shared
args under the broad `isIncusHosts` gate, which also matches the
`tf-incus-operator-hosts` method. That method's effect declares no
`spireNode` option, so `flake show` rejected
`deployments.tf-incus-operator-hosts.spireNode` as unknown for any group
with a `spire` node (the hosting-provider group). Gate on
`hostsType == "tf-incus-hosts"` -- the only method whose effect declares it.
- `openbao.nix` re-points its own `fediversity.openbao-pki.{address,tokenFile}`
only when that option is declared, but guarded on the whole `fediversity`
namespace (`options ? fediversity`). A node that imports the incus app module
(the `integrations-spire-incus` test's `incushost`) now has `fediversity.incus`
declared, tripping the guard and defining `fediversity.openbao-pki` where it
is undeclared. Narrow the guard to the specific option
(`options.fediversity or {} ? openbao-pki`), matching its comment.
Both verified via `nix flake show ./core` (exit 0) and
`nix eval ./core#checks.x86_64-linux.integrations-spire-incus.drvPath` (builds
the test drv) on fediversity-ci; parent-state reproduced both failures.
Assisted-by: Claude:claude-opus-4-8
The `hostingProviderBridgeCnames` served-app list was hand-maintained as
`[openbao windmill netbox]`, which was both over- and under-inclusive: an
audit of the routing + trust-domain data shows `windmill`/`netbox` serve a
PKI cert but are never dialed cross-node by their served name (so they need
no bootstrap alias), while `valkey` (`valkeyHost` = `valkey.<domain>`, dialed
by every redis consumer) and `postgresql` (`postgresql.<domain>`, dialed by
every postgres consumer) DO hit the same served-vs-instance TLS-CN gap and
were missing.
Replace the list with the intersection of "serves a PKI cert under
`<app>.<domain>`" and "dialed cross-node by that served name over TLS":
`openbao`, `valkey`, `postgresql`. `monitoring` is dialed cross-node but its
served domain equals its instance FQDN, so the existing `c.name != c.target`
filter already drops it; `terraform-backend` is off-apex and dropped by the
domain-suffix filter. Both consumers' trust domains
(`valkeyPkiTrustDomains`/`postgresqlTrustDomains`) already name the served
name, so only DNS resolution was missing -- which these CNAMEs now provide.
Assisted-by: Claude:claude-opus-4-8
A fresh `openbao-pki` consumer needs `/var/lib/openbao-init/pki-ca.pem` to
trust the openbao `:443` vhost, but nothing delivered it: the private CA is
generated at OpenBao runtime (key never leaves OpenBao), so there is no
build-time file to bake, and `openbao-pki-trust.service` stayed inert
(`ConditionPathExists` unmet) -> `vars-openbao-login`'s `bao login -method=cert`
failed `x509: certificate signed by unknown authority`.
The consumer cannot fetch the CA over the openbao `:443` face (that TLS
handshake needs the very CA), so mirror the proven SPIRE trust-bundle
self-fetch: the openbao node publishes the (public) CA into a world-readable
web root and serves it over plain HTTP on `caFetchPort` (`8083`, sibling of the
spire-server's `bundleFetchPort` `8082`); each consumer runs an
`openbao-pki-ca-fetch` oneshot ordered `before` both the trust-store install
and `vars-openbao-login`. Opt-out by leaving `caFetchUrl` empty (tests pre-seed
the CA + a static token); a pre-seeded `caCertFile` always wins.
Wired in `hosts-common.nix`: consumers on the tf-incus path fetch from the
central openbao instance FQDN (all PKI-served peers chain to the one central
CA anchor); the openbao node itself opts out.
Assisted-by: Claude:claude-opus-4-8
Production OpenBao runs `tls_disable=true` on loopback with nginx
terminating TLS at `:443`. The SPIRE client cert for `bao login
-method=cert` is consumed by nginx and never reaches OpenBao's cert-auth
backend, which extracts the client cert from OpenBao's OWN TLS handshake
(`400 tls connection required`). So OpenBao itself must terminate mTLS.
OpenBao validates all listeners at boot and refuses to start on a missing
cert file, but the self-issued PKI leaf only exists AFTER OpenBao serves
loopback (to mount its own PKI) and lives on `/run` tmpfs. Seed the mTLS
listener with a store-path self-signed cert (always present -> no boot
deadlock), then `openbao-listener-leaf.service` swaps in the real leaf and
SIGHUPs OpenBao once `openbao-pki-${domain}.service` issues it. Consumers
only trust the real leaf's CA and retry their login, so no insecure window
and no `VAULT_SKIP_VERIFY`.
Move the loopback admin API off `:8200` -> `:8250` (gate-only) so the mTLS
listener owns `0.0.0.0:8200`. Point the cert-login/secret-read consumer
addresses (`vars-openbao.address` HP + operator, `centralAddress`) at
`:8200`; keep the token-authed PKI-issuance address on nginx `:443`. All
gated on `hasOpenbaoPki` (production internalPkiNode) so standalone tests
that `mkForce` their own `:8200` listener are untouched.
Assisted-by: Claude:claude-opus-4-8
Two rotation-robustness fixes for the SPIRE rung, both proven live on the
incus host:
- `spire-agent-trust-bundle` now refreshes a self-fetched bundle when the
server CA rotates, instead of short-circuiting whenever any bundle exists.
A `.self-fetched` marker distinguishes our bundle (refresh it) from an
operator pre-seed (never touch it); the refresh is best-effort and never
wipes a working bundle when the server is briefly unreachable.
- `spire-register` now reconciles on parent ID, not just presence of the
spiffeID. After an EK rotation the old "create if the entry is missing"
left the entry stranded on the OLD agent parent, so the new EK stayed
unregistered; the reconcile deletes the stale entry, recreates it under the
new parent, and prunes the orphaned EK-hash file.
Assisted-by: Claude:claude-opus-4-8
The hosting-provider and operator group skeletons shipped their nodes with
`<app>-placeholder` `network.name` labels. Now that instance names follow
`network.name` and sensible DNS labels are supplied by automation (the incus
bridge and NetBox), the suffix only made the live container/DNS names ugly.
Rename `<app>-placeholder` -> `<app>` for every affected node, and follow the
label through everywhere it is consumed as a string: the cross-node FQDN
fallback literals in `hosts-common.nix`, the monitoring node's own `domain`,
and the octodns zone-check assertion. The `0.0.0.0`/`::` addresses stay as the
sole unallocated-host collision guard, and the `# TODO operator` comments now
name the placeholder ADDRESS (not the label) as that guard.
Unrelated `dev-placeholder` (authelia notifier) and `non-placeholder` (comment
prose) tokens are left untouched.
Assisted-by: Claude:claude-opus-4-8
Cross-node consumers reach the group openbao over its private-CA TLS face
(`https://openbao.<domain>:8200` mTLS listener / `:443` nginx vhost), whose
leaf chains to the openbao node's private CA. Every `bao` call and the openbao
agent that dialed it did so without a CA, so on the live incus deploy they
failed `x509: signed by unknown authority` -- blocking the openbao rung of the
bootstrap ladder.
Thread a `caCertFile` through the `vars-openbao` provider (option) and the
`generateFiles` policy/consumer, defaulting it on the hosting-provider path to
the already-provisioned `/var/lib/openbao-init/pki-ca.pem` (fetched before
`vars-openbao-login` by `openbao-pki-ca-fetch`, the same anchor as
valkey/postgres trust). Export `VAULT_CACERT` in the vars upload, the
`vars-openbao-login` / `openbao-pki-login` `bao login`, the `openbao-pki`
issue unit, and the netbox token `bao kv put`; set `vault.ca_cert` on the
shared agent. All gated on a non-empty `caCertFile`, so plain-`http://` tests
stay byte-identical (empty CA, no `VAULT_CACERT`).
Assisted-by: Claude:claude-opus-4-8
The on-node OpenBao engine reconcilers (`valkey-engine.nix`,
`ldap-engine.nix`, `oidc.nix`) hard-coded `VAULT_ADDR =
http://127.0.0.1:8200`. On a production PKI node the public mTLS listener
owns `:8200` and the plaintext loopback admin/bootstrap API moves to
`:8250` (`openbao.nix`'s `loopbackPort`), so those engines would dial the
mTLS listener over plain HTTP and fail -- blocking the valkey/ldap/oidc
engine configuration on the live incus openbao node.
Factor the port derivation into `loopback-api.nix` (keyed on
`options.fediversity ? openbao-pki`, mirroring `hasOpenbaoPki`) and have
`openbao.nix` plus the three engines import it, so they agree on the port
without re-deriving the rule. Standalone engine tests import their engine
WITHOUT `openbao.nix`, so `hasOpenbaoPki` is false and the address stays
`:8200` -- byte-identical to the former literal (regression-safe).
Verified on `fediversity-ci`: on `hosting-provider-openbao` all three
engines resolve to `http://127.0.0.1:8250`; the standalone
`test-ldap-engine` node keeps `http://127.0.0.1:8200`; `test-ldap-engine`
and `test-valkey-service` build green and the full openbao node toplevel
evaluates.
Assisted-by: Claude:claude-opus-4-8
The `BOOTSTRAP=1` migrate auto-derives the `terraform-backend` node's incus-
bridge lease as its target, but that address is host-internal NAT: reachable from
the incus host, not from a separate workstation running the deploy. Until now a
remote deployer had to look up the lease, open `ssh -L` by hand, and pin
`TF_HTTP_ADDRESS` at a `localhost` forward (documented, but three manual steps on
the critical migrate path).
Add an explicit opt-in seam: when `BOOTSTRAP_TUNNEL_SSH` is set to an SSH
destination for the incus host, the wrapper forwards a free local port to the
derived bridge lease through that host and re-points the migrate at the local
forward itself. The tunnel is torn down by an EXIT trap. `BOOTSTRAP_TUNNEL_SSH`
is deliberately distinct from `TF_VAR_host` (an incus-API coordinate that need
not be SSH-reachable); `BOOTSTRAP_TUNNEL_SSH_OPTS` carries extra `ssh` flags.
Explicit opt-in, no reachability auto-detection: unset leaves the on-host
auto-derive and operator-pinned paths byte-identical, and CI never reaches this
branch (it does not set `BOOTSTRAP=1`). `openssh`/`python3`/`procps` join the
wrapper's `runtimeInputs` for the tunnel, port picker, and child reaping; all
inert unless the seam is set.
Assisted-by: Claude:claude-opus-4-8
The incus deploy runner's `remote_exec` wraps every `incus exec` in a
`timeout ${REMOTE_EXEC_TIMEOUT:-60}` whose purpose is to sever a wedged
*D-Bus* call so the recovery loop can take over. Two bulk-disk steps --
the bridge-cache substitution (`nix-store -r`) and the delta-NAR import
(`nix-store --import`) -- inherited that 60s default.
Under a concurrent multi-node bootstrap (7+ nodes importing at once) the
host's shared store I/O contends and a single import runs well past 60s.
`timeout` then reaps it with rc=124, which propagates as a `local-exec`
provisioner error and aborts the whole apply mid-deploy. Observed live on
`192.168.51.101`: lldap's closure push reached `100%`, the following import
timed out at 60s, and the apply failed before activation even started --
never reaching the state migrate.
Set `REMOTE_EXEC_TIMEOUT=900` on just those two bulk calls (inline env
prefix, so the D-Bus-recovery loop below keeps its tight 60s/30s/8s
bounds). The substitution stays `|| true` (non-fatal), but the headroom
lets it actually complete so the delta pushed over the slow WAN leg stays
small.
Assisted-by: Claude:claude-opus-4-8
The local-then-migrate bootstrap (`BOOTSTRAP=1`) auto-derives the migrate
target from the just-applied `terraform-backend` node's bridge lease, but only
re-pointed `TF_HTTP_ADDRESS`. Two sibling addresses stayed baked at the
eval-time `localhost:8080`:
- The HTTP backend's `lock_address`/`unlock_address` (`TF_HTTP_LOCK_ADDRESS` /
`TF_HTTP_UNLOCK_ADDRESS`, both defaulting to `address`). The migrate's FIRST
action is `LOCK`, so it failed `connect: connection refused` on
`localhost:8080` even though the state address was correct.
- The self-state seed's `TF_VAR_self_state_config.address` (`seed-self-state.sh`,
#833), invoked post-migrate with no local override. It locked `localhost:8080`,
got HTTP 000, and skipped the seed -- leaving the netbox self-read to hard-fail
"No stored state".
Introduce `repoint_http_backend()` to swap the host[:port] across all four in
lockstep (state, lock, unlock, and the `http`-backend self-state config's
`.address`), on both the on-host derive path and the `BOOTSTRAP_TUNNEL_SSH`
remote-deployer path. Scheme, `/state/...` path, `-<group>` suffix, and the
self-state username/password are preserved.
Proven live on 192.168.51.101: the migrate now locks the real bridge lease,
`Apply complete` on the migrated HTTP backend, `.migrated` marker written, and
the openbao/spire/tf-backend ladder converged (openbao initialized + unsealed).
Assisted-by: Claude:claude-opus-4-8
The bootstrap working dir (`$XDG_STATE_HOME/fediversity/bootstrap/<key>`)
holds local tofu state and its `.backup`/`.migrated` copies before the
state is migrated to the HTTP backend. Tofu state is a plaintext secret
sink; this graph only ever REFERENCES secrets by address (e.g.
`data.vault_kv_secret_v2.netbox_api_token`), so nothing sensitive is
serialized today, but restrict the tree to the owner so a future
secret-valued resource could not leak world-readably regardless.
Assisted-by: Claude:claude-opus-4-8
f55e15e99 made the shared openbao agent, vars-openbao-login, the openbao-pki
issue unit, and the netbox token put load a `ca_cert`/`VAULT_CACERT` whenever
`caCertFile` was non-empty. But the hosting-provider default sets `address`
(`https://openbao.<domain>:8200`) and `caCertFile`
(`/var/lib/openbao-init/pki-ca.pem`) via two INDEPENDENT `mkDefault`s, and a
`test-*-service` check overrides only `address` to a plain `http://` dev
openbao -- leaving `caCertFile` at its default. So the agent got a
`ca_cert` pointing at a file that never exists in the test topology:
`openbao-agent-default` failed `Error loading CA File: ... no such file or
directory`, secrets were never delivered, authelia rendered an empty config
(`storage/notifier/oidc clients must be provided`), and mastodon/peertube/
pixelfed/apps-{tf,api,panel,tf-incus} all timed out downstream.
Gate the CA on the resolved address SCHEME (`https://`) instead of merely a
non-empty `caCertFile`, at all four sites. Plain-`http://` tests stay
byte-identical (no CA); the live `https://` path still trusts its private CA.
Also add a direct `openbao-pki-ca-fetch` -> `openbao-agent-default` before-edge
(the transitive path via vars-openbao-login already existed) so the agent never
races the CA fetch on the live path.
Assisted-by: Claude:claude-opus-4-8
The `upload-vars-openbao` / fetch `authEnv` still gated `VAULT_CACERT` on
`caCertFile != ""` alone -- the same coupling bug fixed at the five other
CA sites in `0a99b3259`. A `test-*-service` check overrides only the
`vars-openbao` `address` (to a plain `http://` dev openbao) while leaving
`caCertFile` at its hosting-provider default (`/var/lib/openbao-init/pki-ca.pem`,
which never exists in the test), so the upload exported `VAULT_CACERT` at a
missing path and died `Error loading CA File: ... no such file or directory`
before writing any KV. The agent then read absent secrets
(`authelia-oidc/oidcClientSecretDigest`, `authelia-smtp/relayPassword`), exited
1, and authelia rendered an empty config -- the same downstream stall.
Couple the export to `lib.hasPrefix "https://" cfg.address` so plain-http tests
stay unset while the live https path still verifies against its CA.
Assisted-by: Claude:claude-opus-4-8
C1 renamed the incus instance to `coalesce(network_name, key)`; the operator
group sets `pixelfed`'s `network.name` to a distinct DNS label (`test04`), so the
deployed container is `test04`, not `pixelfed`. The apps-api/apps-panel/apps-tf-incus
test scripts hardcoded `pixelfed` in their `incus exec`/`incus list | grep` checks,
so they hit "Instance not found" / 900s timeouts. Resolve the instance name via a
new `instanceNameFor` helper and substitute `@pixelfedInstance@` at each check site.
Also carries the steady-state bootstrap tunnel fix (`tunnel_steady_if_remote`) so a
remote deployer opens the SSH tunnel on the `.migrated` short-circuit path, not only
in the MIGRATE branch.
Assisted-by: Claude:claude-opus-4-8
`incus admin init --preseed` PATCHes -- it updates only keys PRESENT in the
document and leaves absent keys untouched. The bridge CNAME block was emitted via
`lib.optionalAttrs (bridgeCnames != [])`, so once the served-app set or a node`s
`network.name` changed and the alias list went empty, the omitted `raw.dnsmasq`
could not clear the CNAMEs a prior deploy had written. A stale
`cname=openbao.abundos.eu,openbao-placeholder.abundos.eu` (the `-placeholder`
label predating `540d989d0`) then NXDOMAINed, wedging the cross-node
openbao PKI/secret bootstrap (`openbao-pki-ca-fetch` -> agent -> `upload-vars`),
so authelia never rendered its config and served `502`.
Set the key unconditionally, to an empty string when there are no aliases, so a
redeploy PATCHes the desired absent state over any stale value.
Assisted-by: Claude:claude-opus-4-8
openbao TPM-sealed auto-unseal persisted its owner-hierarchy seal primary
at `0x81010001`, the TCG-reserved RSA Endorsement Key handle. On a node
that also runs a spire-agent (every hosting-provider node with a vTPM),
`tpm2_evictcontrol` clobbered the EK, so the SPIRE `tpm` node-attestor's
`activate_credential` failed deterministically with `0x1d`
(`TPM_RC_POLICY_FAIL`) on openbao alone. Move the seal primary to an
application-range owner handle (`0x81000001`) so the two vTPM consumers no
longer collide.
Assisted-by: Claude:claude-opus-4-8
Add operator recovery guidance for three hazards observed on a live incus
provider bootstrap:
- The `nimbolus/terraform-backend` server holds its lock in memory and
rejects a non-matching unlock with `HTTP 400 (locks not equal)` even under
`-force`, so a lock stranded by a crashed apply is cleared by restarting
the backend service (in-memory lock; on-disk state untouched), not by
`force-unlock`.
- The `-all` graph deploys its own state backend, so a reconcile that rebuilds
the `terraform-backend` node restarts its `:8080` service and can race a
concurrent state upload; prefer a no-op steady apply or `-target` to avoid
reconciling the backend node.
- A vTPM recreate mints a fresh EK, so SPIRE attestation fails server-side with
`could not validate EK` until `register_ek` re-pushes on the next COMPLETED
apply; distinct from a `0x1d` handle collision.
Assisted-by: Claude:claude-opus-4-8
The state-encryption `kmsKey` was generated by `openssl rand` on the
tf-backend node, then round-tripped through the OpenBao `vars` agent for
delivery -- the only `generateFiles` provider is `vars`/openbao. No other
node consumes this key; it is local entropy for encrypting this node's own
state at rest.
That round-trip created a circular deadlock: the `-all` graph's state
backend lives on this node, yet `terraform-backend.service` waited on
`openbao-agent-default` to render the key, and that agent needs a SPIRE
token whose attestation can break (e.g. a vTPM EK rotation). A broken
attestation then wedged the very state store the graph needs to redeploy
and repair that attestation -- the graph could not self-heal.
Generate the key locally via a `terraform-backend-kmskey` oneshot,
persisted (generate-once-if-absent) in the same `StateDirectory` as the
state it encrypts. Drop the `openbao-agent-default` after/wants and the
`terraform-backend` special-case in `hosts-common.nix`'s
`inModuleVarsPlumbing`. The state store now comes up independently of
OpenBao, so the graph can always self-heal an EK rotation.
Assisted-by: Claude:claude-opus-4-8
Replace the bespoke systemd oneshot with the vendored `on-machine`
`varsBackend` provider: keep the kmsKey on the `generateFiles` contract
(consistent with every other generated secret), just route THIS instance
off openbao to local disk. `vars-openbao` gates its upload/agent-templates
on `isSelected`, so with `on-machine` forced the OpenBao agent is inert --
the key renders straight to `/var/lib/vars/secret/terraform-backend/kmsKey`
with no attestation. Mirrors the existing `netbox-test.nix` routing.
The rationale is unchanged and load-bearing: this node hosts the `-all`
graph's own state backend, so it must never sit downstream of the
OpenBao/SPIRE attestation the graph exists to repair. A vTPM EK rotation
that breaks attestation would otherwise wedge the state store and lock out
the redeploy that fixes it.
Update `bootstrap-seeds.md` + `bootstrap-full-ladder.md`: they previously
argued the openbao route was correct for the kmsKey and a local pin would
be "the regression". That assumed openbao is reliably available at the
handoff; the deadlock is exactly when it is not. Local is the correct home
for a per-node, single-consumer, must-not-change state-at-rest key.
Assisted-by: Claude:claude-opus-4-8
The OpenBao cert-auth trusted CA was seeded once at first boot and never
refreshed: `openbao-spire-ca-bundle` skipped the fetch whenever the bundle
file already existed, and `openbao-spire-auth` is a `RemainAfterExit`
oneshot that runs once. SPIRE rotates its CA roughly daily, so after one
CA lifetime (~24h) the pinned CA in every `auth/cert/certs/<host>` role
was stale and every SVID `auth/cert/login` failed with `invalid
certificate` cluster-wide -- any node offline longer than a CA lifetime
stayed locked out until a redeploy.
Give the fetch service a refresh path mirroring `spire-agent.nix`
(`.self-fetched` marker distinguishes our fetch from an operator
pre-seed; re-fetch, `cmp`, install-on-change, keep-old-on-failure). On a
changed bundle it `try-restart`s `openbao-spire-auth`, whose idempotent
`bao write` re-seeds every role from the fresh CA. A 1h timer drives the
refresh between deploys via a small trigger unit, because a timer cannot
re-activate an already-active `RemainAfterExit` oneshot.
Also add `diffutils` to both bundle-fetch services' `path` so `cmp` in
the refresh path resolves (it was absent on the `spire-agent.nix` side,
silently breaking its own refresh change-detection).
Assisted-by: Claude:claude-opus-4-8
The `.self-fetched` marker distinguished a bundle WE fetched from an
operator pre-seed, and refused to refresh a markerless bundle ("never
touch it"). But an openbao/agent upgrading IN PLACE has a pre-fix bundle
with no marker, so it is misclassified as an operator pre-seed and frozen
on whatever CA it booted with -- stranding cert-auth on a stale SPIRE CA
after the daily rotation (every SVID `auth/cert/login` then 400s), which
is the very drift this was meant to fix.
The SPIRE server is authoritative for its own CA, so the bundle it serves
is always the trust root to pin; there is no case where a stale local copy
(pre-seed or otherwise) should win. Drop the exemption and the marker
entirely: a present bundle is best-effort refreshed from the live server,
an absent one cold-starts. A pre-seed is just a warm start of the present
case -- the first refresh brings it up to the current CA.
Applied to both the openbao cert-auth fetch and the spire-agent trust
bundle fetch (same pattern, same latent upgrade-strand).
Assisted-by: Claude:claude-opus-4-8
The CA-bundle fetch service (`openbao-spire-ca-bundle`) is ordered
`before openbao-spire-auth.service` and, on a detected rotation, called
`systemctl try-restart openbao-spire-auth` SYNCHRONOUSLY from inside its
own ExecStart. At boot/redeploy the auth start-job is enqueued in the same
transaction ordered `after` the fetch, so the synchronous restart blocked
forever -- auth could not start until the fetch finished, and the fetch
could not finish until the restart returned. This wedged the fetch, auth,
and `multi-user.target`, so a rotated CA was fetched but never re-seeded
into the cert-auth roles and every SVID login kept 400ing.
Enqueue the restart with `--no-block` and return; systemd runs auth once
the fetch completes (ordering satisfied), and auth re-seeds from the fresh
bundle. On the periodic timer path (auth already active) the non-blocking
restart re-runs it too.
Assisted-by: Claude:claude-opus-4-8
`spire-login.nix` writes the OpenBao token to `/run/vars-openbao/token`, but
`consumer.nix` defaults the shared agent's `token_file` auto-auth (via
`services.vars-openbao.tokenFile`) to `/run/keys/vars-openbao-token`. Production
never reconciled the two, so the login wrote one path while the agent waited
forever on the other -- `openbao-agent-default` logged `error reading token file`
on every retry, no consumer secret (e.g. authelia's `ldap-bind` bindPassword) was
ever rendered, and the frontends stayed 502 even with cert-auth fully working.
Set `services.vars-openbao.tokenFile` from spire-login to the path it writes, so
the agent reads the same file. Tests already set both sides to a matching path
(`openbao-test.nix`, `testers/lib.nix`), which is why only the production SPIRE
path hit this.
Assisted-by: Claude:claude-opus-4-8
The check hard-coded `tokenFile = /run/vars-openbao/token` in its consumer
import, pre-matching what `spire-login.nix` writes -- so it passed even while
production diverged (consumer at the default `/run/keys/vars-openbao-token`,
login writing `/run/vars-openbao/token`). Leave the consumer at its default in
the check (as production does) so `spire-login.nix` must itself reconcile the
agent's token path; give `consumer.nix`'s `tokenFile` arg that same default so
omitting it is production-representative rather than an eval error.
Assisted-by: Claude:claude-opus-4-8
The ancilliary provider nodes that generate and upload secrets via the
setup's `varsProviderResources` registry -- `lldap`, `smtp`, `postgresql`,
`garage` -- had no `auth/cert/certs/<node>` role, because the openbao
module's `consumerHosts` default only lists enabled applications and the
`authelia` generator. Without a role, each node's `bao login -method=cert
name=<node>` fails `invalid certificate`, so its `vars-openbao-login`
never yields a token and `upload-vars-openbao` never lands its instance.
On the hosting-provider deploy this strands the whole authelia chain: the
lldap `admin` bind password never reaches KV, so `openbao-ldap-engine`
waits forever on `vars/secret/ldap-bind-<group>/bindPassword`, the
`authelia-bind` static role is never registered, and `authelia-main`
crash-loops -- taking the SSO/demo/redis frontends down.
Neither existing test caught it: `integrations-spire-openbao` overrides
`consumerHosts` with its own roster, and `apps-tf` uses a dev-stub openbao
that uploads with the root token rather than cert-auth. So the gap only
surfaces on a real hosting-provider deploy.
Thread the generation nodes as a separate `varsProviderHosts` roster that
the module unions into `effectiveConsumerHosts` (mirroring the valkey
writer roster), rather than poking `consumerHosts` from the setup: an
`attrsOf` `default` is replaced wholesale by any definition, so a
setup-level `consumerHosts =` would discard the module's rich static
default. The roster is derived from the same `varsProviderResources` list,
so each node's granted instance names (group-qualified where the resource
qualifies them, e.g. `ldap-bind-<group>`) match exactly what it writes.
Assisted-by: Claude:claude-opus-4-8
The `lldap-provision-users` service was gated wholesale on a non-empty
operator recovery `email`, and derived the machine `authelia-bind`
account's synthetic mailbox from that same `email`. On a bare
`core {}` hosting-provider deploy (no operator config feeding
`default-configuration.email`), `email` is empty, so the service did not
exist at all -- the `authelia-bind` account was never created.
The OpenBao LDAP engine's `authelia-bind` static role pins that exact
entry (`dn=uid=authelia-bind,ou=people,<baseDN>`), so its rotation write
failed `LDAP Result Code 32 "No Such Object"`, leaving the
`authelia-bind` credential unrotated, authelia's bind broken, and the
SSO frontends down. Reproducible for any hosting provider deploying via
`core {}` without a recovery email set.
Decouple the two accounts: the machine `authelia-bind` account is now
ALWAYS seeded (its mailbox domain derived from `baseDN`, not the operator
email), and only the human `administrator` stays gated on a non-empty
`email` (it is unusable without one -- its password comes from lldap's
email reset flow).
Assisted-by: Claude:claude-opus-4-8
The central hosting-provider LDAP engine lists every group's directory --
its own plus each operator group's -- but an operator group is deployed
out-of-band by its own TF graph, after the hosting-provider node, or not
at all (a hosting-provider-only deployment). The per-directory configure
step waited a bounded time for that group's lldap node to upload its
management-bind password to KV and then read it unconditionally, so a
missing entry hard-failed the whole oneshot under `set -e` -- taking down
this group's OWN directory (processed earlier in the same loop) and
leaving Authelia's bind unrotated.
Treat a missing management-bind password as EXPECTED rather than fatal:
after the bounded wait, skip that directory (non-blocking) and let a
later engine rerun configure it once the group deploys. A bare
hosting-provider deploy now configures its own `ldap` directory and skips
the not-yet-present `ldap-operator` one instead of blocking forever on
`vars/secret/ldap-bind-operator/bindPassword`.
Assisted-by: Claude:claude-opus-4-8
The maddy relay owns the single `smtp` vars instance and uploads one
submission password per consumer (`vars/secret/smtp/<consumer>Password`).
Each consumer node (authelia, monitoring, netbox, windmill) renders its
own submission password back through its openbao agent to authenticate
to the relay's `submission://` listener, but no policy granted them read
on that path -- so the render hit `403 permission denied`, the authelia
config template failed, and authelia crash-looped.
Add a shared read grant on `secret/{data,metadata}/vars/secret/smtp/*`
in the seed loop, mirroring the `authelia-oidc`/`postgresql-admin`
pattern (the relay keeps its own per-instance write grant). Gated on a
relay actually being deployed (`smtpProvided`), so a relay-less
deployment grants nothing. Reproducible for any hosting provider whose
group runs the maddy relay.
Assisted-by: Claude:claude-opus-4-8
The openbao agent renders `bindPasswordFile` as its root process, so
without a chown the file lands `0640 root:root` -- unreadable by the
non-root `authelia-main` user whose `template` config filter opens it at
start. Config validation then fails `bindPasswordFile: permission
denied` and authelia never starts.
Tighten the perms to `0400` and chain an explicit `chown
authelia-main:authelia-main` in the render `command`, on both the engine
and non-engine template branches -- the same chown the SMTP
submission-password render already uses (`hosts-common.nix`). On the
engine path the chown runs before the rotation `try-restart` (a no-op
before first start, so it does not fight the boot ordering).
Reproducible for any hosting provider.
Assisted-by: Claude:claude-opus-4-8
The maddy relay's served domain, submission-account username domain, and
TLS cert CN all derived from `mail.<group-domain>`, and authelia's
notifier dialed `submission://mail.<domain>:587`. But no node is named
`mail`, so nothing answers `mail.<domain>` on a fresh incus provider --
the bridge dnsmasq serves `<instance-name>.<domain>` from each lease, and
the relay node is named `smtp` (see `internalOnlyNodes`). The notifier
startup check then failed `dial tcp: lookup mail.<domain>: no such host`
and authelia crash-looped.
Derive the relay domain from the `smtp` label instead
(`smtp.<group-domain>`), at both the resource `domain` and the
`smtpDomain` cert-issuance site, so the notifier address + cert CN both
become `smtp.<domain>` -- the name the bridge already resolves from the
relay's lease, with no extra DNS record. The `dev` group's external
relay (`mail.protagio.nl`) is untouched. Reproducible for any hosting
provider deploying the maddy relay on incus.
Assisted-by: Claude:claude-opus-4-8
The consumer trust bootstrap fetched the served PKI CA only on first boot
(`if [ -s "$ca" ]; then skip`), leaving a stale `caCertFile` in place forever
once the openbao node`s CA rotated (a fresh `openbao-pki-init` or an openbao
re-init). Every TLS client on the node then failed `x509: signed by unknown
authority` cluster-wide -- the `bao login`, the openbao agent`s `vault.ca_cert`,
the DSN-lease clients -- with no self-heal.
`openbao-pki-ca-fetch` now runs on every (re)start and converges on the served
CA: fetch to a tmp file, `cmp -s` against the current one, and `install` only on
mismatch (a no-op once converged, so it does not churn a matching pre-seed). On
a rotation replace it `systemctl try-restart openbao-agent-default.service` so
the agent reloads the new `vault.ca_cert` (cached at start). First boot still
blocks the full `caFetchTimeout`; a rotation check tries briefly and keeps the
existing CA if the openbao node is unreachable, never blocking boot.
Assisted-by: Claude:claude-opus-4-8
The PKI provider installed every issued key `install -m 0644` -- world-readable
on every `openbao-pki` node. nginx tolerated it so it went unnoticed, but it is
a latent credential leak, and PostgreSQL flatly refuses a key with any group or
world access (it FATALs on startup), so the postgres node could never terminate
TLS on a PKI-issued key.
Tighten the provider default to `0640` root-owned (a private key must never be
world-readable), and let a consumer whose terminating service reads the key as a
non-root user thread ownership through its SSL request: add optional
`keyOwner`/`keyGroup`/`keyMode` fields to the shared `ssl` contract `request`,
which `openbao-pki`s issuing unit applies via `install -o/-g -m` (it runs as
root; the consumer`s module declares the user/group). Providers that cannot
honour ownership (a store-baked self-signed key) ignore the fields.
The postgres node threads `postgres:postgres 0600` for its own served domain,
gated on `servesPki` (self-signed already bakes a root-readable store-path key,
which postgres reads fine). Non-postgres PKI consumers keep the safe `0640`
default with no owner.
Assisted-by: Claude:claude-opus-4-8
Follow-up to 7eb38b107 (relay served on `smtp.<domain>` so its FQDN resolves off
the incus bridge lease). The standalone app-model default still read
`mail.abundos.eu`, and four `hosts-common` comments still said
`authelia@mail.<domain>`; align both to `smtp.<domain>` so the default and the
docs match the served name. The wired setup path overrides the app default from
the resource; the default only applies to a standalone app-model instance.
Assisted-by: Claude:claude-opus-4-8
The `spire-agent-trust-bundle` oneshot was `RemainAfterExit = true` and only
`wantedBy` `spire-agent.service`. `switch-to-configuration` leaves an unchanged
`RemainAfterExit` oneshot `active (exited)` and never re-runs it, so its
(already-correct) `cmp`-based refresh logic never fired on a redeploy. The SPIRE
server CA rotates on its own 24h lifetime, so a provider redeploying after a
rotation kept a STALE seed bundle on disk; the agent then could not verify the
server leaf (`x509: certificate signed by unknown authority`) and failed to
attest -- stranding every downstream cert-auth consumer (no SVID -> no
`vars-openbao` token -> `panel`/services down).
Drop `RemainAfterExit` and add `multi-user.target` to `wantedBy` so the fetch
re-runs on every converge; the script already refreshes only on a `cmp`
mismatch, so a current bundle is a no-op ("re-fetch only when stale"). Update
the `spire-incus` check to wait on the installed bundle file rather than the
oneshot being `active` (a non-`RemainAfterExit` oneshot is `inactive` once
done).
Assisted-by: Claude:claude-opus-4-8
`openbao-spire-auth` generated each cert-auth role (its `auth/cert/certs/<N>`
path and `allowed_uri_sans` `spiffe://fediversity/host/<N>`) from the
`consumerHosts`/`varsProviderHosts` roster key -- the application attr-key
(e.g. `panel`). But every node presents its `network.name` instead: its SPIRE
entry is created as `host/<network.name>` (`spire.nix`) and its
`bao login -method=cert name=` sends `networking.hostName` = `network.name`
(`spire-login.nix`, `openbao-pki-login.nix`).
For nodes whose DNS/instance label differs from the app key -- the panel node
is `fedi201`, the terraform-backend node is `tf-backend` -- the login hit a
role that did not exist and whose `allowed_uri_sans` did not match the SVID,
so OpenBao rejected it with `400 invalid certificate`. On the panel node this
left `vars-openbao-login` unable to mint a token, so `panel.service` (and the
demo frontend) stayed down.
Add `services.openbao-spire-auth.spiffeHostNames` (roster key -> `network.name`,
default identity) and key the role path + `allowed_uri_sans` on it; the setup
derives the map from the group's `machines` data so it is correct for any
provider's naming. The `host-<N>` KV policy and per-instance grants stay keyed
on the roster key (unchanged). The 14 nodes where key == `network.name` are
unaffected; only `panel` -> `fedi201` and `terraform-backend` -> `tf-backend`
are remapped (eval-verified).
Invisible to CI: `integrations-spire-openbao` overrides `consumerHosts` and
`integrations-spire-incus` deliberately keys its roster on the SPIFFE name
(`tags-<app>`); `apps-tf` uploads with the root token, not cert-auth. This is
the declarative production equivalent of that test's manual key alignment.
Assisted-by: Claude:claude-opus-4-8
The panel node self-generates its Django `SECRET_KEY` under the `panel`
vars instance (`vars/secret/panel/secret_key`) and uploads it via
`upload-vars-openbao`, exactly like `netbox` and `terraform-backend`.
But its `consumerHosts` default entry was an empty list, so the
generated `host-panel` policy carried no write grant on
`secret/{data,metadata}/vars/secret/panel/*` and the upload failed with
`403 permission denied`.
This was latently masked: before the cert-auth role naming fix, the
panel login itself 400'd, so `upload-vars-openbao` never reached the
403. With the login healed, the missing grant surfaced.
Make `panel` own its `panel` instance, symmetric with the other
self-generating nodes.
Assisted-by: Claude:claude-opus-4-8
The openbao node's own `openbao-agent` logs in via
`bao login -method=cert name=openbao` and templates the shared
`postgresql-admin` password back to configure the database secrets
engine (which mints the per-app postgres roles every app DSN reads).
But `openbao` was absent from `consumerHosts`, so `openbao-spire-auth`
never created a `host-openbao` policy or an `auth/cert/certs/openbao`
role. The node's own login therefore 400s (`invalid certificate`,
no matching role), its agent never gets a token, `postgresql-admin`
is never rendered, `openbao-database-init` defers indefinitely, and no
`database/roles/<app>` exist -- so e.g. the panel's
`postgresql/creds/hosting-provider-panel` read 403s, its DSN env file
is never written, and `panel.service` fails to start (demo 502).
Add `openbao = [ ]` to the `consumerHosts` default: the node owns no
vars instance of its own, but must be listed to get the `host-openbao`
policy (which auto-includes the `postgresql-admin` shared read granted
to every consumer host) and the cert-auth role its login needs.
HP-facing: any fresh-provider deploy hits this the moment the database
engine is enabled -- the openbao node cannot configure its own engine
without authenticating to itself.
Assisted-by: Claude:claude-opus-4-8
The panel's production deployment runs on PostgreSQL: the `panel-db`
`postgresql` resource mints dynamic credentials and the openbao agent
renders a `postgresql://` DSN that `dj_database_url` maps to Django's
postgres backend. But the panel package's Python environment shipped
only `sqlite` and `dj-database-url`, never a postgres driver, so both
the migrate `preStart` and the ASGI server died with
`django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 or
psycopg module` -- leaving `panel.service` crash-looping and demo down.
Add `psycopg` (v3, which Django 5.2 loads for the postgresql backend) to
the package's Python dependencies. `sqlite` stays for the module's sqlite
default (tests + a DB-less deploy).
HP-facing: any deploy that wires the panel to its `postgresql` resource
(the default production path) needs the driver in the package closure.
Assisted-by: Claude:claude-opus-4-8
Since PostgreSQL 15 the `public` schema no longer grants `CREATE` to
`PUBLIC` by default. A dynamic-credential app (e.g. the panel) is served by
an OpenBao `database/roles/<role>` that mints a random-named role per lease;
that role is not the database owner, so it inherits only the schema's
default privileges and cannot create tables. Django's first `migrate` then
dies with `permission denied for schema public` / `Unable to create the
django_migrations table`.
The engine's `GRANT ALL ON SCHEMA public` in its `creation_statements`
cannot fix this: it runs on the admin connection to the `postgres`
database, so it grants on `postgres.public`, not on the app database's
schema.
Restore the pre-PG15 behavior locally on the provider node, scoped to each
dynamic-role app database (single-tenant: only that app's leased roles
connect there), with a `postgresql-dynamic-schema-grants` oneshot that
grants `CREATE ON SCHEMA public TO PUBLIC` on those databases. Rotating-role
databases are unaffected -- their fixed-name role owns the database and
already has `CREATE`. Idempotent.
Assisted-by: Claude:claude-opus-4-8
A `dynamic` postgresql consumer (panel, windmill) is served by a leased
credential: the OpenBao agent renders the DSN of a short-lived
`database/creds/<role>` lease (engine `default_ttl`, an hour by default)
to `result.urlFile`. When the lease rotates the agent writes a fresh DSN,
but a consumer that read the old DSN at start -- into an EnvironmentFile
(panel) or via `LoadCredential` (windmill) -- keeps using the now-revoked
role and starts failing with `password authentication failed for user
"v-cert-..."` within the lease TTL of every deploy.
The template `command` only `chown`ed the file, with no restart hook, so
nothing propagated the new DSN to the running consumer.
Add a `reloadUnits` request option: the dynamic-DSN template `command`
`systemctl try-restart --no-block`s those units after each re-render, so
the consumer reconnects with the fresh credential. `try-restart` skips
units that are not running (a re-render before first start is a no-op).
Wire the panel (`panel-db-env` + `panel`) and windmill (server + both
workers) consumers. Rotating consumers are unaffected -- their fixed-name
role's password is re-set in place, so the DSN string never changes.
Assisted-by: Claude:claude-opus-4-8
`spire-agent` caches its last SVID + key in `agent-data.json` and, on start,
reconnects to the server with that cached SVID BEFORE (re)attesting. A running,
attested agent tracks CA rotations over its authenticated stream and keeps the
file fresh, so it only goes stale when the agent is DOWN across a full
CA-rotation + SVID-expiry window (>24h) -- a node powered off, or a deploy that
left the agent stopped past the 24h CA lifetime. The cached SVID is then both
expired AND signed by a rotated-out CA, so the TLS handshake to the server fails
with `x509: certificate signed by unknown authority` and the agent can never get
far enough to re-attest: it loops forever, so no token, no DSN, and the node's
consumers (e.g. `windmill-server`) can't start.
The `spire-agent-trust-bundle` refresh does not fix this on its own: in the
observed case a peer node had already refreshed the on-disk bundle, so `cmp`
said "unchanged" while the persisted SVID stayed stale.
Add a `prune_stale_svid` guard to that oneshot: it extracts the persisted leaf
and verifies it against the CURRENT trust bundle with `openssl verify` (which
fails on both a rotated-out CA and an expired leaf), removing `agent-data.json`
only when it no longer verifies so the agent does a clean re-attest from the
TPM. The guard is fully defensive -- any parse/tooling failure leaves the file
untouched, so a healthy agent's cache is never wrongly deleted. `data_dir` is
pinned explicitly so the guard operates on the same path the daemon uses.
Assisted-by: Claude:claude-opus-4-8
The upstream `windmill-server`/`windmill-worker`/`windmill-worker-native`
units default to `Restart=always` with `RestartSec=100ms` against systemd's
default start-rate limit (`StartLimitBurst=5` / `StartLimitIntervalUSec=10s`).
On a fresh deploy the units first-start before `openbao-agent-default.service`
has attested and rendered the leased PostgreSQL DSN, so they exit non-zero and
`Restart=always` burns all five allowed restarts inside 500ms -- tripping
`start-limit-hit` permanently. The dynamic-DSN engine's re-render
`try-restart` is a no-op on a start-limited unit, so the units stay dead after
an otherwise-clean deploy and need a manual `systemctl reset-failed`.
Set `RestartSec=5s` on all three units so five retries span longer than the
agent's attest+render delay: the unit stays merely restarting (not
start-limited) until the DSN lands, at which point the normal retry -- or the
engine's re-render `try-restart` -- brings it up. Machine-agnostic: any node
whose openbao agent attests slightly slower than windmill first-starts would
otherwise wedge these units dead.
Assisted-by: Claude:claude-opus-4-8
Windmill's bundled DB migrations (e.g. `20221211192539: grant usage`) GRANT to
two group roles, `windmill_user` and `windmill_admin`, that must already exist,
and windmill `SET ROLE`s to `windmill_admin` at runtime. The upstream nixpkgs
windmill module creates these roles only in its `windmill-initdb` oneshot,
gated on `database.createLocally = true`. Our windmill consumes a REMOTE
database on `dynamic` OpenBao credentials, so `createLocally = false` and that
oneshot never runs -- the first migration dies with
`role "windmill_admin" does not exist` and `windmill-server` never starts.
Rather than baking a windmill-specific initdb into the `postgresql` resource,
extend the resource/contract with two GENERIC, app-declared capabilities so any
service with schema-level prerequisites the credential machinery does not cover
can express them itself (mirroring how an app supplies its own
`nixos-configuration`):
- `bootstrapSQL`: idempotent SQL the provider node runs ONCE as the postgres
superuser against the app's database, after `postgresql-setup`. Threaded to
the database node keyed by database and executed by a new
`postgresql-app-bootstrap-sql` oneshot. This is where the superuser socket
is -- an app config merges onto its OWN node, which has no superuser access
to the remote postgres.
- `dynamicRoleGrants`: roles each freshly-minted `dynamic` lease role is
granted membership in, folded into the OpenBao engine's dynamic
`creation_statements` as `GRANT <role> TO "{{name}}"`. A dynamic consumer
connects as a random per-lease role, so a group role its service `SET ROLE`s
to must be granted to every lease role, not once to a fixed user.
Windmill supplies both from its own app module: `bootstrapSQL` recreates the
`windmill_user`/`windmill_admin` roles exactly as the upstream initdb does
(minus the per-user grant, which `dynamicRoleGrants` handles per-lease), and
`dynamicRoleGrants = [ "windmill_admin" ]` lets each lease role assume it.
Machine-agnostic: any hosting provider deploying windmill on dynamic
credentials would otherwise hit the same dead `windmill-server`.
Assisted-by: Claude:claude-opus-4-8
Three earlier commits on this branch changed behavior without updating the
tests that assert against it, so CI went red on `88e08e3e`:
- `7eb38b10` renamed the maddy relay's served/cert domain from `mail.<domain>`
to `smtp.<domain>`, but the mastodon/peertube/pixelfed service tests still
list `mail.${rootDomain}` in `trustDomains` and look it up in `sslResults`,
which now errors `attribute '"mail.localhost"' missing`. Point the three
tests at `smtp.${rootDomain}`, the domain the relay's self-signed cert now
carries.
- `ca0caa39` stopped shipping the openbao-pki private key world-readable
(`0644` -> `0640 root:root`), but the `integrations-openbao-pki` test's nginx
reads the key as the non-root `nginx` user without threading `keyGroup`, so
nginx fails `cannot load certificate key ... Permission denied`. Set
`keyGroup = "nginx"` on the test's ssl request so nginx reads it via group.
- `07d8a9d7` made `spire-login.nix` repoint the agent's `token_file` at
`/run/vars-openbao/token` (where the login writes), but the
`resources-generateFiles` nix-unit test still expected the environment-level
`tokenFile` override to reach the agent. In the production-shaped operator
setup spire-login owns the token path, so assert the agent reads the file the
login writes.
Assisted-by: Claude:claude-opus-4-8
Records the root cause (windmill.nix + panel assume colocation; `panel`
group/user absent on the split windmill node -> workers 216/GROUP, bootstrap
`install: invalid user 'panel'`), what already flows node-agnostically via the
windmill job payload, and the two-half cross-node design (A: windmill token to
OpenBao KV like `netbox.nix`; B: ssh key + incus certs to KV for the worker),
plus the apps-panel test split needed to cover it. Scope of implementation TBD.
Assisted-by: Claude:claude-opus-4-8
windmill.nix + api assumed windmill and api run on one node, reading the
Windmill API token from a colocated file (`/var/lib/panel/windmill-token`).
In the hosting-provider roster they are separate nodes, so the api node had
no token file and job submission sent no `Authorization` header. A hosting
provider following our instructions hits the same split, so fix it properly
rather than colocate.
Producer (windmill node): the bootstrap oneshot already mints the token; it
now `bao kv put`s it to `vars/secret/windmill/apiToken` (base64) using the
node's vars-openbao token, mirroring the netbox token producer. Ordered after
`openbao-agent-default.service` (needs the vars token). Dropped the
`install -o panel` / `chown panel` local-file write (no panel group on this
node) and the now-dead `services.fediversity-windmill.tokenFile` option.
Grants (openbao node): `consumerHosts.windmill = ["windmill"]` (was `[]`) so
the node's per-instance write grant covers `vars/secret/windmill/*`, plus a
shared `windmill` read grant block (mirroring `authelia-oidc`) so consumer
hosts read it back.
Consumer (api node): the token rotates on every windmill redeploy (the mint
is not idempotent), so deliver it through the OpenBao broker
(`systemd-openbaod` `envSecrets`), which reloads the unit on rotation, rather
than a chown-only `extraTemplates` file that would strand the running api on
a stale token. `api.nix` requests
`contracts.envSecrets.want.api.windmillToken` into `fediversity-api.service`'s
env as `WINDMILL_TOKEN`; `windmill.py` reads `os.environ["WINDMILL_TOKEN"]`.
Dropped the `windmill_token_file` config field. `hosts-common.nix` emits the
bare vars/agent/broker consumer plumbing onto the api node (its
`inModuleVarsPlumbing` hook) since it now consumes a shared secret but
generates none.
The panel node is NOT a token consumer: `panel/nix/configuration.nix` declares
`windmill.tokenFile` but never reads it, and no panel Python touches the token
(panel only orchestrates via `incus_bootstrap.py`). So there is one consumer.
api-python-tests 199/199 green; windmill producer + api broker wiring eval
green. Half B (ssh key + incus certs) and C (drop cross-node restart) follow.
Assisted-by: Claude:claude-opus-4-8
Half A moved the windmill API token off the local
`/var/lib/panel/windmill-token` file onto an OpenBao KV path
(`bao kv put`), delivered to `fediversity-api` as `WINDMILL_TOKEN` via
the `systemd-openbaod` broker. Both `apps-api` and `apps-panel` still
waited on the bootstrap to write that file and `chmod o+r`'d it, so they
failed once the file stopped being written (`apps-api` additionally
`cp`'d into `/var/lib/panel`, which the bootstrap no longer creates).
Rewire both tests to the new mechanism, staying single-node: enable the
`systemd-openbaod` broker + its `envSecrets` provider (mirroring
`generateFiles/consumer.nix`) and declare the same
`envSecrets.want.api.windmillToken` request `api.nix` uses in
production, so the KV-publish -> broker -> env round-trip is exercised
against the dev openbao stub. Drop the now-dead `tokenFile` option and
its `chmod` wiring. `apps-api` creates `/var/lib/panel` via a tmpfiles
rule (`apps-panel` gets it from the panel service's `StateDirectory`).
Splitting the tests onto >=2 nodes is deferred to the test-split step
after half B.
Assisted-by: Claude:claude-opus-4-8
The incus (hypervisor) host now mints a one-time incus trust token pair
per deploy-runner node (`windmill` worker, `api`) and publishes them to
its `incus-deploy` vars KV instance under per-runner subpaths, so a
runner on a SEPARATE host can reach the incus HTTPS API. A trust token
is one-time (consumed by the first client-cert enrollment), so runners
cannot share one -- each gets its own fresh pair. The tf provider
self-generates its `client.crt`/`client.key` locally from the token and
self-fetches the servercert, so only the one-time token travels.
The producer oneshot is non-blocking (`WantedBy` + `Restart=on-failure`,
mirroring the incus host's spire/openbao agents): the incus host is
deployed before the OpenBao node exists, so the `bao kv put` settles
`failed` on the first fresh-provider boot and converges on a redeploy.
- `incus.nix`: `fediversity.incus.deployRunners` option + the
`incus-deploy-token-publish` producer oneshot; retires the #1013
"future work" TODO.
- `openbao.nix`: `consumerHosts.incus = ["incus-deploy"]` (write grant)
+ shared-read grant on `vars/secret/incus-deploy/*` (mirrors the
windmill API token).
- `hosts-common.nix`: derive `hostingProviderDeployRunners` from the
hosting-provider group's runner nodes and thread it into the incus
node's `deployRunners`.
Assisted-by: Claude:claude-opus-4-8
The windmill worker and api node now read their OWN one-time incus trust
token from OpenBao KV instead of a panel-owned local file, so they can
run on nodes separate from the panel.
- windmill.nix: openbao-agent `extraTemplates` renders this worker's
`incus-deploy/<hostName>/tokenHosts` to a `TF_VAR_token` EnvironmentFile;
worker units get `TF_VAR_incus_config_dir` + a `StateDirectory` for the
tf provider's self-generated client cert. Drops `SupplementaryGroups =
"panel"` + `/var/lib/panel/tf-env` from both worker units.
- api.nix: `incusToken` envSecrets request delivers `TF_VAR_token` into
`fediversity-api.service`'s env via the broker (like the windmill token).
- configuration.nix: `TF_VAR_incus_config_dir` + nested `StateDirectory`;
drops the `/var/lib/panel/tf-env` EnvironmentFile.
- deployment.py: exclude the runner-local `TF_VAR_token` /
`TF_VAR_incus_config_dir` from the windmill job payload so they cannot
clobber the worker's own values; the opaque `TF_VAR_image_fingerprint`
still rides the payload. The subprocess-fallback path already uses the
api node's own env.
- deploy.py (windmill job): strip the same runner-local vars from the
payload before merging over worker env, so the worker-local credential
always wins (defense-in-depth against a stale payload).
Assisted-by: Claude:claude-opus-4-8
The api and windmill deploy runners now read their incus credentials
from OpenBao KV and run on separate hosts from the panel, so the
panel-node `manage incus_bootstrap` convenience can neither reach nor
needs to restart them. Trim `PROD_SERVICES` to `panel.service` and
update the docstring + test to match. This tool stays the local-dev /
colocated path; the remote flow provisions runner creds via OpenBao.
Assisted-by: Claude:claude-opus-4-8
Rewrite the gap #38 notes + api/README + bootstrap.md to reflect the
implemented cross-node incus-credential flow: the incus host mints a
one-time trust token pair per deploy-runner node (`windmill` worker,
`api`), publishes them to OpenBao KV under `vars/secret/incus-deploy/`,
and each runner reads back its own token so the tf provider self-enrolls
its `client.crt`/`client.key` locally. Only the one-time token travels;
no key pair or servercert is shipped, and no `panel` group is needed.
Also documents dropping the panel-node cross-node `systemctl restart`
(`incus_bootstrap` now restarts only `panel.service`).
Assisted-by: Claude:claude-opus-4-8
The upstream nixpkgs windmill module already sets `StateDirectory` on the
`windmill-worker`/`windmill-worker-native` units (to their own state dirs).
Setting a bare `incus-deploy` collided (scalar conflict) and broke eval of
any node running the windmill worker (caught by apps-api). Preserve the
upstream dir by `mkForce`-ing the space-separated union so both the worker
state dir and the incus client-cert dir are created.
Assisted-by: Claude:claude-opus-4-8
The prior gap #38 Half B assumed the windmill worker unit env would reach
the deploy job, but our windmill build has no `WHITELIST_ENVS`, so it
sanitizes worker-service env out of job subprocesses. It also stripped
`TF_VAR_incus_config_dir` from the job payload and removed the api's
`tf-env` `EnvironmentFile`, so tofu got neither the config dir nor the
image fingerprint -- `apps-api`/`apps-panel` failed at the deploy step
with `No value for required variable "incus_config_dir"` /
`"image_fingerprint"`.
Classify each `TF_VAR_*` by how it must travel to a windmill worker that
may run on a SEPARATE node from the api submitter:
- `TF_VAR_token` -- a per-runner secret. The worker reads its OWN from the
openbao-agent render file directly (`deploy.py`), since neither the
payload nor the worker env can carry it. Rendered `root:windmill 0440`
via a post-render `command` chown (the openbao agent `template` stanza
has no `group` field, only `perms`) so the worker's DynamicUser
(primary group `windmill`) can read it.
- `TF_VAR_incus_config_dir` -- a node-local path. The api node's value
names its own dir, so it is stripped from the payload
(`_RUNNER_LOCAL_TF_VARS`); the worker defaults it to its own
`StateDirectory` (`INCUS_CONFIG_DIR`).
- `TF_VAR_image_fingerprint` -- a host-agnostic built-image ID. It rides
the job payload, so the api submitter reads it from its optional
`tf-env` `EnvironmentFile` (fingerprint only now; token via broker,
config dir per-node).
Drop the vestigial worker-unit `environment.TF_VAR_incus_config_dir` and
token `EnvironmentFile` (they never reached the job); keep the worker
`StateDirectory` that owns the cert dir. Realign the `apps-api`/
`apps-panel` tests to seed the client cert into the worker's own
`/var/lib/incus-deploy` (owned `windmill:windmill`) and write only the
fingerprint to `tf-env`.
Assisted-by: Claude:claude-opus-4-8
The incus HOST runs its own `spire-agent`/`openbao-agent-default` (its
`spiffe`/`oidc` requests resolve cross-group against the hosting-provider
group), so it must resolve the same `<network.name>.abundos.eu` peer FQDNs its
containers do -- the trust-bundle fetch dials `http://spire.abundos.eu:8082`,
the openbao agent dials `https://openbao.abundos.eu`.
The container-side peer-DNS fix (`hosts-common.nix` `isIncusHosts` block) points
only the deployed nodes' resolver at the bridge `10.0.100.1`; the host, deployed
out-of-band over SSH (`hypervisor-ssh-incus`) with the group's upstream Procolix
resolvers, was left resolving internal `abundos.eu` names against a resolver
that cannot serve them before the group's `bind` node exists -> NXDOMAIN ->
the host's trust-bundle fetch hangs -> spire/openbao/tf-backend/migrate chain
never starts.
Prepend the bridge (`mkBefore`) to the host's `networking.nameservers` so it is
consulted first: the bridge dnsmasq is authoritative for `abundos.eu` (peer
FQDNs from leases) and forwards the rest upstream. Co-located with the bridge
definition, so it is machine-agnostic -- any host running this incus app gets
its own bridge as a resolver, with no per-machine address in the physical-host
config.
Assisted-by: Claude:claude-opus-4-8
The bare-metal incus hypervisor cannot get a SPIRE identity when its
hardware TPM is a 1.2 (or absent): the `tpm` NodeAttestor
(`spire-tpm-plugin` -> go-attestation `OpenTPM(TPMVersion20)`) requires a
2.0 device and has no device-path knob, so its `probeSystemTPMs()` skips
the real 1.2 `tpm0` (it has a `caps` file) and grabs a leaked container
vTPM -> `permission denied` -> the host never attests -> its
`openbao-agent-default` never renders the vars token -> the gap#38
`incus-deploy-token-publish` producer stalls -> the bootstrap chain
(spire -> openbao -> tf-backend -> migrate) never completes. HP-facing:
any hypervisor with a 1.2 / no TPM hits it.
Give the incus-app host its own persistent software TPM 2.0
(`swtpm --vtpm-proxy --tpm2`), mask `/sys/class/tpm` down to a single
synthetic 2.0 `tpm0` bound to the swtpm devnode (mirroring the container
`mask-leaked-host-tpms`), and register the host swtpm's EK-pubkey hash on
the hosting-provider spire node over `incus exec` (the transport the host
already owns as the incus daemon). The host then attests with a STABLE EK
across reboots. All co-located in the machine-agnostic incus application
(same shape as the peer-DNS `mkBefore` and `bridgeCnames`), so ssh /
plain-tf / operator / container deploys are untouched and the producer
self-disables when no spire node is threaded.
- `core/applications/hosting-provider/incus.nix`: `host-swtpm` (persistent
EK via guarded `swtpm_setup --not-overwrite`), `host-mask-tpms` (tmpfs
synthetic `tpm0` + bind-mount + `tss` perms), `host-register-ek`
(non-blocking local EK read -> spire drop-dir), `tpm_vtpm_proxy` module,
and the `spireInstance`/`spireHostName` options.
- `core/effects/tf/incus/host/register-ek.sh`: `ek_source=local` branch so
the host reads `get_tpm_pubhash` locally; delivery + drop-dir contract
unchanged (single source of truth).
- `core/setups/hosts-common.nix`: thread the hosting-provider spire node's
`network.name` (instance) + the host's own `network.name` (identity) into
the incus app; defaulted empty for spire-less deploys.
Assisted-by: Claude:claude-opus-4-8
The `host-swtpm` / `host-mask-tpms` / `boot.kernelModules` additions were
unconditional, so ANY node importing the incus app -- including an incus host
that only provisions container vTPMs -- would start a second swtpm and
`mount -t tmpfs` over `/sys/class/tpm`, hiding the sysfs the incus daemon uses
to hand vTPMs to containers. `integrations-spire-incus` regressed: the
container `forkstart` failed on the masked host.
Gate all four host-side attrs on the same `hostSpireAttest` condition already
used for `host-register-ek` (a non-empty threaded `spireInstance` +
`spireHostName`). Only the hosting-provider hypervisor, which is configured to
SPIRE-attest AS itself, mints its own swtpm identity and masks its TPM sysfs; a
plain incus host keeps its native `/sys/class/tpm` intact.
Assisted-by: Claude:claude-opus-4-8
The `host-mask-tpms` service tmpfs-mounted `/sys/class/tpm` on the
bare-metal incus host to hide the hardware TPM 1.2 chip from the SPIRE
`tpm` NodeAttestor. But a tmpfs child-mount under `/sys` on the HOST
breaks LXC's recursive sysfs setup for every incus container
(`safe_mount: Operation not permitted` on the container's `/sys`), so
the node fails to start any of its containers.
The mask is also unnecessary: go-attestation's `probeSystemTPMs` already
skips any `/sys/class/tpm` entry that has a `caps` file (the TPM-1.2
signal) and picks the lexically-first remaining 2.0 entry. The hardware
1.2 chip is `tpm0` (has `caps`, skipped); the host swtpm is `tpm1` (no
`caps`, `/dev/tpm1` `root:tss 0660`, readable by the `tss`-group
spire-agent); container vTPMs are `tpm2`+ and never precede the swtpm.
Verified live on the hypervisor: with the mask dropped the spire
container starts and the host attests, getting an X509-SVID for
`spiffe://fediversity/host/<network.name>` off the swtpm EK, and
`host-register-ek` delivers that EK hash to the spire drop-dir.
Keep `host-swtpm` (the swtpm 2.0 itself), the `tpm_vtpm_proxy` kmod, and
`host-register-ek`; only the masking is removed.
Assisted-by: Claude:claude-opus-4-8
Dropping `host-mask-tpms` also dropped the `chgrp tss` it did on the
swtpm devnode as a side effect. The `swtpm --vtpm-proxy` kernel-created
`/dev/tpmN` defaults to `root:root`, but the spire-agent runs as the
`spire-agent` user in the `tss` group and opens the raw device, so
without the group it fails to attest (`open /dev/tpm1: permission
denied`).
Add a minimal `host-swtpm-perms` oneshot that discovers the swtpm's own
devnode from its `New TPM device: /dev/tpmN` startup log and sets
`root:tss 0660` -- no `/sys` mount, so containers are unaffected. It is
ordered strictly between the swtpm and spire-agent and runs long before
incus auto-starts containers, so no container vTPM is touched, and the
plugin's `caps`-skip still lands on the swtpm (`tpm1`) directly.
Verified live: with the swtpm devnode group-set, the host attests off
its swtpm EK and gets the `host/<network.name>` X509-SVID while all
container vTPMs are present at `tpm2`+.
Assisted-by: Claude:claude-opus-4-8
The bare-metal `hypervisor` group incus host runs the same
`vars-openbao` -> OpenBao login rails as the containers, but is deployed
over SSH (`hostsType == "ssh-hosts"`), so it fell outside the
`tf-incus-hosts` gate that emits the `openbao-pki-ca-fetch` unit. Its
`vars-openbao-login` and `openbao-agent-default` dial the central
hosting-provider openbao cross-group over `https://` with
`caCertFile = /var/lib/openbao-init/pki-ca.pem`, but nothing provisioned
that CA -> both failed `Error loading CA File: ... pki-ca.pem: no such
file`, so no vars token was rendered and the gap#38
`incus-deploy-token-publish` producer stalled.
Emit the runtime PKI trust side (which fetches the CA over the central
openbao's plain-HTTP `:8083` face) whenever the incus host runs the
login rails. Keyed on `app == "incus"`, the only hypervisor-group node
with them, so container and operator deploys are untouched.
Assisted-by: Claude:claude-opus-4-8
Since `ca0caa39e` the `openbao-pki` provider mints leaf keys
`0640 root:root` (private keys must never be world-readable, and
PostgreSQL refuses a key with any group/world access). But every node
that terminates a PKI-issued vhost on `nginx` -- openbao (its `:443` API
face and the plain-HTTP `:8083` CA-fetch face), windmill, monitoring,
netbox -- runs nginx as the unprivileged `nginx` user, which can no
longer read its own key (`cannot load certificate key ... Permission
denied`). nginx crash-looped cluster-wide, taking down every openbao
`:443` consumer and blocking a fresh node's plain-HTTP CA self-fetch on
`:8083`. nginx had tolerated the old world-readable `0644`, so this went
unnoticed until the key was tightened (only the postgres consumer got a
`keyGroup` wired in that commit).
Request `nginx` group read on every domain the ssl `applicationSide`
terminates on nginx, so the one place that builds the nginx vhosts also
grants the key access they need -- covering all four nodes uniformly.
Applied to all terminated domains unconditionally (not filtered by
resolved provider: reading `providerRequests` there is circular, and
acme/self-signed providers ignore `keyGroup`), `mkDefault` so a consumer
that reads the key as a different user (postgres) still overrides.
Assisted-by: Claude:claude-opus-4-8
`openbao-pki-ca-fetch` compares the freshly-served CA against the current
one with `cmp` before installing, but `cmp` ships in `diffutils`, not
`coreutils`, which was not on the unit's PATH. The first-boot fetch (no
CA yet) takes the `else` branch and never calls `cmp`, so this stayed
hidden; every subsequent re-run (the rotation-refresh path) failed
`cmp: command not found` and the oneshot never reached `active`.
Assisted-by: Claude:claude-opus-4-8
The openbao module's `consumerHosts` default carries `incus =
["incus-deploy"]` (the gap #38 deploy-token producer), so a cert-auth
role is created for the bare-metal incus host. But the SPIFFE label map
`spiffeHostNames` was built only from the hosting-provider group's own
machines, and the `incus` node lives in the split-out `hypervisor`
group -- so its label fell back to the app key and the role bound
`spiffe://fediversity/host/incus`. The host actually presents SVID
`host/192.168.51.101` and logs in `name=192.168.51.101` (its
`network.name`), so `vars-openbao-login` 400s `invalid certificate` and
the deploy-token producer never authenticates.
Merge the hypervisor group's machines into `spiffeHostNames` so the
`incus` role binds `host/<network.name>`, mirroring the cross-group
`spireInstanceName` / `centralOpenbaoHost` resolution already threaded
here.
Assisted-by: Claude:claude-opus-4-8
The SPIRE-token login (`spire-login.nix`) sent `bao login -method=cert
name=<networking.hostName>`, but the openbao node keys each cert-auth role on
the node's `network.name` (SVID `host/<network.name>`, role
`auth/cert/certs/<network.name>`). For every node whose label is a valid
hostname these coincide, so login matched. The bare-metal incus hypervisor is
keyed on its IP `192.168.51.101`, which is not a valid hostname, so
`networking.hostName` stays at the NixOS default `nixos` -- its login sent
`name=nixos`, matched no role, and 400'd, stranding the vars token and gap#38.
Add a `services.vars-openbao.loginName` option (default `networking.hostName`)
and set it from `network.name` for every node in the setup, so login, role, and
SVID agree regardless of whether the label is hostname-shaped.
Assisted-by: Claude:claude-opus-4-8
The maddy relay reads its own TLS key directly as the `maddy` user via
`services.maddy.tls.loader = "file"`, but the `openbao-pki` provider
delivers a runtime key defaulting to `root:nginx 0640`, which `maddy`
cannot read -- so `maddy.service` dies at startup with `key.pem:
permission denied` and never binds submission. Thread the daemon's own
`maddy:maddy 0600` through the mail domain's SSL request (mirroring the
postgres node's `postgres:postgres` block), so the issuing unit
`install`s the key readable by the consuming daemon. Gated on
`servesPki`, since a self-signed provider bakes a store-path key maddy
already reads.
Assisted-by: Claude:claude-opus-4-8
Every `oidc` consumer (openbao/incus/netbox/windmill) fetches the OIDC
discovery document at `https://auth.<domain>/.well-known/openid-configuration`
and TLS-verifies the authelia leaf CN. The incus bridge dnsmasq serves only
each node's INSTANCE FQDN (`<network.name>.<domain>`) from its DHCP lease, and
the authelia node's `network.name` is `authelia`, not `auth` -- so a consumer
dialing `auth.<domain>` NXDOMAINs during the fresh-provider bootstrap, before
the group's own bind/NetBox records exist. Live symptom: `openbao-oidc-auth`
fails with `Could not resolve host: auth.abundos.eu`.
The existing `hostingProviderBridgeCnames` set covers the
`<app>.<domain> -> <app>-node` cases (openbao/valkey/postgresql), where the
served subdomain equals the node name. `auth` is the one served subdomain that
differs from its node name, so add it as an explicit
`auth.<domain> -> authelia.<domain>` alias, gated on the group actually running
an authelia node. Both belt-and-suspenders filters still apply.
Assisted-by: Claude:claude-opus-4-8
The `ssl` resource's `applicationSide` blindly emits an nginx vhost for every
domain in the node's SSL request set (`genAttrs allDomains`). On the postgres
node that set includes `postgresql.<domain>` -- the cert postgres terminates on
`:5432` itself, whose key the node deliberately restricts to
`0600 postgres:postgres` (postgres refuses a group/world-readable key). nginx
runs as `nginx`, so it cannot load that key: the vhost fails
`cannot load certificate key ... Permission denied` and takes the WHOLE nginx
down -- including the node's legitimate `pgweb` admin-UI vhost. The same latent
trap exists for the maddy relay's `smtp.<domain>` key (`0600 maddy:maddy`).
Exclude from the nginx vhost set any domain whose resolved SSL request carries a
non-nginx `keyOwner` -- the signal that a dedicated daemon terminates that
domain's TLS itself and nginx has no business fronting it. Read across every
provider's `providerRequests`; this is the safe READ side (the same read
`acmeRequests` already does for `enableACME`), not a feedback into
`contracts.ssl.want`, so no circularity. Domains with no `keyOwner` (or
`keyOwner == "nginx"`) stay served by nginx exactly as before.
Assisted-by: Claude:claude-opus-4-8
The `vars-openbao-login` oneshot exchanges a SPIRE SVID for an OpenBao token
and its `.timer` re-runs it on `OnUnitActiveSec=30min` (under the 1h token
TTL). `OnUnitActiveSec` is measured from the last active-ENTER transition, but
with `RemainAfterExit=true` the oneshot enters `active (exited)` once and never
re-enters, so systemd computes no next elapse (`Trigger: n/a`) and the timer
fires exactly once. The minted token then expires at its 1h TTL and is never
refreshed, leaving every `openbao-agent` consumer `403 permission denied` on
`lookup-self` (observed live on the incus host: last login succeeded ~8h before
the agent 403s, timer stuck at `Trigger: n/a`).
Drop `RemainAfterExit` on both login variants (`generateFiles/spire-login.nix`
and the PKI-consumer `ssl/openbao-pki-login.nix`) so the oneshot goes `dead`
after success; each timer trigger is then a fresh start that re-anchors the
30min interval. This mirrors the existing `spire-agent.nix` trust-bundle
oneshot, which drops `RemainAfterExit` for the same class of reason. The login
script is idempotent (atomically replaces the token file), so re-running is
cheap.
Assisted-by: Claude:claude-opus-4-8
A `dynamic` postgres consumer (windmill/panel) connects as a random
per-lease role, and any object that role creates -- e.g. sqlx's
`_sqlx_migrations` bookkeeping table, which windmill creates as the
CONNECTING user before its `SET ROLE windmill_admin` -- stays owned by
it. Once the lease is revoked the object is stranded: the engine's
`REASSIGN OWNED`/`DROP OWNED` `revocation_statements` run on the
maintenance-database connection, and those statements only reach
objects in the CURRENT database, never the app's. Every later lease
then hits `permission denied for table`. This surfaced live the first
time a lease actually rotated under a running windmill (the
`reloadUnits` re-render restart made that reachable): `windmill-server`
dies with `Error migrating db: ... permission denied for table
_sqlx_migrations`.
Give each dynamic role a persistent NOLOGIN `ownerRole`: an app can
name one (windmill reuses `windmill_admin`, where its migrations
already land), otherwise `<username>_owner` is derived from the
operator-prefixed username (panel). The engine's `creation_statements`
grant each lease membership in it and default the lease's sessions to
it (`ALTER ROLE ... SET role`), so everything the app creates is owned
by the persistent role from the start. The provider node's
`postgresql-dynamic-schema-grants` oneshot creates the role (guarded,
ordered after `postgresql-app-bootstrap-sql` so windmill's own
attribute-carrying `windmill_admin` creation wins) and re-owns any
public-schema objects stranded on other roles -- healing pre-fix
databases on the next deploy.
`contracts-postgresql-rotation` now covers the regression: a leased
session runs as the owner role, a table created under one lease is
read and written by the next lease after revocation, and the sweep
re-owns a table stranded on another role.
Assisted-by: Claude:claude-opus-4-8
A sequence linked to a table column (`serial`/identity) cannot take a
direct `ALTER SEQUENCE ... OWNER` -- postgres aborts with `cannot change
owner of sequence ... linked to table`, which under `ON_ERROR_STOP` took
the whole `postgresql-dynamic-schema-grants` oneshot down on the first
database with such a sequence (live: the panel's
`django_migrations_id_seq`, leaving the windmill database never swept).
A linked sequence follows its table's `ALTER TABLE ... OWNER`
automatically, so the sweep now skips sequences that have an `a`/`i`
`pg_depend` link and lets the table alter carry them.
The rotation check's stranded table now carries a `serial` column and
asserts the linked sequence ends up owned by the persistent role.
Assisted-by: Claude:claude-opus-4-8
The `services.vars-openbao` module is imported via
`generateFiles/consumer.nix`, so it only exists on nodes with vars
consumers. Defining `loginName` unconditionally for every machine with a
`network` attr aborted the eval of nodes without any consumer (spire,
bind, otel-collector) with `The option 'services.vars-openbao' does not
exist`, failing their deploys. Guard the definition on the option being
declared; nodes without the module have no OpenBao login to name.
Assisted-by: Claude:claude-opus-4-8
Closes #954.
Makes the incus hosting-provider deployment machine-agnostic and reproducible end to end, and converges it live on the bare-metal incus host
192.168.51.101: a hosting-provider admin can follow the documented bootstrap and have all 16 hosting-provider containers come up from the base LXC image onto real per-node configs, with the secret mesh (SPIRE, OpenBao, PKI), internal DNS, mail, and the admin UIs all working.Fresh-provider bootstrap
tf/incus/image) and a stagedBOOTSTRAP=1local-then-migrate tofu backend, with durable local state, a pinnedTF_HTTP_ADDRESSoverride, and an auto-opened remote-deployer migrate tunnel (BOOTSTRAP_TUNNEL_SSH).incus file push(not exec-stdin streaming), with realistic import timeouts.<served domain> -> <instance FQDN>CNAMEs derived from the group's ssl requests), before bind/NetBox exist; the host resolver joins the same chain.eth0via networkd DHCP,systemd-machine-id-commitsuppressed so the first switch is not killed mid-flight.Bare-metal host integration
host-swtpm/host-register-ek), so hosts with a TPM 1.2 or no TPM still get an SVID and the gap #38 token producer runs.:443(the daemon's:8443self-signed cert stays stable for deploy-runner TOFU pinning), with OIDC against authelia (public client, JWT-profile access tokens, portless callback)./1.0/metricsthrough the same nginx front (dedicated metrics-type client certificate minted and enrolled by the host) into the otel-collector -> OpenObserve pipeline.Cross-node deploy credentials (gap #38)
incus-deployvars KV, and each deploy runner self-enrolls its incus client certificate from runner-local state.Secret-mesh convergence
Live bring-up surfaced and fixed a long tail of structural gaps, including: cert-auth roles keyed on
network.name(not app attr keys) and registered for every consumer and generation node including the openbao node itself; SPIRE trust-bundle export on a timer plus self-healing of agents, cached PKI CAs, and cert-auth trusted CAs across CA and EK rotations; OpenBao agents/logins couplingVAULT_CACERTto thehttps://scheme; dynamic postgres DSN leases owned by a persistent role, with schema grants and consumer restarts on lease re-render; vars uploads ordered after token login with KV-backed secrets no longer rotating every reboot; the terraform-backend state store decoupled from OpenBao (itskmsKeyrouted via the on-machine provider) so it never sits downstream of the secret system it repairs.Services and admin surfaces
smtp.<domain>; failed DSNs surface as OpenObserve alerts. (The upstream account seed is a temporary reuse ofgit@fediversity.eu, flagged in-tree as structural follow-up.)client_secret_post, gunicorn otel instrumentation), pgweb across application databases, redis-commander/garage-webui, and an eager per-group valkey namespace.fediversity-adminsgroup provisioned in lldap and enrolled for the seeded administrator; docs cover private access, cert regimes, first-login seeding, and bootstrap recovery.Verification
192.168.51.101: fullBOOTSTRAP=1ladder converged (base image -> per-node configs -> local->HTTP state migrate), secret mesh healthy, panel demo deploy green, admin UIs and metrics verified.treefmt,deployment-tf-incus-hosts, and the service checks are green on earlier batches;apps-tf-incusand a heavy sweep on the rebased tip are delegated to CI.Disclaimer: I used a coding agent in the creation of this patch.
incus file push, not exec-stdin stream 69a347dbcaAn incus deploy pushes each node's real config into an LXC container, which has no bootloader and no disko-managed disks. The reused roster config targets a physical/VM host: the shared `setups/nixos` base enables `systemd-boot` plus a `/boot` vfat mount (`boot.nix`) and disko partitioning (`disks.nix`). Inside the container `switch-to-configuration` therefore failed at `Failed to install bootloader` (`efiSysMountPoint '/boot' is not a mounted partition`) and the node never activated its real config -- it stayed on the base image. Fold `lxc-container.nix` into the pushed config and neutralize the host boot/disk options (`systemd-boot.enable = mkForce false`, `fileSystems = mkForce {}`) whenever `isIncusHosts`, exactly as the `deployment-tf-incus-hosts` test's `shared.nix` already does. The gate keeps the ssh / plain-tf deploy of the same node (to a real host) unchanged. The image build's `nodeConfiguration` reuses this same per-node config, so its previously-separate `lxc-container.nix` + `mkForce` block is now redundant and is removed to avoid a double-definition collision. Assisted-by: Claude:claude-opus-4-8The `isIncusHosts` neutralization (`ce594e2d2`) folds `lxc-container.nix` into each incus node so `switch-to-configuration` works in an LXC. That makes the node config container-shaped, where `boot.initrd.enable` defaults off and `config.system.build.initialRamdisk` is absent. `checks/common/deployerNode.nix` preloads every `system.extraSystems` machine's closure and collects `machine.system.build.vm.inputDerivation`. Its `or { }` guard only skips a missing attribute, but `vm` is always declared (`build-vm.nix` defaults it to `config.virtualisation.vmVariant.system.build.vm`); evaluating it pulls in `qemu-vm.nix`, whose `virtualisation.directBoot.initrd` default references the absent `initialRamdisk`, tripping `attribute 'initialRamdisk' missing`. This broke `apps-tf-incus` eval (whose deployer uses `extraSystems`); `deployment-tf-incus-hosts` was unaffected because it preloads via `extraDependencies` drvPaths without forcing full config eval. Force `system.build.vm` to a plain attrset on the neutralized nodes so the guard skips it without ever evaluating `vmVariant`. A container is never booted as a QEMU VM, so the VM product is meaningless there. Assisted-by: Claude:claude-opus-4-8nix-store -ron incus deploy a57518a6f0On incus-deployed nodes SPIRE TPM node-attestation failed with `stat /dev/tpm1: no such file or directory`, wedging the whole fresh-provider bootstrap: the agent never got an SVID, so `vars-openbao-login` hung, no vars were uploaded to OpenBao, the openbao/tf-backend PKI chain never came up, `terraform-backend` never bound `:8080`, and the local->HTTP state migrate never completed. Two compounding causes, both fixed here: - `/sys` is not namespaced in an incus container, so `/sys/class/tpm/` leaks the HOST enumeration (the host's hardware TPM plus every container's swtpm). The `spire-tpm-plugin` (go-attestation `attest.OpenTPM(&OpenConfig{})`, which exposes no device-path config) picks the first non-1.2 entry and opens `/dev/<name>` -- landing on `/dev/tpm1`, which has no devnode in the container (only the injected `/dev/tpm0` exists). The picked index is also non-deterministic across container-start order. Overlay `/sys/class/tpm` with a tmpfs holding a single synthetic `tpm0` entry (no `caps` => TPM 2.0), so the scan resolves to the container's real `/dev/tpm0`. Gated on the incus methods (`isIncusHosts`); ssh / real-host deploys have a proper per-host `/sys`. - A SPIRE-node redeploy resets its ephemeral EK allow-list drop-dir (`/var/lib/spire-server/host-ek-hashes`), but `register_ek`'s triggers keyed only on `instance_name` did not change, so registration never re-fired and the allow-list stayed empty -- no node could attest. Thread the SPIRE node's (and each node's own) `nixos_id` rebuild id into the `register_ek` triggers so a redeploy re-pushes every node's EK. Adds a `nixos_id` output to the incus host module. Assisted-by: Claude:claude-opus-4-8TF_HTTP_ADDRESSover the derived bridge IPTwo flake-show / spire-incus regressions surfaced once the incus app module started declaring a `fediversity.incus` option: - The deploy-time SPIRE EK-registration `spireNode` was added to the shared args under the broad `isIncusHosts` gate, which also matches the `tf-incus-operator-hosts` method. That method's effect declares no `spireNode` option, so `flake show` rejected `deployments.tf-incus-operator-hosts.spireNode` as unknown for any group with a `spire` node (the hosting-provider group). Gate on `hostsType == "tf-incus-hosts"` -- the only method whose effect declares it. - `openbao.nix` re-points its own `fediversity.openbao-pki.{address,tokenFile}` only when that option is declared, but guarded on the whole `fediversity` namespace (`options ? fediversity`). A node that imports the incus app module (the `integrations-spire-incus` test's `incushost`) now has `fediversity.incus` declared, tripping the guard and defining `fediversity.openbao-pki` where it is undeclared. Narrow the guard to the specific option (`options.fediversity or {} ? openbao-pki`), matching its comment. Both verified via `nix flake show ./core` (exit 0) and `nix eval ./core#checks.x86_64-linux.integrations-spire-incus.drvPath` (builds the test drv) on fediversity-ci; parent-state reproduced both failures. Assisted-by: Claude:claude-opus-4-8Production OpenBao runs `tls_disable=true` on loopback with nginx terminating TLS at `:443`. The SPIRE client cert for `bao login -method=cert` is consumed by nginx and never reaches OpenBao's cert-auth backend, which extracts the client cert from OpenBao's OWN TLS handshake (`400 tls connection required`). So OpenBao itself must terminate mTLS. OpenBao validates all listeners at boot and refuses to start on a missing cert file, but the self-issued PKI leaf only exists AFTER OpenBao serves loopback (to mount its own PKI) and lives on `/run` tmpfs. Seed the mTLS listener with a store-path self-signed cert (always present -> no boot deadlock), then `openbao-listener-leaf.service` swaps in the real leaf and SIGHUPs OpenBao once `openbao-pki-${domain}.service` issues it. Consumers only trust the real leaf's CA and retry their login, so no insecure window and no `VAULT_SKIP_VERIFY`. Move the loopback admin API off `:8200` -> `:8250` (gate-only) so the mTLS listener owns `0.0.0.0:8200`. Point the cert-login/secret-read consumer addresses (`vars-openbao.address` HP + operator, `centralAddress`) at `:8200`; keep the token-authed PKI-issuance address on nginx `:443`. All gated on `hasOpenbaoPki` (production internalPkiNode) so standalone tests that `mkForce` their own `:8200` listener are untouched. Assisted-by: Claude:claude-opus-4-8-placeholdersuffix from nodenetwork.namelabels:8200BOOTSTRAP_TUNNEL_SSH)The incus deploy runner's `remote_exec` wraps every `incus exec` in a `timeout ${REMOTE_EXEC_TIMEOUT:-60}` whose purpose is to sever a wedged *D-Bus* call so the recovery loop can take over. Two bulk-disk steps -- the bridge-cache substitution (`nix-store -r`) and the delta-NAR import (`nix-store --import`) -- inherited that 60s default. Under a concurrent multi-node bootstrap (7+ nodes importing at once) the host's shared store I/O contends and a single import runs well past 60s. `timeout` then reaps it with rc=124, which propagates as a `local-exec` provisioner error and aborts the whole apply mid-deploy. Observed live on `192.168.51.101`: lldap's closure push reached `100%`, the following import timed out at 60s, and the apply failed before activation even started -- never reaching the state migrate. Set `REMOTE_EXEC_TIMEOUT=900` on just those two bulk calls (inline env prefix, so the D-Bus-recovery loop below keeps its tight 60s/30s/8s bounds). The substitution stays `|| true` (non-fatal), but the headroom lets it actually complete so the delta pushed over the slow WAN leg stays small. Assisted-by: Claude:claude-opus-4-8The `.self-fetched` marker distinguished a bundle WE fetched from an operator pre-seed, and refused to refresh a markerless bundle ("never touch it"). But an openbao/agent upgrading IN PLACE has a pre-fix bundle with no marker, so it is misclassified as an operator pre-seed and frozen on whatever CA it booted with -- stranding cert-auth on a stale SPIRE CA after the daily rotation (every SVID `auth/cert/login` then 400s), which is the very drift this was meant to fix. The SPIRE server is authoritative for its own CA, so the bundle it serves is always the trust root to pin; there is no case where a stale local copy (pre-seed or otherwise) should win. Drop the exemption and the marker entirely: a present bundle is best-effort refreshed from the live server, an absent one cold-starts. A pre-seed is just a warm start of the present case -- the first refresh brings it up to the current CA. Applied to both the openbao cert-auth fetch and the spire-agent trust bundle fetch (same pattern, same latent upgrade-strand). Assisted-by: Claude:claude-opus-4-8The `lldap-provision-users` service was gated wholesale on a non-empty operator recovery `email`, and derived the machine `authelia-bind` account's synthetic mailbox from that same `email`. On a bare `core {}` hosting-provider deploy (no operator config feeding `default-configuration.email`), `email` is empty, so the service did not exist at all -- the `authelia-bind` account was never created. The OpenBao LDAP engine's `authelia-bind` static role pins that exact entry (`dn=uid=authelia-bind,ou=people,<baseDN>`), so its rotation write failed `LDAP Result Code 32 "No Such Object"`, leaving the `authelia-bind` credential unrotated, authelia's bind broken, and the SSO frontends down. Reproducible for any hosting provider deploying via `core {}` without a recovery email set. Decouple the two accounts: the machine `authelia-bind` account is now ALWAYS seeded (its mailbox domain derived from `baseDN`, not the operator email), and only the human `administrator` stays gated on a non-empty `email` (it is unusable without one -- its password comes from lldap's email reset flow). Assisted-by: Claude:claude-opus-4-8The maddy relay owns the single `smtp` vars instance and uploads one submission password per consumer (`vars/secret/smtp/<consumer>Password`). Each consumer node (authelia, monitoring, netbox, windmill) renders its own submission password back through its openbao agent to authenticate to the relay's `submission://` listener, but no policy granted them read on that path -- so the render hit `403 permission denied`, the authelia config template failed, and authelia crash-looped. Add a shared read grant on `secret/{data,metadata}/vars/secret/smtp/*` in the seed loop, mirroring the `authelia-oidc`/`postgresql-admin` pattern (the relay keeps its own per-instance write grant). Gated on a relay actually being deployed (`smtpProvided`), so a relay-less deployment grants nothing. Reproducible for any hosting provider whose group runs the maddy relay. Assisted-by: Claude:claude-opus-4-8smtp.<domain>so its FQDN resolves 7eb38b107dsmtp.<domain>The `spire-agent-trust-bundle` oneshot was `RemainAfterExit = true` and only `wantedBy` `spire-agent.service`. `switch-to-configuration` leaves an unchanged `RemainAfterExit` oneshot `active (exited)` and never re-runs it, so its (already-correct) `cmp`-based refresh logic never fired on a redeploy. The SPIRE server CA rotates on its own 24h lifetime, so a provider redeploying after a rotation kept a STALE seed bundle on disk; the agent then could not verify the server leaf (`x509: certificate signed by unknown authority`) and failed to attest -- stranding every downstream cert-auth consumer (no SVID -> no `vars-openbao` token -> `panel`/services down). Drop `RemainAfterExit` and add `multi-user.target` to `wantedBy` so the fetch re-runs on every converge; the script already refreshes only on a `cmp` mismatch, so a current bundle is a no-op ("re-fetch only when stale"). Update the `spire-incus` check to wait on the installed bundle file rather than the oneshot being `active` (a non-`RemainAfterExit` oneshot is `inactive` once done). Assisted-by: Claude:claude-opus-4-8The panel node self-generates its Django `SECRET_KEY` under the `panel` vars instance (`vars/secret/panel/secret_key`) and uploads it via `upload-vars-openbao`, exactly like `netbox` and `terraform-backend`. But its `consumerHosts` default entry was an empty list, so the generated `host-panel` policy carried no write grant on `secret/{data,metadata}/vars/secret/panel/*` and the upload failed with `403 permission denied`. This was latently masked: before the cert-auth role naming fix, the panel login itself 400'd, so `upload-vars-openbao` never reached the 403. With the login healed, the missing grant surfaced. Make `panel` own its `panel` instance, symmetric with the other self-generating nodes. Assisted-by: Claude:claude-opus-4-88514e4bee95341db0adcWindmill's bundled DB migrations (e.g. `20221211192539: grant usage`) GRANT to two group roles, `windmill_user` and `windmill_admin`, that must already exist, and windmill `SET ROLE`s to `windmill_admin` at runtime. The upstream nixpkgs windmill module creates these roles only in its `windmill-initdb` oneshot, gated on `database.createLocally = true`. Our windmill consumes a REMOTE database on `dynamic` OpenBao credentials, so `createLocally = false` and that oneshot never runs -- the first migration dies with `role "windmill_admin" does not exist` and `windmill-server` never starts. Rather than baking a windmill-specific initdb into the `postgresql` resource, extend the resource/contract with two GENERIC, app-declared capabilities so any service with schema-level prerequisites the credential machinery does not cover can express them itself (mirroring how an app supplies its own `nixos-configuration`): - `bootstrapSQL`: idempotent SQL the provider node runs ONCE as the postgres superuser against the app's database, after `postgresql-setup`. Threaded to the database node keyed by database and executed by a new `postgresql-app-bootstrap-sql` oneshot. This is where the superuser socket is -- an app config merges onto its OWN node, which has no superuser access to the remote postgres. - `dynamicRoleGrants`: roles each freshly-minted `dynamic` lease role is granted membership in, folded into the OpenBao engine's dynamic `creation_statements` as `GRANT <role> TO "{{name}}"`. A dynamic consumer connects as a random per-lease role, so a group role its service `SET ROLE`s to must be granted to every lease role, not once to a fixed user. Windmill supplies both from its own app module: `bootstrapSQL` recreates the `windmill_user`/`windmill_admin` roles exactly as the upstream initdb does (minus the per-user grant, which `dynamicRoleGrants` handles per-lease), and `dynamicRoleGrants = [ "windmill_admin" ]` lets each lease role assume it. Machine-agnostic: any hosting provider deploying windmill on dynamic credentials would otherwise hit the same dead `windmill-server`. Assisted-by: Claude:claude-opus-4-8vars-openbao.loginNamewhere the option exists 795380d575