No reviewers
fediversity/developers
Labels
No labels
0 points
0.5 points
1 point
13 points
2 points
21 points
3 points
34 points
5 points
55 points
8 points
ambition
application-offering
ambition
configure-applications
ambition
front-end
ambition/install-applications
ambition
security
ambition
switch-host
ambition
update-applications
ambition
user-management
api service
blocked
component: fediversity panel
component: nixops4
documentation
estimation high: >3d
estimation low: <2h
estimation mid: <8h
infinite points
productisation
project-management
question
role: application developer
role: application operator
role: hosting provider
role: maintainer
security
technical debt
testing
type unclear
type: bug
type: deliverable
type: key result
type: objective
type: task
type: user story
user experience
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
fediversity/fediversity!1172
Loading…
Reference in a new issue
No description provided.
Delete branch "kiara/fediversity:bootstrap-order"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Graceful fresh-provider bootstrap ladder (#336, #882)
A fresh hosting provider comes up along the chain
incus -> terraform-backend -> openbao -> valkey -> spire -> netbox, but every rung wants something a later rung provides: the TF graph needs a state backend it itself deploys, secret-bearing components want their secrets from openbao-via-SPIRE before openbao is initialized or SPIRE has a trust root, and NetBox self-discovery reads state from a backend that does not exist yet. This climbs the ladder rung by rung so an external team can deploy a provider without owning any Fediversity-team secret beyond the irreducible floor.The design is written up in
notes/bootstrap-full-ladder.md; the remaining load-bearing seeds and their provenance are audited innotes/bootstrap-seeds.md; operator steps are innotes/bootstrap.md.State chicken-and-egg (#336, #515)
The hosting-provider group deploys the nimbolus
terraform-backend(an HTTP state store) as one of its own nodes, yet TF deployments need an HTTP backend to store state. Add the standard OpenTofu local-then-migrate bootstrap, gated behind an explicitBOOTSTRAP=1env flag so a transient backend outage in steady state can never silently fork state to a local file.The mechanism is a thin generic wrapper (
src/effects/tf/common/bootstrap-apply.sh) in method-agnostic sharedtfRun, sotf-incus-hosts(the real path),tf-hosts, andtf-incus-operator-hostsinherit it. On the normal path (noBOOTSTRAP=1) it is a byte-identical pass-through to the baked-HTTP apply; SSH deploys have no HTTP backend so it is force-disabled. UnderBOOTSTRAP=1:backend "local"override in a writable copy of the read-only bakedtf-env, self-state pointed at the local file.init -migrate-state -force-copycopies state up to the running backend, then a.migratedmarker gates re-migration.BOOTSTRAP): short-circuit to the plain HTTP apply; idempotent.A single
BOOTSTRAP=1invocation converges both applies on one clock, blocking on backend reachability between them via the reusablesrc/effects/tf/common/wait-for-http.sh(loud-fail-past-timeout, no silent fallback to local state). Deploy-timetofu initstays offline: thewithPluginstofuon PATH resolves providers from its baked filesystem mirror.Secret-provenance chicken-and-egg (#493, #882)
The state store does not recur, but every secret-bearing component hits the same first-boot problem: it wants its secret from openbao-via-SPIRE, and on a fresh provider openbao is empty, SPIRE has no trust root, and the agent has no SVID. Each rung is made graph-native:
:8443that cannot be brought up by the graph depending on it. Formalize thehosting-provider-ssh-incustarget (SSH effect, no HTTP backend sobootstrapis force-disabled) as documented step 0, plus the redeploy/reset-failedrecovery for the incus node'sspire-agent/openbao-agentunits that fail before the ladder is up.openbao-init.servicedecides whether to init from openbao's own-status, never re-initing an already-initialized node (which would orphan the unseal key); thealready-init but sealedpath re-unseals from the persisted key and a lost seed fails loud. Adds aservices.openbao-init.unsealMethod(manual-key|tpm-sealed) seam, and implementstpm-sealed: the unseal key is sealed to the node's vTPM (the same vTPM SPIRE attests against, #633) instead of persisted to disk, retiring the on-disk unseal-key seed on TPM nodes.manual-keystays the default.register-ek.shin the tf-incus-hosts post-apply hook). Close the last out-of-band step: the SPIRE server serves the (public) CA trust bundle over plain HTTP:8082, and each agent'sspire-agent-trust-bundleoneshot self-fetches its trust root on first boot with a bounded retry, ordered beforespire-agent.service. A pre-seeded bundle still short-circuits the fetch.redismodule rendersrequirepassfrom a file with an unguardedcat; an absent/empty bootstrap-password file (failed render on a fresh provider) would silently start valkey passwordless and publicly reachable. Promote each instance's edge ongenerate-vars/openbao-agent-defaultfromWantstoRequiresand add atest -sExecStartPregate so the instance refuses to start without a non-empty password.tofu applywithout theBOOTSTRAP=1wrapper trapped at plan time on aterraform_remote_state.selfread of the not-yet-existing backend. Adata "external"reachability probe (self-state-reachable.sh, same "any HTTP response = up" definition) gates the self-read'scountto 0 when the backend is unreachable, so a naive fresh-provider apply degrades to DHCP-only instead of hanging.BOOTSTRAP=1stays the blessed entrypoint.openbao-spire-authself-fetches the same public SPIRE CA bundle over HTTP instead of an out-of-band copy. Thesecretsresource gains a per-secretprovider(agenix|vars) so any post-ladder secret can be sourced from SPIRE-attested openbao over thecontracts.fileSecretsinterface, letting an external team avoid owning Fediversity agenix recipients.notes/bootstrap-seeds.mdaudits the four irreducible seeds that remain (operator SSH access, openbao unseal key on non-TPM nodes, agenix recipients, deployment SSH key).Tests
checks/deployment/tf-hosts-bootstrapmodels the true state chicken-and-egg (theterraform-backendnode is a deployed target, not the deployer) and proves the before / local-state / migrate / idempotent progression.deployment-tf-self-state-reachable-- a cheap native check running the real probe against all six backend scenarios (empty / unknown / local-present / local-absent / http-down / http-up).integrations-spire-openbaocovers openbao idempotency + crash-recovery re-unseal and runs the openbao node undertpm-sealedwith a swtpm, assertinginit.jsoncarries no plaintext key material.integrations-spire-incusasserts a fresh agent self-fetches a trust bundle matching the server with no pre-seeded bundle.test-valkey-serviceasserts the instance refuses to start with a truncated bootstrap-password file, then starts cleanly once restored.Disclaimer: I used a coding agent in the creation of this patch.
@ -231,0 +295,4 @@# The self-state seed (`seed-self-state.sh`, #833), invoked before each# bootstrap apply with the cycle-appropriate `TF_VAR_self_state_*` so# the netbox self-read survives. Empty when the effect sets no preRun.BOOTSTRAP_SEED = if preRun == "" then "" else preRun;in other words...
= if preRun;?@ -0,0 +126,4 @@with subtest("Normalizing reinvocation: migrate local state up to HTTP"):# Backend node is live now, so a second BOOTSTRAP=1 run migrates the local# state up to the HTTP backend and applies against it.if one is to run the bootstrap command twice, then why not just have the bootstrap env var(s) already do it twice so it already reaches the stable state?
@ -0,0 +1,82 @@# Graceful bootstrap of a fresh hosting-provider TF backend (#336, #882)so far this bootstrap script just resolves the bootstrap challenge for the initially lacking TF backend? because #336 lists at least 6 known components that we want to use in our deployments yet initially have yet to deploy themselves - of which this is only the 2nd one (out of 6).
6. to be fair, #1150 may have tackled it for netbox, the last item.
*-alldeployments deployed by incus, so it should probably be deployed separately upfront by ssh - and maybe incus just either shouldn't be exposed in*-all, or even should go in a separate group from the rest there (such that like*-alllogic would not need to be maimed just because hypervisors deploy differently from the services they get used to deploy).so overall i'm still wondering for such a bootstrap test/script about at least openbao, valkey and spire.
0a44cabeedc84a1d62965662a1eeb208bba42251WIP: Graceful bootstrap of the hosting-provider TF backend (#336, #882)to Graceful bootstrap of the hosting-provider TF backend (#336, #882)Graceful bootstrap of the hosting-provider TF backend (#336, #882)to Graceful fresh-provider bootstrap ladder (#336, #882)@ -76,0 +104,4 @@"vars"];default = "agenix";};handle routing thru the contracts system - no provider should be special-based in the resource, which should essentially act as a provider-agnostic wrapper over the
fileSecretscontract@ -109,2 +109,4 @@expected = "/run/agenix/forgejo-runner_forgejo-runner-token";};# Every secret defaults to the `agenix` provider (#882/#493 Stage 6 keeps theheck no they shouldn't, agenix goes against our goal to eliminate all hard-coding to ensure anyone could fork the code and run it without needing any special access - if anything agenix should therefore be more opt-in than any of our other providers still
@ -209,0 +249,4 @@port = bundleFetchPort;}];locations."= /${bundleFileName}".root = bundleWebRoot;are there security implications to publicly serving this?
@ -0,0 +6,4 @@operator must supply it -- there is no earlier link to derive it from).The bootstrap ladder brings a fresh provider up along`incus -> terraform-backend -> openbao -> valkey -> spire -> netbox`, climbing ais there stuff that might depend on garage or postgresql as well? would those also require special consideration here? and what of our monitoring nodes?
@ -0,0 +26,4 @@| Seed | Where | Why irreducible || -------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || Operator SSH access to the bare host | the deploy runner's SSH agent/key | The very first link: incus is stood up over SSH (`hosting-provider-ssh-incus`) before any of the ladder exists. Nothing precedes it. || OpenBao unseal key (`manual-key` nodes only) | `/var/lib/openbao-init/init.json` (0400), `/var/lib/openbao-init/root-token` | `openbao-init` generates these on first boot and they gate all of openbao's contents. Under `unsealMethod = "manual-key"` (the default) the plaintext unseal key persists to disk as the single load-bearing local seed. It is **reducible on a TPM node** via `unsealMethod = "tpm-sealed"` (see the reducible table) -- irreducible only where no usable TPM exists. The root token stays on disk either way, but it is rendered durably into openbao's own KV downstream rather than escrowed out of band. |should it, or is this an unnecessary security risk?
@ -0,0 +28,4 @@| Operator SSH access to the bare host | the deploy runner's SSH agent/key | The very first link: incus is stood up over SSH (`hosting-provider-ssh-incus`) before any of the ladder exists. Nothing precedes it. || OpenBao unseal key (`manual-key` nodes only) | `/var/lib/openbao-init/init.json` (0400), `/var/lib/openbao-init/root-token` | `openbao-init` generates these on first boot and they gate all of openbao's contents. Under `unsealMethod = "manual-key"` (the default) the plaintext unseal key persists to disk as the single load-bearing local seed. It is **reducible on a TPM node** via `unsealMethod = "tpm-sealed"` (see the reducible table) -- irreducible only where no usable TPM exists. The root token stays on disk either way, but it is rendered durably into openbao's own KV downstream rather than escrowed out of band. || agenix recipient public keys | `keys/` (`contributors`, `cd`, per-system) | The operator controls decryption authority for any `.age`-sourced secret. A fresh external team either owns these recipients or routes those secrets to the `vars` provider instead (see the agenix->vars migration below). || Deployment SSH key (`api-ssh-key`) | `src/resources/secrets/api-ssh-key.age` | The key the API/deployer uses to reach target nodes. It authorizes the deploy itself, so it cannot be sourced from a service the deploy brings up. Stays agenix. |could this not be generated as well?
@ -0,0 +25,4 @@| Seed | Where | Why irreducible || -------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || Operator SSH access to the bare host | the deploy runner's SSH agent/key | The very first link: incus is stood up over SSH (`hosting-provider-ssh-incus`) before any of the ladder exists. Nothing precedes it. |right, we should like implement a live USB effect to reduce this
@ -0,0 +27,4 @@| -------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || Operator SSH access to the bare host | the deploy runner's SSH agent/key | The very first link: incus is stood up over SSH (`hosting-provider-ssh-incus`) before any of the ladder exists. Nothing precedes it. || OpenBao unseal key (`manual-key` nodes only) | `/var/lib/openbao-init/init.json` (0400), `/var/lib/openbao-init/root-token` | `openbao-init` generates these on first boot and they gate all of openbao's contents. Under `unsealMethod = "manual-key"` (the default) the plaintext unseal key persists to disk as the single load-bearing local seed. It is **reducible on a TPM node** via `unsealMethod = "tpm-sealed"` (see the reducible table) -- irreducible only where no usable TPM exists. The root token stays on disk either way, but it is rendered durably into openbao's own KV downstream rather than escrowed out of band. || agenix recipient public keys | `keys/` (`contributors`, `cd`, per-system) | The operator controls decryption authority for any `.age`-sourced secret. A fresh external team either owns these recipients or routes those secrets to the `vars` provider instead (see the agenix->vars migration below). |yes, this needs to move to env vars / TF state sooner or later
@ -0,0 +38,4 @@| Former seed | Mechanism | Stage || ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- || `terraform-backend` HTTP state backend | local-then-migrate under `BOOTSTRAP=1` (`bootstrap-apply.sh`) | 1 || `KMS_KEY_FILE` (terraform-backend) | `generateFiles` (`openssl rand -base64 32`) bridged into openbao KV via `vars`; `generate-vars` seeds it on first boot, `openbao-agent-default` renders it back durably -- the "migrate the boot-generated key into openbao, then read it back" self-heal | 6 (structurally already in place) |what does this
KMS_KEY_FILEworkaround even buy us? if we need to fall back to local back-end until this one is available anyway, then might we not as well have things migrate to this only once it's good already, rather than introducing such additional steps that buy us (iiuc) little?@ -0,0 +39,4 @@| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- || `terraform-backend` HTTP state backend | local-then-migrate under `BOOTSTRAP=1` (`bootstrap-apply.sh`) | 1 || `KMS_KEY_FILE` (terraform-backend) | `generateFiles` (`openssl rand -base64 32`) bridged into openbao KV via `vars`; `generate-vars` seeds it on first boot, `openbao-agent-default` renders it back durably -- the "migrate the boot-generated key into openbao, then read it back" self-heal | 6 (structurally already in place) || OpenBao unseal key (on a TPM node) | `unsealMethod = "tpm-sealed"`: `openbao-init` seals the `bao operator init` unseal key to the node's (v)TPM (`tpm2_create`/`load`/`unseal` under a persistent primary), writing only inert `tpm-sealed.pub`/`.priv` and stripping every plaintext key field from `init.json`; auto-unseals from the TPM on every boot. Reuses the same vTPM SPIRE attests against (`spiffe-tpm`/#633). `manual-key` stays the default for TPM-less nodes. | 2 |are these temp workarounds, or just initial setup? cuz, does spire even have dependencies? if not, might we not as well only start bothering with openbao (and whatever needs it) once that one is good already? or, differently put, if these are workarounds, what would they buy us concretely?
@ -0,0 +63,4 @@The `.age` secrets under `src/resources/secrets/` are all **dev-group** andpre-date the ladder -- the forgejo host _is_ the CI runner that later bringsopenbao/SPIRE online, so its own bootstrap secrets cannot be sourced from aservice it has not deployed yet. They stay agenix by necessity, not by omission.wait this bootstrap process so far was just about group hosting-provider still right, even if deployed from there? cuz the dev group i imagine might need its own bootstrap process still, even if it might be able to reuse stuff from our current scope.
@ -0,0 +67,4 @@To let an **external** team deploy over SSH without owning Fediversity agenixrecipients, the `secrets` resource now exposes a per-secret `provider` choice(`src/resources/secrets/default.nix`):did it not already, thru the contracts mechanism?
@ -0,0 +94,4 @@fetches in the self-fetch oneshots, and the irreducible floor in `keys/` +`openbao-init`). The irreducible rows above are the real, minimal seed set --three unconditional (operator SSH access, agenix recipients, deployment SSH key)plus the openbao unseal key on TPM-less nodes; everything else the ladder derives. Keeping the catalog here -- next to the laddershouldn't we just assert TPM? cuz does like
KMS_SECRET_FILEeven offer security benefits over just {sw,v}tpm? if not, why bother special-casing things?@ -0,0 +52,4 @@`BOOTSTRAP=1`:```bashBOOTSTRAP=1 nix run .#hosting-provider-tf-incus-allas-is this would create incus-ception, no? cuz then wouldn't it like make more sense to split out incus to some new like
hypervisorgroup, even if this were to technically break the semantic boundary of 'who is this node meant for'?@ -0,0 +41,4 @@(5 starts / 10s), so after a few retries they settle into `failed` rather thanlooping forever. Once the rest of the ladder is deployed (step 1 brings upopenbao + spire), converge them with either a redeploy(`nix run .#hosting-provider-ssh-incus` again) or, on the incus host:redeploy to incus node to reload its services should be mentioned at its actual position in the process (in
bootstrap-full-ladder.md?) if it wasn't yet@ -0,0 +53,4 @@```bashBOOTSTRAP=1 nix run .#hosting-provider-tf-incus-all```this bootstrap command fails to explain how to e.g. connect to the incus (relevant env vars)
@ -0,0 +69,4 @@If the backend node never comes up within `BOOTSTRAP_MIGRATE_TIMEOUT`(default 1800s), the run **fails** rather than silently leaving state local --so the operator sees the real problem. The local apply is persisted, so simplysome mentions of 'operator' in this document seem to use the dictionary description, potentially bringing some confusion as we use this term in another sense in our project as well (to describe the person using the hosting provider's front-end to deploy applications for them to use), while these instances seem to instead describe the hosting provider
@ -0,0 +1,355 @@# Full fresh-provider bootstrap ladder (#882, #493, #515)while this speaks of a full bootstrap ladder, what of the internal-TLS bootstrap mentioned in #1119 for openbao and incus (if not our other internal nodes that have use for it)?
@ -0,0 +29,4 @@TF-state backends).2. **Secret-provenance chicken-and-egg** (#493) -- a workload needs a secret(valkey password, netbox token, KMS key) that is _supposed_ to come fromhuh, weren't we using netbox just to allocate public IP addresses? that's why that one was part of the bootstrapping problem (so i think we got more challenges there than two)
@ -0,0 +73,4 @@## Stage 1 -- terraform-backend (DONE)Local-then-migrate under `BOOTSTRAP=1`. See `notes/bootstrap.md`. The KMS key isuse an actual relative link
@ -0,0 +83,4 @@**Current:** `openbao-init.service` runs `bao operator init` on first boot,writes unseal key + root token to `/var/lib/openbao-init/`, unseals. Single-nodeSQLite. Works on a fresh single node with no gap.didn't our nodes include postgres and garage as well? shouldn't we use any of those over sqlite, at least once the bootstrap has settled?
@ -0,0 +15,4 @@deploy without owning Fediversity-team secrets.This note is the _design_. Each stage below is a separable PR; they are orderedby dependency, not by priority. Only the terraform-backend stage is done today.this should be updated to reflect the current status - we did implement more since.
if all are indeed done then i'm not sure we even still need to track progress in this document per item.
08bba42251d3f7162788@ -0,0 +57,4 @@inherit domain;applications = lib.mapAttrs (_: _: { enable = true; }) machines.application;};applications = [ ../../applications/hosting-provider ];this new group could just get its own dir there
@ -0,0 +90,4 @@| agenix recipient public keys | `keys/` (`contributors`, `cd`, per-system) | The admin controls decryption authority for any `.age`-sourced secret. Only the **human-team** recipients (`contributors`, `cd`) are truly irreducible -- they are the trust root, so "generating" them would defeat the point (nothing an outside deploy generates is an authority the admin controls). But the recipient set does not have to be a checked-in `keys/` seed: it should move into deploy-time **env-vars / TF-state** the deployer already holds (see "Intended reductions"), so a fresh external team supplies its own recipients at deploy time rather than editing the tree. The **per-system** host keys under `keys/` are moreover _generatable_ at provisioning (a fresh host key minted on the node, its public half registered in the same apply), so they are not an admin seed on the incus path at all. Failing either, a secret that carries no admin-controlled authority can be produced through `generateFiles` -> openbao instead (see the migration note below). || Deployment SSH key (`api-ssh-key`) | `src/resources/secrets/api-ssh-key.age` | The key the API/deployer uses to reach target nodes. It authorizes the deploy itself, so it cannot be sourced from a service the deploy brings up -- but it need not be a checked-in `.age` file: the deployer could **generate** the keypair at deploy time and push the public half onto the target nodes' `authorized_keys` in the same apply (nothing about the deploy identity requires a long-lived pre-shared key), reducing it from a stored seed to a per-deploy ephemeral. Tracked under "Intended reductions"; until then it stays agenix. |### Intended reductions (tracked, not yet done)these we may note as potential follow-ups in the PR
@ -0,0 +234,4 @@is issued by a `pki/issue` call, which needs the node's SPIRE SVID, which needsthe trust root -- so a PKI cert can only appear once the node is already up thespire rung. That is fine because both listeners degrade gracefully toself-signed + trust-on-first-use in the gap:both todo
@ -342,3 +343,4 @@zone = mkZone [machines.applicationmachines.ancilliary(import ./hypervisor.nix { inherit lib; }).machines.applicationlet's say the hypervisor group got a new node
proxmox(not actually deployed): would it then try to add that to the zone as well somehow?given that would not be the intent, might there be a better way to handle this - taken into account one could potentially deploy the
hosting-providergroup to any of potentially multiple such hypervisors?89021714966cddad8f5aa7083eecf9df913f332d