netbox: operator TF effect consumes the allocated v4+v6 IP (#833) #1150
No reviewers
fediversity/developers
Labels
No labels
ambition
application-offering
ambition
configure-applications
ambition
front-end
ambition/install-applications
ambition
security
ambition
switch-host
ambition
update-applications
ambition
user-management
blocked
component
api-service
component
fediversity-panel
component
nixops4
documentation
points
0
points
0.5
points
1
points
13
points
2
points
21
points
3
points
34
points
5
points
55
points
8
points
infinite
productisation
project-management
question
role
application-developer
role
application-operator
role
hosting-provider
role
maintainer
security
technical debt
testing
type
bug
type
deliverable
type
key-result
type
objective
type
task
type
unclear
type
user-story
user experience
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
fediversity/fediversity!1150
Loading…
Reference in a new issue
No description provided.
Delete branch "kiara/fediversity:operator-netbox"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes part of #833 (folding the unit-tested NetBox integration into the real deployment pipelines).
PR #993 landed the NetBox plumbing -- the
tf-netbox-*effects and the genericallocate/releasewiring inhosts-common.nix. The operator OpenTofu effect did not yet consume the allocated IP, though: the incus instance got its address from DHCP onincusbr0and the NetBox-issued IP just landed unused in the merged deploy-output JSON.What this does
Deployed operator nodes that expose public services now get an available IPv4 and IPv6 sourced from NetBox and registered there, applied via their NixOS
networkingconfig (the routed address sits on top of the bridge's DHCP link, which still carries the deploy transport). Re-deploys and other nodes never reallocate the same address. Internal-only nodes (windmill, monitoring, netbox, terraform-backend, valkey, postgresql, smtp) settle for internal/PKI TLS and are skipped.How the runtime IP reaches the NixOS config
writeConfigalready reads runtime data viabuiltins.getEnv "DEPLOYMENT"at deploy-script runtime, so the deploy shell's hash stays stable. The NetBox IP rides the same channel: the run wrappers allocate each public node's address and export aNETBOX_NODE_IPSJSON map (instance -> { ipv4, ipv6 });writeConfigreads it and merges each node's address into itsnetworkresource policy. Absent env keeps the staticgroups/operator.nixaddress.Changes
src/effects/tf/netbox/get-ip.nix: allocate a second,count-gated IPv6 off the same interface, selecting fromvar.ip_range_id_v6. v4-only deploys pass null and the v6 resource drops out.src/resources/network/default.nix: add aneedsPublicIpoption (defaulttrue) as the per-app override surface, and deriveprefixLengthfrom the allocated CIDR (was hard-coded24/64).src/setups/hosts-common.nix: selective allocation for public nodes only; wrap the aggregateallrun/destroy for the operator method; exportNETBOX_NODE_IPS.src/effects/default.nix: readNETBOX_IPV6_RANGE_*intoipRangeV6, and mergeNETBOX_NODE_IPSinto each node's network config inwriteConfig.checks/apps/tf-incus/: giveincusbr0an IPv6 subnet, seed a v6 NetBox range, and assert the deployed node'seth0carries the NetBox v4+v6 address, plus re-deploy idempotence, release, and the internal-node skip.Verification
Local: operator aggregate + per-app + internal-node wrappers eval to executables; the generated
configuration.nixparses and the merge is functionally correct (public node gets the CIDR, internal node keeps its static address);all-run-with-netboxis shellcheck/bash -nclean;apps-tf,apps-tf-incus,deployment-tf-incus-hostsdrvPaths eval;treefmtreports 0 changed.The heavy
apps-tf-incusrun is left for CI.Disclaimer: I used a coding agent in the creation of this patch.
5149661aa9cc2c164bbb@ -154,0 +177,4 @@"netbox""terraform-backend""valkey""postgresql"this feels flaky - if the node were renamed to
postgres, could that make it accidentally end up with a public IP?@ -109,0 +123,4 @@"valkey""postgresql""smtp"];this feels flaky - if the node were renamed to
postgres, could that make it accidentally end up with a public IP?@ -102,0 +118,4 @@mergeNode =node: env:letips = nodeIps.''${node} or null;what's with this type of escaping in this file? we weren't in some string here, right?
cc2c164bbb978e59e7ea@ -411,0 +499,4 @@apps = publicNodes;outFile = "\"$tmp/node-ips.json\"";}}NETBOX_NODE_IPS="$(cat "$tmp/node-ips.json")"what is the distinction between
NETBOX_NODE_IPSvsNETBOX_IPV6_RANGE_START/NETBOX_IPV6_RANGE_END?@ -368,0 +419,4 @@node_ips='{}'${lib.concatMapStringsSep "\n" (app: ''alloc="$("${lib.getExe netboxCfg.perApp.${app}.allocate}")"if netbox is used stateless (?), and we then deploy a single machine by incus here, would the current method still not just wholesale override netbox's prior allocations (rather than just on behalf of the allocations of the nodes in question)?
@ -411,0 +510,4 @@donejq -s 'reduce .[] as $x ({}; . * $x)' "$tmp/alloc.json" "$tmp/main.json"'';there may be internal (e.g. for hosting-provider iff not provided, so
netbox == null) vs external (e.g. for operator group) netbox, tho even if internal we should be wary to handle it as such given a deploy of one machine should not have notion of the other machines managed by a netbox. so, we should not do netbox seeding (outside init of the actual netbox) but just interface with it 'externally' through TF rather than thru some shared (between seeding and allocation)mkNetboxAllocFragmentas if we always owned the netbox.fwiw, i could maybe still use suggestions on how to easily have this work for both incus deploymen effects (tf-incus+operator, i removed netbox args for just
tf[-ssh]in the last commit on main).as such, i feel perhaps wary to find
hosts-common.nixcontain (both new and existing) shell scripts writing things to files - or with seeing shell scripts like these in there (rather than incorporated into the TF graphs) in general.(note that this would apply both to this allocate script, as well as to the below script about IP release.)
978e59e7ea6c89857a8d6c89857a8d325228c8a0Re-architect NetBox IP consumption per review comment 898: the deploy must not (re)create NetBox infrastructure, must interface through TF rather than shell scripts in `hosts-common.nix`, and must work for both incus effects (`tf-incus-hosts` and `tf-incus-operator-hosts`). Allocation now lives in the incus graph (`tf/incus/{hosts,operator}/netbox.tf`): each `needs_public_ip` node gets a `netbox_virtual_machine` + interface and a `netbox_available_ip_address` (v4, plus v6 when the pool has a v6 prefix), read from an owner-provisioned PREFIX the deploy only reads. The prefix is discovered by a selector (`netbox_prefix_selector`, default `tag:fediversity-operator`) matched via the generic `netbox_prefixes` filter -- so the deploy needs no specific knowledge of what subnets the NetBox has, and the hosting provider can renumber or grow the pool freely; a `cidr:` selector pins an exact subnet when wanted. The allocated address rides into each node's `networking` config via the existing `NETBOX_NODE_IPS` channel, passed through the `local-exec` environment in the one deploy apply. Release is automatic on `tofu destroy`. This drops the entire shell-wrapper machinery from `hosts-common.nix` (`mkNetboxAllocFragment`, the per-node/aggregate/per-verb run+destroy wrappers, the `netbox` model submodule, and the seed/perApp assembly) -- ~330 lines -- in favour of threading a small `netbox` option (cluster name + prefix selector) and a per-node `needsPublicIp` flag into the incus effect. The `netboxArgs` env reads collapse to a single `NETBOX_PREFIX_SELECTOR` (`<kind>:<value>`). An operator deploy no longer seeds ranges, and a deploy with no public node reads no NetBox at all (the data sources are count-gated). The `apps-tf-incus` test now seeds a tagged v4+v6 prefix once as init (it owns its NetBox), then the deploy allocates from it in-graph; assertions keep the eth0-carries-v4+v6, idempotent-re-deploy, distinct-address, internal-node-skip, and destroy-releases coverage, reading allocations back over the NetBox REST API (`netbox_vm_ips`) rather than driving a standalone `get-ip`. A new `tf-netbox-store-prefix` effect seeds the tagged prefix. The standalone `get-ip`/`store-ips` effects and `test-netbox-service` are unchanged (they still unit-test the plumbing). Assisted-by: Claude:claude-opus-4-8`netbox.tf` added an `e-breuninger/netbox` provider to the `tf-incus/{hosts,operator}` graphs, but their `package.nix` `withPlugins` lists still only carried `incus`/`null`/`external` (+`garage`/`vault` for operator). So the `tf-repo` provider mirror those graphs build lacked the netbox plugin, and `tofu init` fell back to the network registry -- `provider registry.opentofu.org/e-breuninger/netbox was not found in any of the search locations` -- which the network-free sandbox cannot reach. Add the `netbox` provider to both `withPlugins` lists (it is already the pinned `e-breuninger/terraform-provider-netbox` v5.3.0 from `common/providers.nix`, matching the `5.3.0` source address the `.tf` files declare). This was the single root cause of the `deployment-tf-incus-hosts`, `apps-tf-incus`, `integrations-spire-incus`, `apps-api`, and `apps-panel` failures. Assisted-by: Claude:claude-opus-4-8Round 1's provider-bundle fix let `tofu init` resolve `e-breuninger/netbox`, but plan then failed `Missing required attribute` on the implicit netbox provider in the `tf/incus/{hosts,operator}` graphs. The provider's `server_url` and `api_token` are required HCL attributes with no env-var fallback when no `provider "netbox"` block is present, so a DHCP-only deploy (`pkgs.hello`, spire-incus -- `netbox_enabled = false`, all NetBox resources count-gated to zero) had no way to satisfy them and errored before doing anything. Add an explicit `provider "netbox"` block to both graphs, sourcing `server_url` /`api_token` from new `netbox_server_url`/`netbox_api_token` variables that default to an unreachable placeholder, with `skip_version_check = true` so the provider does not contact that placeholder during configuration. A DHCP-only deploy now plans cleanly (the provider is configured but, with no resources referencing it, never used). A NetBox-enabled deploy overrides both variables with the real instance URL/token, exported as `TF_VAR_netbox_server_url`/ `TF_VAR_netbox_api_token` alongside the existing `NETBOX_SERVER_URL`/ `NETBOX_API_TOKEN` the standalone seed effects read. Assisted-by: Claude:claude-opus-4-8netbox_enabled(#833)length(...), not!= {}(#833)`local.netbox_nodes` is map-typed (a `for`-comprehension result), so OpenTofu's `!=` against the object-typed literal `{}` is ALWAYS true -- the two operands have different types. The three count-gated NetBox data sources (`by_cidr`, `by_filter`, `cluster`) therefore fired on EVERY deploy regardless of the gate, including DHCP-only deploys that left `netbox_server_url` at its `http://netbox.invalid` placeholder. That is why `apps-api`/`apps-panel` (operator graph, `netbox_enabled = true`, no NetBox server) still hit `lookup netbox.invalid: connection refused` after the `netbox_active` gate landed: `netbox_active` correctly resolved false and `netbox_nodes` was correctly empty, but the downstream `!= {}` comparison fired the data sources anyway. Gate on `length(local.netbox_nodes) > 0` instead, which compares the element count. Verified on the runner via `tofu console` against both real graph bundles: `netbox_enabled=false` -> 0; `netbox_enabled=true` + placeholder URL -> 0 (the `apps-api`/`apps-panel` case, previously 1); `netbox_enabled=true` + real URL -> 1 (the `apps-tf-incus` case, unchanged). Both graphs `tofu validate` clean. Assisted-by: Claude:claude-opus-4-8store-tageffect (#833)familyattributeallaggregate; document the test path (#833)The default-on NetBox self-discovery reads this deploy's OWN published state (`data "terraform_remote_state" "self"`) to find its self-deployed netbox/openbao node addresses. On the FIRST apply that state key has never been written, and `terraform_remote_state` treats a missing state as a fatal error (`No stored state was found ...`) for BOTH the `http` and `local` backends -- there is no "tolerate missing" flag. A deploy reading its own state key is circular: the key this apply writes does not exist until after the first apply. This broke every fresh incus-hosts deploy: `deployment-tf-incus-hosts`, `integrations-spire-incus`, and (via `operator-tf-incus-all`) `apps-api` / `apps-panel`. Seed a valid empty state (outputs `{}`) at the self-state location before any state-touching `tofu` verb runs, so the self-read succeeds-as-empty and the graph falls through to DHCP-only on the first deploy (`try(..., {})` in `main.tf`), picking up the real addresses on redeploys once the apply has published them. The seed is idempotent -- it only writes when no state exists yet, so a populated state is never clobbered, and the following apply overwrites the seeded-empty state regardless. `seed-self-state.sh` is backend-agnostic to match the self-read: it POSTs the empty doc to the `http` state address (nimbolus backend) in steady state, or writes the file for a `local` backend during the fresh-provider bootstrap (#336/#882). It reads the same `TF_VAR_self_state_*` the self-read uses, so a deployer's bootstrap `local` override is honored too. The self-read is a `data` source evaluated by every plan/refresh, not just `apply`, so the seed fronts every state-touching verb (`run`/`apply`/`plan`/ `refresh`/`destroy`) via a new `preRun` hook + `seedState` flag in `tfRun`. The pure-inspection verbs (`output`/`show`/`validate`/`state-*`) do not refresh, so they skip it and stay side-effect free. Effects that set no `preRun` (ssh, tf-hosts) are unchanged. Assisted-by: Claude:claude-opus-4-8The first cut of the empty-self-state seed POSTed the state document straight to the `terraform-backend` state address. The nimbolus backend guards every write with a lock: a bare POST is rejected `400` ("attempting to write state with wrong lock") unless its `?ID=<lock-id>` matches a lock the caller currently holds. That `400` is what made the first apply fail with `curl: (22) ... 400` -- the seed aborted before `tofu` ever ran, so the default-on self-read never got its empty state. Mirror what OpenTofu itself does instead: `LOCK`, `POST` with that lock ID, then `UNLOCK` (via an `EXIT` trap so a failed POST cannot strand the state locked). `basic` auth hashes the password into the state ID, so the seed keeps using the `username`/`password` from `self_state_config` -- the same creds the self-read uses -- for the seeded key to line up with where the deploy reads. Also stop passing `--fail` to the existence GET: a never-written state returns `404`, which `--fail` would turn into curl exit 22; capture the code without it. Verified end-to-end against a live `terraform-backend`: seeds an empty backend (GET then reads 200), is idempotent on a populated one (no clobber of a real `ipv4` output), leaves the backend writable for a subsequent real apply, and no-ops when `self_state_backend` is empty. Assisted-by: Claude:claude-opus-4-8WIP: netbox: operator TF effect consumes the allocated v4+v6 IP (#833)to netbox: operator TF effect consumes the allocated v4+v6 IP (#833)