fediversity/fediversity
Watch
18
2
Fork
You've already forked fediversity
 10
Code Issues 93 Pull requests 17 Projects Activity Actions 1

reproducible project infrastructure #336

New issue
Closed
opened 2025-05-20 17:37:50 +02:00 by kiara · 1 comment
kiara commented 2025-05-20 17:37:50 +02:00
Owner
Copy link

As a developer,
I want to be able to reproduce relevant project infrastructure,
so that I may test the project and independently work on the project.

component test wiring (#833) infra (#433) document (#86) TLS backups SSO
SSO: authelia ? ? ? ? ✅ ? -
panel ? ? ? 🚧 ✅ ? ✅
api ? ? ? ? ? ? ?
queue: windmill ? ? ? ? ✅ ? ?
opentelemetry storage: prometheus? ? ? ? ? ? ? ?
S3-like: garage ? ? #111? ? ✅ ? ✅
RDBMS: postgresql ? ? #115 ? ✅ ? ✅
k/v store: valkey ? ? ? ? ✅ ? ✅
DNS: bind? integrations-octodns ? #200 ? ? ? ?
hypervisor (#724): incus apps-tf-incus tf-incus ? ? ✅ ? ✅
IPAM: netbox ? ? ? ? ? ? ?
TF backend: nimbolus terraform-backend ? ? #515 ? ? ? ?
back-up service: restic? ? ? ? ? ? ? ?
secrets store: openbao ? ? #493 ? ✅ ? ✅
SPIFFE: spire ? ? #633 (PR: #970) ? ? ? ?
SMTP: maddy ? ? #117 ? ? ? ?
k8s (#598) ? ? #619 ? ? ? ?

deployment order of core components should probably be (see #882):

  1. incus (after this the rest may be deployed to LXC containers)
  2. nimbolus (if needed earlier maybe transfer from a temp back-end like TF cloud, otherwise seed by non-TF deployment method, e.g. ssh/nixos-infect/usb / manual install instructions, maybe parameterized using cli params or examples/hosting-env-config.nix - could be updated by TF-based deployment methods after if needed) - standard TF bootstrap pattern: local-then-migrate (state migration):
    1. First apply runs with a local backend (state file on the deployer's disk, no remote needed). This brings up the full all graph including the terraform-backend and netbox nodes.
    2. Once those nodes are live, run tofu init -migrate-state pointing at the now-running HTTP backend. OpenTofu copies the local state up.
    3. All subsequent applies use the remote HTTP backend normally.
  3. openbao
  4. valkey
  5. spire
  6. netbox

once this bootstrap has resolved, it can restore (for this set of machines) missing/seeded functionality like:

  • internal TLS certs (#1119)
  • secrets (#882) (e.g. cyclical dependency between openbao x valkey)
  • IP management
  • deployment state
  • inter-node auth to openbao

these so far both derive from openbao.

dependencies

  1. #200 reproduce DNS VM
  2. #325 Reproducible proxmox installation
  3. #433 deployment provisions host infra
  4. #356 reproduce CI runner
  5. #309 port infra to terraform
  6. #370 fediversity apps reused in infra

multi-tenancy

Can one shared instance serve multiple isolated tenants?
Service Shared? Isolation primitive Enforced?
OpenBao Yes Namespaces (open-source core, GA 2.3.x, 2025-06-25) Strong
PostgreSQL Yes DB-per-tenant + per-tenant role Strong
BIND Yes (by design) Per-tenant TSIG key + update-policy zonesub, ACLs, catalog zones Strong (per zone)
maddy Yes (many domains) Email-keyed virtual users; auth_map/storage_map Soft only
lldap No — (single fixed base DN) —
Authelia No (end-users) ACLs over one shared user pool Soft only
restic No (one repo) per-repo master key Strong only as 1-repo-per-tenant

Clean split:

  • Graduate into shared HP group with enforced slices: OpenBao (namespace), PostgreSQL (DB+role), BIND (zone+TSIG).
  • Must be one-per-tenant via the operator group: lldap (upstream FAQ: no multi-tenancy), Authelia (no tenant concept — maintainers say run one per org), restic (single master key → any password reads the whole repo). maddy can be shared only under one trusted operator; for real isolation, per-tenant too.

Key caveats to know: OpenBao shares one seal/storage today (single unseal exposes all tenants); BIND isolates the zone not the process; restic storage-layer isolation needs rest-server --private-repos --append-only.

On the garage/valkey-style constructions

I documented the reusable minting pattern in the plan. One honest correction to the framing: on this branch, garage per-app minting runs as an ensure-garage systemd unit on the garage node (resource request → aggregate → apply emits garageSide), not an operator-TF graph. The "mint per-tenant in the operator-TF graph" model (so a shared HP node isn't redeployed on operator change) is the target from sibling branches (central-valkey, host-garage-central) — valkey doesn't even exist as a resource here yet. A real shared-HP rollout should adopt that operator-TF variant: same request/apply shape, but apply emits OpenTofu resources (garage TF provider / OpenBao namespace / BIND zone+TSIG) keyed per tenant.

This also points to introducing an explicit tenant entity in src/lib/data-model.nix so per-tenant-vs-shared routing becomes first-class (today it's implicit in operator == tenant).

Want me to go deeper on any one service's concrete implementation (e.g. sketch the OpenBao-namespace or BIND-zone resource against the data-model), or draft the tenant entity addition?

Multi-tenancy assessment: shared hosting-provider services

Context

The setup splits work into two deployment groups (src/setups/groups/):

  • operator (operator.nix) — single-tenant applications (mastodon/peertube/pixelfed on test04/05/06) plus per-deployment ancilliary provider nodes (garage, authelia, smtp/maddy, lldap). One operator deployment == effectively one tenant today.
  • hosting-provider (hosting-provider.nix) — shared infrastructure meant to serve many operators: panel, api, incus, terraform-backend, netbox, openbao, windmill, spire, otel-collector, monitoring, plus authelia+lldap as the admin-UI IdP.

Today there is no explicit tenant entity in src/lib/data-model.nix. Shared services namespace per-application (per-app garage buckets/keys, per-app lldap groups, per-app authelia OIDC clients, per-app smtp accounts, per-consumer secret key prefixes) — not per-tenant. The existing per-app minting precedent is garage (see the reference-pattern section below); operator == tenant today, so per-app == per-tenant.

We want to expand the hosting-provider offering. This document answers, per service, whether one shared instance can safely serve multiple isolated tenants, distinguishing enforced isolation from soft/config-only separation, and — for services that cannot — points the deployment at the existing one-instance-per-tenant pattern in the operator group.

This is an assessment / decision document, not a code change. No files are edited as part of this plan; the "next step" sub-sections describe where implementation would land if/when we act on a verdict.

Per-service verdicts

Legend: Shared-safe = one instance, many tenants, enforced isolation. Shared-soft = one instance can serve many tenants but separation is config-driven, not enforced. Per-tenant = must run one instance per tenant.

OpenBao — Shared-safe (strongest)

  • Verdict: one instance via namespaces. Unlike Vault (namespaces are Enterprise-only), OpenBao namespaces are in the open-source core, GA 2025-06-25 (2.3.x). Each namespace is a real partition: own secret engines, auth methods, ACL/EGP/RGP policies, identities, tokens, login paths.
  • Caveats: shared seal + storage backend today (per-namespace sealing/storage are unshipped RFCs) — a single unseal exposes all tenants; ~30 root-only system paths stay with the provider; watch parent->child identity inheritance.
  • Current state: hosting-provider application node (src/applications/hosting-provider/openbao/); consumed via vars-openbao contract + ssl openbao-pki provider. Today flat key prefixes, no namespaces.
  • Next step if adopted: introduce a per-tenant namespace and route vars/PKI paths under it; provider keeps the root namespace.

PostgreSQL — Shared-safe

  • Verdict: one cluster, database-per-tenant + per-tenant login role (revoke CONNECT/public from other tenants' DBs). Strongest of the three PG models (vs schema-per-tenant = logical only; vs RLS = weakest physical separation).
  • Caveats: PgBouncer pools are per-(db,user) so many DBs strain max_connections; migrations fan out per DB.
  • Current state: per-application local PG on each operator app node (mastodon/peertube/pixelfed each run their own services.postgresql); no shared cluster, no fediversity resource wrapping it.
  • Next step if a shared cluster is offered: new resource minting a per-tenant DB + role on a hosting-provider PG node; otherwise the status quo (local PG per app, isolated by separate nodes) already suffices.

BIND DNS — Shared-safe (by design)

  • Verdict: one named serves thousands of zones with enforced per-tenant write authority: per-tenant TSIG key + update-policy grant <tenantkey> zonesub (or subdomain) binds a key to exactly that tenant's zone/namespace slice; allow-transfer/allow-query ACLs stop cross-tenant reads (AXFR); catalog zones automate onboarding/offboarding across the primary/secondary fleet.
  • Caveats: isolation boundary is the zone, not the process (shared named, no per-tenant resource isolation); per-tenant distinct TSIG keys are mandatory (reuse collapses the boundary); lock down allow-transfer defaults.
  • Current state: no BIND deployment in-tree — DNS is external (only octodns integration tests reference DNS). Greenfield.
  • Next step if adopted: hosting-provider BIND node; per-tenant zone + TSIG key minted in the operator-TF graph (DNS records track operator changes), update-policy zonesub per tenant, optional catalog zone for fleet scale.

maddy (SMTP) — Shared-soft

  • Verdict: one instance handles many mail domains (full mailbox setup), but separation is administrative, not enforced: virtual users keyed on full email address give per-domain namespaces by default, yet all tenants share one credential DB / storage backend, no per-tenant admin delegation, no per-tenant quota/rate-limit primitive. A misconfigured auth_map/storage_map collapses it.
  • Current state: operator-group ancilliary (src/resources/smtp/maddy-provider.nix); one shared relay with per-app submission accounts (<app>@<domain> + KV-stored password).
  • Verdict for HP group: acceptable as shared-soft only under one trusted operator. For enforced per-tenant isolation (separate credential stores, quotas), prefer one maddy per tenant via the operator group, like lldap/authelia below. Keep storage keyed on the full address (the default).

lldap — Per-tenant (cannot share)

  • Verdict: explicitly single-organization. Official FAQ: "LLDAP does not natively support virtualhosts, sometimes known as multi-tenancy." Single fixed base DN, global ou=people/ou=groups, global permissions. Only soft workaround = fully-qualified-email usernames; no per-tenant ACLs.
  • Recommendation (per FAQ): one lldap instance per tenant (~15MiB RAM each). For richer multi-org needs upstream points to OpenLDAP/FreeIPA/Kanidm — out of scope.
  • Current state: already a per-deployment provider node in the operator group (src/resources/ldap/lldap-provider.nix), plus one in hosting-provider for the admin UIs. Per-app groups exist but that is app-scoping, not tenant-scoping.
  • Next step: mint one lldap per tenant via the operator group (operator-TF graph), mirroring the existing garage/valkey per-tenant minting. The hosting-provider lldap stays a single shared instance only for HP-internal admin UIs (one trusted org), not for tenant end-users.

Authelia — Per-tenant (cannot share for end-users)

  • Verdict: no tenant concept. One global user backend (single file DB or single LDAP base), one global config, one flat OIDC client list, one global ACL set, one session/cookie domain. Closest approximation is ACL rules (access_control.rules, regex/capture groups from #1339) over a shared user pool — logical filtering, not isolation. Sibling subdomains of one root can't be separate tenants (shared cookie domain); multi-root-domain support exists but with no inter-domain SSO. Maintainers decline to build true tenancy and recommend one instance per org.
  • Recommendation: one Authelia instance per tenant (~20MB RAM), each backed by that tenant's own lldap. (Avoid the multiple-instances-one-storage-backend setup, #5184.)
  • Current state: operator-group ancilliary + hosting-provider node (src/resources/sso/authelia-sso-provider.nix), LDAP-backed via authelia-ldap-backend.nix. Per-app OIDC clients = app-scoping, not tenant-scoping.
  • Next step: mint one Authelia + lldap pair per tenant via the operator group (operator-TF graph). Shared hosting-provider Authelia stays for HP-internal admin-UI SSO only (one trusted org).

restic (backup) — Per-tenant repository (cannot share one repo)

  • Verdict: a repo has one master key encrypting everything; multiple passwords each just wrap that same master key, so any valid password = full read/write to the whole repo, and password rotation does not revoke access. No cryptographic per-tenant isolation within a repo. Safe model = one repository per tenant (separate repo => separate master key).
  • Storage-layer isolation that does work: rest-server --private-repos (each authenticated user confined to /<user>/ subtree) + --append-only (block deletion of existing backups). Server-wide flags, not per-tenant toggles (#74).
  • Current state: per-application already — each operator app configures its own restic via the streaming-backup contract (nix/contracts/nixos/modules/services/backup/restic.nix), backing PG + redis dumps to S3 (garage). Already effectively one-repo-per-(app), and thus per-tenant since operator == tenant.
  • Next step: keep the per-tenant repo model. If consolidating storage behind a shared endpoint, front it with rest-server --private-repos --append-only and mint a per-tenant repo path + credential + restic password in the operator group (operator-TF graph).

Summary table

Service One shared instance for many tenants? Isolation primitive Enforced? Recommendation for HP group
OpenBao Yes Namespaces (OSS, GA 2.3.x) Strong Share + per-tenant namespace
PostgreSQL Yes DB-per-tenant + per-tenant role Strong Share cluster (or keep local-per-app)
BIND Yes (by design) Per-tenant TSIG key + update-policy zonesub, ACLs, catalog zones Strong (per zone) Share; mint zone+key per tenant via operator group
maddy Yes (many domains) Email-keyed virtual users; auth_map/storage_map Soft only Share only if single trusted op; else per-tenant
lldap No n/a (single base DN) — One instance per tenant (operator group)
Authelia No (end-users) ACLs over one shared user pool Soft only One instance per tenant (operator group)
restic No (one repo) per-repo master key; rest-server --private-repos Strong only as 1-repo-per-tenant One repo per tenant (already so)

Reference pattern: how garage mints per-app (per-tenant) slices

This is the reusable construction the per-tenant services above should mirror. The driving model lives in src/lib/data-model.nix (resource = request + policy, lines ~62-125): an app declares a request, the policy's generic process collects all apps' requests of that type into one list, and the policy's apply turns that aggregated list into node configs.

garage (src/resources/garage/default.nix) is the worked example:

  1. Request (app side). Each app declares the slice it needs. Mastodon (src/applications/operator/mastodon/default.nix): ensureBuckets.mastodon, ensureBuckets.mastodon-backups, ensureKeys.mastodon with per-bucket ensureAccess (read/write/owner), plus a nixos-configuration that wires the app's S3 endpoint.
  2. Aggregation. resource.policy.process (data-model.nix) filters every app's requests down to the garage ones and flattens them to a single list — collected in the setup at src/setups/hosts-common.nix (the ancilliaryRequests/process wiring, ~lines 302-328, 649).
  3. Apply. garage/default.nix's apply(requests) emits three things from the aggregated list:
    • applicationSide — per-app NixOS config (S3 endpoint, bucket domain, key file paths) wired into each consuming app node.
    • garageSide — a ensure-garage systemd oneshot on the garage node that, after garage.service, replays the aggregated requests: garage bucket create / key import / bucket allow for every app's buckets+keys in one pass.
    • sslDomains — cross-node TLS domains to register.
  4. Routing. The setup wires garageSide onto the garage node and applicationSide onto each app node (hosts-common.nix ~lines 869-873, 947-952).

Net shape: per-app request -> generic aggregation -> single apply -> (per-app config wired to app nodes) + (one provisioning unit on the shared node that mints all slices) + (cross-node artifacts). Isolation is by bucket name + per-app key with scoped grants.

Important caveat about where minting runs. In this branch garage provisioning runs on the garage node (the ensure-garage systemd unit), so it re-runs on every deploy of that node and picks up the current aggregated request set — it is not an operator-side TF graph here. The "mint per-tenant valkey/garage in the operator-TF graph" model (so a shared HP node is not redeployed when an operator changes) is the target architecture from sibling branches (central-valkey, host-garage-central), not yet realized on multi-tenant-infra. A real shared-HP rollout of the new per-tenant services should adopt that operator-TF variant: same request/apply shape, but apply emits OpenTofu resources (e.g. a garage TF provider, an OpenBao namespace resource, a BIND zone+TSIG resource) into the operator graph keyed per tenant, rather than a systemd unit on a shared node. The two-system distinction matters:

  • Resource system (garage today): provisioning materializes at NixOS eval/activation time on a node. Good when the provider node is (re)deployed alongside.
  • Contract providerRequests routing (nix/contracts/lib/contracts/module.nix, ~444-485): groups each contract request by the provider it's routed to — the hook a TF-apply-time per-tenant minting path would use to fan requests to a per-provider operator-TF graph.

valkey: no src/resources/valkey/ exists on this branch; the per-tenant valkey construction referenced in design notes lives on the central-valkey branch (shared HP valkey + redis contract/external provider, per-tenant valkey minted by an operator-tf-incus-hosts TF graph). When ported here it would follow the same request -> aggregate -> apply-emits-TF shape.

Key architectural implication

The clean split: OpenBao, PostgreSQL, BIND can graduate into the shared hosting-provider group with enforced per-tenant slices (namespace / DB+role / zone+TSIG). lldap, Authelia, restic, and maddy-for-isolation must be minted one-per-tenant via the operator group / operator-TF graph, exactly like the existing garage/valkey per-tenant pattern — because (a) they have no enforced in-instance tenant boundary, and (b) per-tenant resources must track operator changes without redeploying a shared HP node. A shared hosting-provider lldap/Authelia remains valid only for HP-internal admin-UI SSO, which is a single trusted org, not a multi-tenant surface.

All of the "next step" verdicts above reuse the same construction described in the reference-pattern section: a resource request per consumer, generic aggregation, and an apply that emits either a provisioning unit on a shared node (resource system, like garage today) or per-tenant OpenTofu resources in the operator graph (the central-valkey/host-garage-central target). The new enforced-isolation slices map onto apply outputs directly: an OpenBao namespace, a PostgreSQL DB+role, a BIND zone + TSIG key + update-policy — each keyed per tenant.

This also argues for introducing an explicit tenant notion in src/lib/data-model.nix so per-tenant minting (today implicit in operator == tenant) becomes first-class and the per-tenant-vs-shared routing above is expressible.

Verification

This is an assessment; "verification" = confirming the upstream claims behind each verdict before acting on one. Authoritative sources:

When a verdict is implemented, validate via the existing apps-tf nspawn test harness (per-tenant resource minting already exercised there for garage/valkey) plus a per-service isolation subtest (e.g. tenant-A TSIG key rejected against tenant-B zone; tenant-A OpenBao token denied in tenant-B namespace).

**As** a developer, **I want** to be able to reproduce relevant project infrastructure, **so that** I may test the project and independently work on the project. | component | test | wiring (#833) | infra (#433) | document (#86) | TLS | backups | SSO | |-|-|-|-|-|-|-|-| | SSO: authelia | ? | ? | ? | ? | ✅ | ? | - | | panel | ? | ? | ? | [🚧](https://panel.pages.git.fediversity.eu/) | ✅ | ? | ✅ | | api | ? | ? | ? | ? | ? | ? | ? | | queue: windmill | ? | ? | ? | ? | ✅ | ? | ? | | opentelemetry storage: prometheus? | ? | ? | ? | ? | ? | ? | ? | | S3-like: garage | ? | ? | #111? | ? | ✅ | ? | ✅ | | RDBMS: postgresql | ? | ? | #115 | ? | ✅ | ? | ✅ | | k/v store: valkey | ? | ? | ? | ? | ✅ | ? | ✅ | | DNS: bind? | `integrations-octodns` | ? | #200 | ? | ? | ? | ? | | hypervisor (#724): incus | `apps-tf-incus` | `tf-incus` | ? | ? | ✅ | ? | ✅ | | IPAM: netbox | ? | ? | ? | ? | ? | ? | ? | | TF backend: nimbolus terraform-backend | ? | ? | #515 | ? | ? | ? | ? | | back-up service: restic? | ? | ? | ? | ? | ? | ? | ? | | secrets store: openbao | ? | ? | #493 | ? | ✅ | ? | ✅ | | SPIFFE: spire | ? | ? | #633 (PR: #970) | ? | ? | ? | ? | | SMTP: maddy | ? | ? | #117 | ? | ? | ? | ? | | ~k8s (#598)~ | ? | ? | #619 | ? | ? | ? | ? | - [ ] #111 garage - [ ] #115 postgresql - [x] #200 DNS - [ ] #724 incus - [x] #515 tf-backend - [ ] #493 openbao - [x] #633 spiffe - [ ] #117 smtp deployment order of core components should probably be (see #882): 1. incus (after this the rest may be deployed to LXC containers) 1. nimbolus ~(if needed earlier maybe transfer from a temp back-end like TF cloud, otherwise seed by non-TF deployment method, e.g. ssh/nixos-infect/usb / manual install instructions, maybe parameterized using cli params or `examples/hosting-env-config.nix` - could be updated by TF-based deployment methods after if needed)~ - standard TF bootstrap pattern: local-then-migrate (state migration): 1. First apply runs with a local backend (state file on the deployer's disk, no remote needed). This brings up the full all graph including the terraform-backend and netbox nodes. 2. Once those nodes are live, run tofu init -migrate-state pointing at the now-running HTTP backend. OpenTofu copies the local state up. 3. All subsequent applies use the remote HTTP backend normally. 1. openbao 1. valkey 1. spire 1. netbox once this bootstrap has resolved, it can restore (for this set of machines) missing/seeded functionality like: - internal TLS certs (#1119) - secrets (#882) (e.g. cyclical dependency between openbao x valkey) - IP management - deployment state - inter-node auth to openbao these so far both derive from openbao. ### dependencies 1. #200 reproduce DNS VM 1. #325 Reproducible proxmox installation 1. #433 deployment provisions host infra 1. #356 reproduce CI runner 1. #309 port infra to terraform 1. #370 fediversity apps reused in infra ### multi-tenancy <details> <summary> Can one shared instance serve multiple isolated tenants? </summary> | Service | Shared? | Isolation primitive | Enforced? | | -------------- | ------------------- | ------------------------------------------------------------------ | -------------------------------- | | **OpenBao** | **Yes** | Namespaces (open-source core, GA 2.3.x, 2025-06-25) | Strong | | **PostgreSQL** | **Yes** | DB-per-tenant + per-tenant role | Strong | | **BIND** | **Yes** (by design) | Per-tenant TSIG key + `update-policy zonesub`, ACLs, catalog zones | Strong (per zone) | | **maddy** | Yes (many domains) | Email-keyed virtual users; `auth_map`/`storage_map` | **Soft only** | | **lldap** | **No** | — (single fixed base DN) | — | | **Authelia** | **No** (end-users) | ACLs over one shared user pool | **Soft only** | | **restic** | **No** (one repo) | per-repo master key | Strong only as 1-repo-per-tenant | **Clean split:** - **Graduate into shared HP group with enforced slices:** OpenBao (namespace), PostgreSQL (DB+role), BIND (zone+TSIG). - **Must be one-per-tenant via the operator group:** lldap (upstream FAQ: *no multi-tenancy*), Authelia (no tenant concept — maintainers say run one per org), restic (single master key → any password reads the whole repo). maddy can be shared only under one trusted operator; for real isolation, per-tenant too. **Key caveats to know:** OpenBao shares one seal/storage today (single unseal exposes all tenants); BIND isolates the *zone* not the process; restic storage-layer isolation needs `rest-server --private-repos --append-only`. ## On the garage/valkey-style constructions I documented the reusable minting pattern in the plan. One honest correction to the framing: on **this** branch, garage per-app minting runs as an **`ensure-garage` systemd unit on the garage node** (resource `request` → aggregate → `apply` emits `garageSide`), **not** an operator-TF graph. The "mint per-tenant in the operator-TF graph" model (so a shared HP node isn't redeployed on operator change) is the **target from sibling branches** (`central-valkey`, `host-garage-central`) — `valkey` doesn't even exist as a resource here yet. A real shared-HP rollout should adopt that operator-TF variant: same request/`apply` shape, but `apply` emits OpenTofu resources (garage TF provider / OpenBao namespace / BIND zone+TSIG) keyed per tenant. This also points to introducing an explicit **`tenant`** entity in `src/lib/data-model.nix` so per-tenant-vs-shared routing becomes first-class (today it's implicit in operator == tenant). Want me to go deeper on any one service's concrete implementation (e.g. sketch the OpenBao-namespace or BIND-zone resource against the data-model), or draft the `tenant` entity addition? # Multi-tenancy assessment: shared hosting-provider services ## Context The setup splits work into two deployment groups (`src/setups/groups/`): - **operator** (`operator.nix`) — single-tenant *applications* (mastodon/peertube/pixelfed on `test04/05/06`) plus per-deployment ancilliary provider nodes (`garage`, `authelia`, `smtp`/maddy, `lldap`). One operator deployment == effectively one tenant today. - **hosting-provider** (`hosting-provider.nix`) — shared *infrastructure* meant to serve many operators: `panel`, `api`, `incus`, `terraform-backend`, `netbox`, `openbao`, `windmill`, `spire`, `otel-collector`, `monitoring`, plus `authelia`+`lldap` as the admin-UI IdP. Today there is **no explicit `tenant` entity** in `src/lib/data-model.nix`. Shared services namespace **per-application** (per-app garage buckets/keys, per-app lldap groups, per-app authelia OIDC clients, per-app smtp accounts, per-consumer secret key prefixes) — not per-*tenant*. The existing per-app minting precedent is **garage** (see the reference-pattern section below); operator == tenant today, so per-app == per-tenant. We want to expand the hosting-provider offering. This document answers, per service, whether **one shared instance can safely serve multiple isolated tenants**, distinguishing **enforced** isolation from **soft/config-only** separation, and — for services that cannot — points the deployment at the existing one-instance-per-tenant pattern in the operator group. This is an **assessment / decision document**, not a code change. No files are edited as part of this plan; the "next step" sub-sections describe where implementation would land if/when we act on a verdict. ## Per-service verdicts Legend: **Shared-safe** = one instance, many tenants, enforced isolation. **Shared-soft** = one instance can serve many tenants but separation is config-driven, not enforced. **Per-tenant** = must run one instance per tenant. ### OpenBao — Shared-safe (strongest) - **Verdict:** one instance via **namespaces**. Unlike Vault (namespaces are Enterprise-only), OpenBao namespaces are in the **open-source core**, GA 2025-06-25 (2.3.x). Each namespace is a real partition: own secret engines, auth methods, ACL/EGP/RGP policies, identities, tokens, login paths. - **Caveats:** shared seal + storage backend today (per-namespace sealing/storage are unshipped RFCs) — a single unseal exposes all tenants; ~30 root-only system paths stay with the provider; watch parent->child identity inheritance. - **Current state:** hosting-provider application node (`src/applications/hosting-provider/openbao/`); consumed via `vars-openbao` contract + `ssl` `openbao-pki` provider. Today flat key prefixes, no namespaces. - **Next step if adopted:** introduce a per-tenant namespace and route `vars`/PKI paths under it; provider keeps the root namespace. ### PostgreSQL — Shared-safe - **Verdict:** one cluster, **database-per-tenant + per-tenant login role** (revoke `CONNECT`/`public` from other tenants' DBs). Strongest of the three PG models (vs schema-per-tenant = logical only; vs RLS = weakest physical separation). - **Caveats:** PgBouncer pools are per-(db,user) so many DBs strain `max_connections`; migrations fan out per DB. - **Current state:** **per-application local PG** on each operator app node (mastodon/peertube/pixelfed each run their own `services.postgresql`); no shared cluster, no fediversity resource wrapping it. - **Next step if a shared cluster is offered:** new resource minting a per-tenant DB + role on a hosting-provider PG node; otherwise the status quo (local PG per app, isolated by separate nodes) already suffices. ### BIND DNS — Shared-safe (by design) - **Verdict:** one `named` serves thousands of zones with **enforced per-tenant write authority**: per-tenant **TSIG key** + `update-policy grant <tenantkey> zonesub` (or `subdomain`) binds a key to exactly that tenant's zone/namespace slice; `allow-transfer`/`allow-query` ACLs stop cross-tenant reads (AXFR); **catalog zones** automate onboarding/offboarding across the primary/secondary fleet. - **Caveats:** isolation boundary is the *zone*, not the process (shared `named`, no per-tenant resource isolation); per-tenant distinct TSIG keys are mandatory (reuse collapses the boundary); lock down `allow-transfer` defaults. - **Current state:** **no BIND deployment in-tree** — DNS is external (only octodns integration tests reference DNS). Greenfield. - **Next step if adopted:** hosting-provider BIND node; per-tenant zone + TSIG key minted in the **operator-TF graph** (DNS records track operator changes), `update-policy zonesub` per tenant, optional catalog zone for fleet scale. ### maddy (SMTP) — Shared-soft - **Verdict:** one instance handles many mail domains (full mailbox setup), but separation is **administrative, not enforced**: virtual users keyed on full email address give per-domain namespaces by default, yet all tenants share one credential DB / storage backend, no per-tenant admin delegation, no per-tenant quota/rate-limit primitive. A misconfigured `auth_map`/`storage_map` collapses it. - **Current state:** operator-group ancilliary (`src/resources/smtp/maddy-provider.nix`); one shared relay with per-app submission accounts (`<app>@<domain>` + KV-stored password). - **Verdict for HP group:** acceptable as **shared-soft** only under one trusted operator. For enforced per-tenant isolation (separate credential stores, quotas), prefer **one maddy per tenant** via the operator group, like lldap/authelia below. Keep storage keyed on the full address (the default). ### lldap — Per-tenant (cannot share) - **Verdict:** **explicitly single-organization.** Official FAQ: *"LLDAP does not natively support virtualhosts, sometimes known as multi-tenancy."* Single fixed base DN, global `ou=people`/`ou=groups`, global permissions. Only soft workaround = fully-qualified-email usernames; no per-tenant ACLs. - **Recommendation (per FAQ):** **one lldap instance per tenant** (~15MiB RAM each). For richer multi-org needs upstream points to OpenLDAP/FreeIPA/Kanidm — out of scope. - **Current state:** already a per-deployment provider node in the operator group (`src/resources/ldap/lldap-provider.nix`), plus one in hosting-provider for the admin UIs. Per-app *groups* exist but that is app-scoping, not tenant-scoping. - **Next step:** mint one lldap per tenant via the **operator group** (operator-TF graph), mirroring the existing garage/valkey per-tenant minting. The hosting-provider lldap stays a single shared instance only for HP-internal admin UIs (one trusted org), not for tenant end-users. ### Authelia — Per-tenant (cannot share for end-users) - **Verdict:** **no tenant concept.** One global user backend (single file DB or single LDAP base), one global config, one flat OIDC client list, one global ACL set, one session/cookie domain. Closest approximation is ACL rules (`access_control.rules`, regex/capture groups from #1339) over a *shared user pool* — logical filtering, not isolation. Sibling subdomains of one root can't be separate tenants (shared cookie domain); multi-root-domain support exists but with no inter-domain SSO. Maintainers decline to build true tenancy and recommend one instance per org. - **Recommendation:** **one Authelia instance per tenant** (~20MB RAM), each backed by that tenant's own lldap. (Avoid the multiple-instances-one-storage-backend setup, #5184.) - **Current state:** operator-group ancilliary + hosting-provider node (`src/resources/sso/authelia-sso-provider.nix`), LDAP-backed via `authelia-ldap-backend.nix`. Per-app OIDC clients = app-scoping, not tenant-scoping. - **Next step:** mint one Authelia + lldap pair per tenant via the **operator group** (operator-TF graph). Shared hosting-provider Authelia stays for HP-internal admin-UI SSO only (one trusted org). ### restic (backup) — Per-tenant repository (cannot share one repo) - **Verdict:** a repo has **one master key** encrypting everything; multiple passwords each just wrap that same master key, so **any valid password = full read/write to the whole repo**, and password rotation does not revoke access. No cryptographic per-tenant isolation within a repo. Safe model = **one repository per tenant** (separate repo => separate master key). - **Storage-layer isolation that does work:** `rest-server --private-repos` (each authenticated user confined to `/<user>/` subtree) + `--append-only` (block deletion of existing backups). Server-wide flags, not per-tenant toggles (#74). - **Current state:** **per-application** already — each operator app configures its own restic via the `streaming-backup` contract (`nix/contracts/nixos/modules/services/backup/restic.nix`), backing PG + redis dumps to S3 (garage). Already effectively one-repo-per-(app), and thus per-tenant since operator == tenant. - **Next step:** keep the per-tenant repo model. If consolidating storage behind a shared endpoint, front it with `rest-server --private-repos --append-only` and mint a per-tenant repo path + credential + restic password in the **operator group** (operator-TF graph). ## Summary table | Service | One shared instance for many tenants? | Isolation primitive | Enforced? | Recommendation for HP group | |---|---|---|---|---| | **OpenBao** | Yes | Namespaces (OSS, GA 2.3.x) | Strong | Share + per-tenant namespace | | **PostgreSQL** | Yes | DB-per-tenant + per-tenant role | Strong | Share cluster (or keep local-per-app) | | **BIND** | Yes (by design) | Per-tenant TSIG key + `update-policy zonesub`, ACLs, catalog zones | Strong (per zone) | Share; mint zone+key per tenant via operator group | | **maddy** | Yes (many domains) | Email-keyed virtual users; `auth_map`/`storage_map` | Soft only | Share only if single trusted op; else per-tenant | | **lldap** | **No** | n/a (single base DN) | — | One instance per tenant (operator group) | | **Authelia** | **No** (end-users) | ACLs over one shared user pool | Soft only | One instance per tenant (operator group) | | **restic** | **No** (one repo) | per-repo master key; rest-server `--private-repos` | Strong only as 1-repo-per-tenant | One repo per tenant (already so) | ## Reference pattern: how garage mints per-app (per-tenant) slices This is the reusable construction the per-tenant services above should mirror. The driving model lives in `src/lib/data-model.nix` (resource = `request` + `policy`, lines ~62-125): an app declares a **request**, the policy's generic `process` collects all apps' requests of that type into one list, and the policy's **`apply`** turns that aggregated list into node configs. **garage** (`src/resources/garage/default.nix`) is the worked example: 1. **Request (app side).** Each app declares the slice it needs. Mastodon (`src/applications/operator/mastodon/default.nix`): `ensureBuckets.mastodon`, `ensureBuckets.mastodon-backups`, `ensureKeys.mastodon` with per-bucket `ensureAccess` (read/write/owner), plus a `nixos-configuration` that wires the app's S3 endpoint. 2. **Aggregation.** `resource.policy.process` (data-model.nix) filters every app's requests down to the garage ones and flattens them to a single list — collected in the setup at `src/setups/hosts-common.nix` (the `ancilliaryRequests`/`process` wiring, ~lines 302-328, 649). 3. **Apply.** `garage/default.nix`'s `apply(requests)` emits three things from the aggregated list: - `applicationSide` — per-app NixOS config (S3 endpoint, bucket domain, key file paths) wired into each consuming app node. - `garageSide` — a `ensure-garage` **systemd oneshot on the garage node** that, after `garage.service`, replays the aggregated requests: `garage bucket create` / `key import` / `bucket allow` for *every* app's buckets+keys in one pass. - `sslDomains` — cross-node TLS domains to register. 4. **Routing.** The setup wires `garageSide` onto the garage node and `applicationSide` onto each app node (`hosts-common.nix` ~lines 869-873, 947-952). **Net shape:** *per-app request -> generic aggregation -> single `apply` -> (per-app config wired to app nodes) + (one provisioning unit on the shared node that mints all slices) + (cross-node artifacts).* Isolation is by bucket name + per-app key with scoped grants. **Important caveat about where minting runs.** In *this* branch garage provisioning runs **on the garage node** (the `ensure-garage` systemd unit), so it re-runs on every deploy of that node and picks up the current aggregated request set — it is **not** an operator-side TF graph here. The "mint per-tenant valkey/garage in the **operator-TF graph**" model (so a shared HP node is *not* redeployed when an operator changes) is the **target architecture from sibling branches** (`central-valkey`, `host-garage-central`), not yet realized on `multi-tenant-infra`. A real shared-HP rollout of the new per-tenant services should adopt that operator-TF variant: same request/`apply` shape, but `apply` emits **OpenTofu resources** (e.g. a garage TF provider, an OpenBao namespace resource, a BIND zone+TSIG resource) into the **operator graph** keyed per tenant, rather than a systemd unit on a shared node. The two-system distinction matters: - **Resource system** (garage today): provisioning materializes at NixOS eval/activation time on a node. Good when the provider node is (re)deployed alongside. - **Contract `providerRequests` routing** (`nix/contracts/lib/contracts/module.nix`, ~444-485): groups each contract request by the provider it's routed to — the hook a TF-apply-time per-tenant minting path would use to fan requests to a per-provider operator-TF graph. **valkey:** no `src/resources/valkey/` exists on this branch; the per-tenant valkey construction referenced in design notes lives on the `central-valkey` branch (shared HP valkey + redis contract/external provider, per-tenant valkey minted by an `operator-tf-incus-hosts` TF graph). When ported here it would follow the same request -> aggregate -> `apply`-emits-TF shape. ## Key architectural implication The clean split: **OpenBao, PostgreSQL, BIND** can graduate into the shared hosting-provider group with enforced per-tenant slices (namespace / DB+role / zone+TSIG). **lldap, Authelia, restic, and maddy-for-isolation** must be minted **one-per-tenant via the operator group / operator-TF graph**, exactly like the existing garage/valkey per-tenant pattern — because (a) they have no enforced in-instance tenant boundary, and (b) per-tenant resources must track operator changes without redeploying a shared HP node. A shared hosting-provider lldap/Authelia remains valid only for HP-internal admin-UI SSO, which is a single trusted org, not a multi-tenant surface. All of the "next step" verdicts above reuse the **same construction** described in the reference-pattern section: a resource `request` per consumer, generic aggregation, and an `apply` that emits either a provisioning unit on a shared node (resource system, like garage today) or per-tenant OpenTofu resources in the operator graph (the central-valkey/host-garage-central target). The new enforced-isolation slices map onto `apply` outputs directly: an OpenBao **namespace**, a PostgreSQL **DB+role**, a BIND **zone + TSIG key + `update-policy`** — each keyed per tenant. This also argues for introducing an explicit **`tenant`** notion in `src/lib/data-model.nix` so per-tenant minting (today implicit in operator == tenant) becomes first-class and the per-tenant-vs-shared routing above is expressible. ## Verification This is an assessment; "verification" = confirming the upstream claims behind each verdict before acting on one. Authoritative sources: - OpenBao namespaces (OSS, GA): https://openbao.org/docs/concepts/namespaces/ , https://openbao.org/blog/namespaces-announcement/ ; sealing caveat https://openbao.org/docs/rfcs/namespace-sealing/ - PostgreSQL tenancy models: https://www.postgresql.org/docs/current/ddl-rowsecurity.html , https://planetscale.com/blog/approaches-to-tenancy-in-postgres - BIND `update-policy`/TSIG/catalog zones: https://bind9.readthedocs.io/en/stable/reference.html , https://bind9.readthedocs.io/en/stable/chapter6.html - maddy multiple domains: https://maddy.email/multiple-domains/ - lldap single-org FAQ: https://github.com/lldap/lldap/blob/main/docs/faq.md - Authelia no-tenancy: https://github.com/authelia/authelia/discussions/5438 , https://www.authelia.com/roadmap/active/multi-domain-protection/ - restic single-master-key + rest-server: https://restic.readthedocs.io/en/stable/070_encryption.html , https://github.com/restic/rest-server/blob/master/README.md When a verdict is implemented, validate via the existing apps-tf nspawn test harness (per-tenant resource minting already exercised there for garage/valkey) plus a per-service isolation subtest (e.g. tenant-A TSIG key rejected against tenant-B zone; tenant-A OpenBao token denied in tenant-B namespace). </details>
kiara changed title from reproduce project infrastructure to reproducible project infrastructure 2025-06-01 15:09:13 +02:00
kiara added this to the Dev project 2026-08-04 15:07:25 +02:00
kiara self-assigned this 2026-08-04 15:07:44 +02:00
kiara commented 2026-08-10 23:17:54 +02:00
Author
Owner
Copy link

all up and running by now

all up and running by now
kiara closed this issue 2026-08-10 23:19:12 +02:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
operator-group-comes-up
ci-var-image-fingerprint-null
deploy-report-degraded
automation/npins-update/systemd-openbao
automation/npins-update/nixpkgs
automation/npins-update/git-hooks
deprecate-deployments
jsonschema-frontend
update-nixpkgs-forgejo-16
fix-git-pages-native-cli
automation/npins-update/nixpkgs-unstable
automation/npins-update/forgejo-src
promote-updater
test-action-main
host-garage-central
demo-new
demo-old
schema-change-visuals
schema-change-viz
ci/panel-incus
automation/npins-update/terraform-provider-proxmox
automation/npins-update/nix-src
automation/npins-update/nix
automation/npins-update/flake-parts
automation/npins-update/clan-core
kiara/nspawn-rebase
fix/nixpkgs-contracts-26-05
feat/garage-to-hosting-provider
feat/authelia-ancilliary
kiara/nspawn-panel-fix
kiara/nspawn
issues/124-database-backups
fix/forgejo-ci-secrets-rekey
fix/forgejo-ci-network-gateway
fix/forgejo-ci-ipv6-gateway-interface
feat/tests-thru-flake
issue/126-filesystem-backup
automation/update-workflow
kiara/fix-pixelfed
issues/125-garage-backups
ci/pixelfed
pixelfed-ci
No results found.
Labels
Clear labels   
0 points
fibonacci complexity: 0 points

  
0.5 points
fibonacci complexity: 0.5 points

  
1 point
fibonacci complexity: 1 point

  
13 points
fibonacci complexity: 13 points

  
2 points
fibonacci complexity: 2 points

  
21 points
fibonacci complexity: 21 points

  
3 points
fibonacci complexity: 3 points

  
34 points
fibonacci complexity: 34 points

  
5 points
fibonacci complexity: 5 points

  
55 points
fibonacci complexity: 55 points

  
8 points
fibonacci complexity: 8 points

  
ambition
application-offering

  
ambition
configure-applications

  
ambition
front-end

  
ambition/install-applications
  
ambition
security

  
ambition
switch-host

  
ambition
update-applications

  
ambition
user-management

  
api service
features that may relate more to a (to be created) API service

  
blocked

   
component: fediversity panel

   
component: nixops4

   
documentation

   
estimation high: >3d

   
estimation low: <2h

   
estimation mid: <8h

   
infinite points
fibonacci complexity: infinite points

  
productisation
features out-scoped with the decoupling of the productisation effort

  
project-management
Project management related tasks

  
question

   
role: application developer

   
role: application operator

   
role: hosting provider

   
role: maintainer

   
security

   
technical debt

   
testing

   
type unclear

   
type: bug
Anything that doesn't work as advertised

  
type: deliverable

   
type: key result

   
type: objective

   
type: task

   
type: user story

   
user experience
No labels
0 points
0.5 points
1 point
13 points
2 points
21 points
3 points
34 points
5 points
55 points
8 points
ambition
application-offering
ambition
configure-applications
ambition
front-end
ambition/install-applications
ambition
security
ambition
switch-host
ambition
update-applications
ambition
user-management
api service
blocked
component: fediversity panel
component: nixops4
documentation
estimation high: >3d
estimation low: <2h
estimation mid: <8h
infinite points
productisation
project-management
question
role: application developer
role: application operator
role: hosting provider
role: maintainer
security
technical debt
testing
type unclear
type: bug
type: deliverable
type: key result
type: objective
type: task
type: user story
user experience
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Dev
Assignees
Clear assignees
No assignees
kiara
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#288 external developers empowered to contribute
fediversity/fediversity
Depends on
#309 port infra to terraform
fediversity/fediversity
#325 Reproducible proxmox installation
fediversity/fediversity
#356 reproduce CI runner
fediversity/fediversity
#370 fediversity apps reused in infra
fediversity/fediversity
#433 deployment provisions host infra
fediversity/fediversity
#833 unit-tested functionality integrated in deployment pipelines
fediversity/fediversity
Reference
fediversity/fediversity#336
No description provided.