centralize operator redis onto a shared OpenBao-brokered valkey #1124
No reviewers
fediversity/developers
Labels
No labels
0 points
0.5 points
1 point
13 points
2 points
21 points
3 points
34 points
5 points
55 points
8 points
ambition
application-offering
ambition
configure-applications
ambition
front-end
ambition/install-applications
ambition
security
ambition
switch-host
ambition
update-applications
ambition
user-management
api service
blocked
component: fediversity panel
component: nixops4
documentation
estimation high: >3d
estimation low: <2h
estimation mid: <8h
infinite points
productisation
project-management
question
role: application developer
role: application operator
role: hosting provider
role: maintainer
security
technical debt
testing
type unclear
type: bug
type: deliverable
type: key result
type: objective
type: task
type: user story
user experience
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
fediversity/fediversity!1124
Loading…
Reference in a new issue
No description provided.
Delete branch "kiara/fediversity:central-valkey"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Replace the per-app local redis (mastodon/pixelfed/peertube running their
own
createLocallyredis over a unix socket) with a single sharedvalkeynode in the hosting-provider group, served through a new
rediscontract andan
externalprovider.Per-tenant auth is delegated to OpenBao's
valkey-databasesecrets engine:the valkey node holds only one admin ACL user, and OpenBao mints a per-tenant
static role (stable username = namespace, rotated password, ACL keyspace
scoped to
~<namespace>:*). The openbao agent renders the rotated password toa file the app reads, so valkey never learns tenant passwords and onboarding a
tenant needs no valkey redeploy.
The valkey-database plugin static role only rotates the password of a
pre-existing ACL user (
rotation_statementswithACL SETUSERis rejected),so the engine pre-creates the scoped user via
valkey-cli ACL SETUSERthen arotate-only static role.
Consumers flip to remote TCP+auth only when the
externalprovider is routed,keeping the inert
createLocallybaseline for the single-app service tests.The per-app redis
streamingBackuprequests and dead restic redis entries aredropped; on-node RDB/AOF persistence is configured and offsite backup deferred
to operator-side garage (#911).
test-valkey-serviceexercises the real production credential path end to end:the engine mints and rotates the tenant's password, the openbao agent renders
it via
extraTemplates, and a client authenticates as the per-tenant ACL user,is denied outside its keyspace, and reconnects after a forced rotation.
TLS to valkey is deferred (plaintext, consistent both sides); OpenBao's PKI
secrets engine is the preferred future alternative to self-signed certs.
follow-ups:
@ -350,4 +409,0 @@restoreCmd = ''tee /var/lib/redis-mastodon/dump.rdb > /dev/null'';};for backups this is still pending on #911.
5d4c7a53db6203e682c56203e682c579602c12e3e662bf49ffb5d7fe2b07garagefrom operator ancilliary to hosting-provider (multi-tenant) #911tenanttonamespacefor the shared-node keyspace`tenant` was overloaded: the garage resource used it for the OPERATOR (`tenant = group`, prefix `<operator>-<name>`), while the valkey/redis path used it for the keyspace PARTITION (the `<namespace>:*` ACL scope, `redisTenants`, the operator-TF `tenants` var) -- bare app names today. The two senses were only safe to conflate because there is one static operator group. Reserve `tenant` for the operator (the isolation unit), and rename the partition sense to `namespace` throughout: - TF operator graph: `var.tenants` -> `var.namespaces`, `tenant_rotation_period` -> `namespace_rotation_period`, and the per-namespace resource addresses (`redisacl_user.namespace`, `garage_bucket.namespace{,_backups}`, `garage_key.namespace*`, `vault_kv_secret_v2.namespace_*`, `null_resource.namespace_valkey_backup_timer`). - effects: the `tf-incus-operator-hosts` option `tenants` -> `namespaces` (and its `TF_VAR_namespaces` env key + `namespacesMap`). - setups: `redisTenants` -> `redisNamespaces`, the `tf-incus-operator-hosts.nix` `namespaces` arg, and the consuming `namespaces = redisNamespaces`. - valkey/openbao/redis modules + tests + `valkey-backup.py`/`-timer.sh`: prose `per-tenant`/`tenant-blind`/`co-tenant` -> per-namespace etc., and `tenantPasswordFile` -> `namespacePasswordFile`. The garage resource's `tenant` option (= operator prefix), `garageTenantPrefix`, `garage_tenant_prefix`, `tenant = group`/`""`, and "multi-tenant" (= multi-operator) are left intact: those are now the consistent operator-sense uses. Pure rename: no behavior change. `redisNamespaces`/`var.namespaces` still hold bare app names (the under-keying fix is a separate commit). Assisted-by: Claude:claude-opus-4-8The `valkey-cli ... --pass wrongpassword PING` line interpolates only Nix `${cli}`/`${userArg}`, leaving no Python `{}` placeholder, so the rendered test script tripped ruff `F541` (f-string without placeholders) and failed the `nixos-test-driver` lint step. Drop the stray `f` prefix. Assisted-by: Claude:claude-opus-4-8WIP: centralize operator redis onto a shared OpenBao-brokered valkeyto centralize operator redis onto a shared OpenBao-brokered valkey@ -0,0 +5,4 @@}:{applications.valkey = {description = "The group's valkey cache node (the `external` redis provider's daemon): one `valkey@<namespace>` instance per redis-routed application in its group, each on its own namespace-derived port. Isolation is physical (one process per namespace). Each instance's bootstrap password is generated here; the central OpenBao valkey-database engine rotates the per-instance `app` user.";why external provider if this is explicitly duplicated across groups so as to make sure it would not need to facilitate such out-of-band interactions?
@ -0,0 +51,4 @@qemuNodeMemory = 2048;requiredMemory =lib.throwIf (2560 < qemuNodeMemory)why compare static values?
@ -0,0 +36,4 @@## `ssl` cross-node-trust machinery (`security.pki.certificateFiles`).#### FUTURE: self-signed is a stop-gap. Prefer OpenBao's PKI secrets engine as a## proper internal CA once that is stood up.weren't they since #1119?
@ -21,0 +25,4 @@# preferred future CA). When on, the engine connects over TLS and trusts the# valkey cert via `valkeyCaCert`.valkeyTls = config.services.openbao-spire-auth.valkeyTls;valkeyCaCert = config.services.openbao-spire-auth.valkeyCaCert;might it have made sense here to use contracts?
@ -340,1 +411,4 @@contracts.streamingBackup.defaultProviderName = "restic";# The redis cache moved to the shared valkey node (persisted there# via RDB/AOF; offsite backup deferred to operator-side garage /# #911), so it is no longer backed up per-app here -- only postgres.TODO look into if we can do better here (can that valkey node handle this already? garage i thought we felt we might not need operator-side?)
@ -165,3 +165,3 @@# raise Exception("cannot detect the uploaded image on mastodon page.")for db in ["redis", "postgres"]:for db in ["postgres"]:had this been handled elsewhere yet? and if so, do we feel confident it works for this application?
@ -88,4 +91,0 @@};# Use self-signed certs in tests; the `ssl` resource's `applicationSide`# (wired in `hosts-common.nix`) owns the provider and nginx cert/key.environments.garage.resources."tls".ssl.routing.contracts.ssl.defaultProviderName = "self-signed";did we no longer need something like this?
@ -0,0 +31,4 @@"mastodon""peertube""pixelfed"];this seems prone to going out of sync in case of newly added operator applications, raising questions on to what extent this test might serve as a canary there in its current form
@ -0,0 +4,4 @@## *outside* any single operator's knowledge (in the hosting-provider group, like## openbao), so from an operator's vantage point the cache is an external service## it is merely pointed at. This provider holds none of the daemon: it is a pure## cross-node function of static facts.i thought we moved valkey to
sharedso we could settle for regular contract providers to transmit info? is this used in the sense of 'not node-local', or could (/ should) we not have the provider instantiateconfigon its node for some reason? or did we never have a need to for this particular provider (or even this contract in general)?@ -0,0 +89,4 @@"uni-mastodon" = 7012;"uni-peertube" = 6906;"uni-pixelfed" = 7251;};why is a test concern in the implementation
hardcoded-redistest provider tovalkey731266a7f6superseded by #1132 + #1141
Pull request closed