WIP: move garage from operator ancilliary to hosting-provider (multi-tenant) #911

Closed

kiara wants to merge 1 commit from feat/garage-to-hosting-provider into main

pull from: feat/garage-to-hosting-provider

merge into: fediversity:main

fediversity:main

fediversity:ci-var-image-fingerprint-null

fediversity:deploy-report-degraded

fediversity:automation/npins-update/systemd-openbao

fediversity:automation/npins-update/nixpkgs

fediversity:automation/npins-update/git-hooks

fediversity:deprecate-deployments

fediversity:jsonschema-frontend

fediversity:update-nixpkgs-forgejo-16

fediversity:fix-git-pages-native-cli

fediversity:automation/npins-update/nixpkgs-unstable

fediversity:automation/npins-update/forgejo-src

fediversity:promote-updater

fediversity:test-action-main

fediversity:host-garage-central

fediversity:demo-new

fediversity:demo-old

fediversity:schema-change-visuals

fediversity:schema-change-viz

fediversity:ci/panel-incus

fediversity:automation/npins-update/terraform-provider-proxmox

fediversity:automation/npins-update/nix-src

fediversity:automation/npins-update/nix

fediversity:automation/npins-update/flake-parts

fediversity:automation/npins-update/clan-core

fediversity:kiara/nspawn-rebase

fediversity:fix/nixpkgs-contracts-26-05

fediversity:feat/authelia-ancilliary

fediversity:kiara/nspawn-panel-fix

fediversity:kiara/nspawn

fediversity:issues/124-database-backups

fediversity:fix/forgejo-ci-secrets-rekey

fediversity:fix/forgejo-ci-network-gateway

fediversity:fix/forgejo-ci-ipv6-gateway-interface

fediversity:feat/tests-thru-flake

fediversity:issue/126-filesystem-backup

fediversity:automation/update-workflow

fediversity:kiara/fix-pixelfed

fediversity:issues/125-garage-backups

fediversity:ci/pixelfed

fediversity:pixelfed-ci

Conversation 4 Commits 1 Files changed 12 +586 -243

kiara commented 2026-05-07 09:52:03 +02:00

Owner

Copy link

Summary

• Moves garage S3 storage from operator-level ancilliary to the hosting-provider layer, so all operators share one garage cluster
• Operator app bucket/key names are transparently prefixed with the operator group name at the policy apply boundary (e.g. mastodon -> operator-mastodon)
• Cross-group bucket/key requests are aggregated at eval time in hosts-common.nix and injected into the hosting-provider garage module via _module.args.aggregatedGarageRequests
• Tests that bundle a local garage (mastodon/peertube/pixelfed service tests) are unaffected via the ancilliaryHosts ? garage guard
• CI workflows regenerated: adds cd-hosting-provider-garage.yaml, nixosConfigurations-hosting-provider-garage.yaml; removes nixosConfigurations-operator-garage.yaml

Test plan

• nix build --impure .#nixosConfigurations.hosting-provider-garage.config.system.build.toplevel -L builds clean
• nix eval --impure .#nixosConfigurations.operator-mastodon.config.systemd.services.mastodon-web.environment.S3_BUCKET == "operator-mastodon"
• nix build .#checks.x86_64-linux.test-mastodon-service -L passes
• nix build .#checks.x86_64-linux.test-peertube-service -L passes
• nix build .#checks.x86_64-linux.test-pixelfed-service -L passes
• nix build .#checks.x86_64-linux.pre-commit -L passes

Disclaimer: I used a coding agent in the creation of this patch.

## Summary - Moves garage S3 storage from operator-level ancilliary to the hosting-provider layer, so all operators share one garage cluster - Operator app bucket/key names are transparently prefixed with the operator group name at the policy `apply` boundary (e.g. `mastodon` -> `operator-mastodon`) - Cross-group bucket/key requests are aggregated at eval time in `hosts-common.nix` and injected into the hosting-provider garage module via `_module.args.aggregatedGarageRequests` - Tests that bundle a local garage (mastodon/peertube/pixelfed service tests) are unaffected via the `ancilliaryHosts ? garage` guard - CI workflows regenerated: adds `cd-hosting-provider-garage.yaml`, `nixosConfigurations-hosting-provider-garage.yaml`; removes `nixosConfigurations-operator-garage.yaml` ## Test plan - [ ] `nix build --impure .#nixosConfigurations.hosting-provider-garage.config.system.build.toplevel -L` builds clean - [ ] `nix eval --impure .#nixosConfigurations.operator-mastodon.config.systemd.services.mastodon-web.environment.S3_BUCKET` == `"operator-mastodon"` - [ ] `nix build .#checks.x86_64-linux.test-mastodon-service -L` passes - [ ] `nix build .#checks.x86_64-linux.test-peertube-service -L` passes - [ ] `nix build .#checks.x86_64-linux.test-pixelfed-service -L` passes - [ ] `nix build .#checks.x86_64-linux.pre-commit -L` passes Disclaimer: I used a coding agent in the creation of this patch.

kiara added 1 commit 2026-05-07 09:52:03 +02:00

move garage from operator ancilliary to hosting-provider (multi-tenant) ... 6f9285659c

- `garage` is now a `hosting-provider` application at `garage.abundos.eu`;
  bucket/key names are transparently prefixed `<operator>-<name>` at
  policy `apply` time so multiple operator groups share one cluster
  without namespace collisions
- `resources/garage/default.nix`: add `tenant` option; rewrite bucket/key
  names in `rawRequests` and `applicationSide` via `prefixName`; add
  `rawRequests` field to the resource-type submodule; keep `mainConfig`
  intact for tests that bundle a local garage
- new `src/applications/hosting-provider/garage.nix`: NixOS module that
  hosts the garage server and an `ensure-garage` oneshot provisioning
  all operator buckets/keys from `aggregatedGarageRequests`
- `hosts-common.nix`: cross-group evaluation collects every operator
  app's `ancilliaryRequests.garageRequests` into `aggregatedGarageRequests`
  and injects it as `_module.args` into the hosting-provider's garage node;
  guarded by `!(ancilliaryHosts ? garage)` so test groups that bundle a
  local garage still take the existing code path unchanged
- operator app modules (mastodon, peertube, pixelfed) call
  `garage.bucketName` to obtain the physically-prefixed bucket name for
  environment variables
- `groups/hosting-provider.nix`: add `garage` machine entry; auto-import
  loop picks up the new application module automatically
- `groups/operator.nix`: remove `ancilliary.garage`; `machines.ancilliary`
  now contains only `authelia`
- CI workflows regenerated: add `cd-hosting-provider-garage.yaml` and
  `nixosConfigurations-hosting-provider-garage.yaml`; remove
  `nixosConfigurations-operator-garage.yaml`; add `garage.nix` to the
  `hosting-provider-combined` path filter

Assisted-by: Claude:claude-opus-4-7

kiara commented 2026-05-18 13:23:07 +02:00

Author

Owner

Copy link

while this was intended as a PoC for setup interaction between machines between the operator vs hosting provider groups (on top of... it is actually supposed to be shared), it has instead mostly demonstrated conceptual challenges: whereas in the current setup operators' machines were presumed to rebuild in conjunction as requirements evolve, with ancilliary machines (so far garage) gathering requests from applications to assemble their NixOS configuration (src/resources/garage/default.nix's garageSide), the elephant in the room here is that machines part of the hosting provider group may not really be expected to rebuild with info gathered from operator-side machines - i.e. where the current PR intended to move garage.

for garage, that had been used in the current set-up to for applications' various requests invoke:

• garage bucket create (#187)
• garage bucket allow
• garage bucket deny
• garage bucket alias
• garage bucket website --import
• garage key import
• aws put-bucket-cors

now, lacking the ability to handle these thru nixos config, that would leave approaches like:

• LDAP (potentially relevant for authorization-related logic): not so far supported by garage, tho it is supported by ceph, which additionally is distributed unlike garage
• terraform (providers with various support for different resources available for garage and ceph - in case of garage further being split by garage version)
• otherwise ensuring the CLI commands are run on the shared machine

while this was intended as a PoC for setup interaction between machines between the operator vs hosting provider groups (on top of... it is actually supposed to be shared), it has instead mostly demonstrated conceptual challenges: whereas in the current setup operators' machines were presumed to rebuild in conjunction as requirements evolve, with ancilliary machines (so far garage) gathering requests from applications to assemble their NixOS configuration (`src/resources/garage/default.nix`'s `garageSide`), the elephant in the room here is that machines part of the hosting provider group may not really be expected to rebuild with info gathered from operator-side machines - i.e. where the current PR intended to move `garage`. for garage, that had been used in the current set-up to for applications' various requests invoke: - [ ] `garage bucket create` (#187) - [ ] `garage bucket allow` - [ ] `garage bucket deny` - [ ] `garage bucket alias` - [ ] `garage bucket website --import` - [ ] `garage key import` - [ ] `aws put-bucket-cors` now, lacking the ability to handle these thru nixos config, that would leave approaches like: - LDAP (potentially relevant for authorization-related logic): not so far supported by garage, tho it is supported by [ceph](https://docs.ceph.com/en/latest/radosgw/ldap-auth/), which additionally is distributed unlike garage - terraform (providers with various support for different resources available for [garage](https://registry.terraform.io/search/providers?q=garage) and [ceph](https://registry.terraform.io/search/providers?q=ceph) - in case of garage further being split by garage version) - otherwise ensuring the CLI commands are run on the shared machine

toonn commented 2026-05-18 14:46:59 +02:00

Member

Copy link

LDAP would decouple creation of buckets from deploying of services IIUC? It would basically store "Application X needs buckets Y and Z," the Garage host would then make sure it creates the required buckets (how is this triggered, polling?).

Terraform would be run as part of a deployment, ensuring the buckets are deployed before the Application comes online, right?

How about the removal of buckets if an Operator decides to stop using an Application or migrates?

LDAP would decouple creation of buckets from deploying of services IIUC? It would basically store "Application X needs buckets Y and Z," the Garage host would then make sure it creates the required buckets (how is this triggered, polling?). Terraform would be run as part of a deployment, ensuring the buckets are deployed before the Application comes online, right? How about the removal of buckets if an Operator decides to stop using an Application or migrates?

kiara commented 2026-05-18 16:58:41 +02:00

Author

Owner

Copy link

Terraform would be run as part of a deployment, ensuring the buckets are deployed before the Application comes online, right?

right.

How about the removal of buckets if an Operator decides to stop using an Application or migrates?

so, CRUD is what TF is about, for what it's worth, so should know how to clean up after itself, tho with migration i'm not positive yet if that's the way to do it.

if they stop using an application, another factor is user preference on what to do with the data, see #186.

LDAP would decouple creation of buckets from deploying of services IIUC? It would basically store "Application X needs buckets Y and Z," the Garage host would then make sure it creates the required buckets (how is this triggered, polling?).

i'd have imagined the CRUD things to be closer to terraform's thing than to LDAP's, which i would expect more like to handle the authorization bits

> Terraform would be run as part of a deployment, ensuring the buckets are deployed before the Application comes online, right? right. > How about the removal of buckets if an Operator decides to stop using an Application or migrates? so, CRUD is what TF is about, for what it's worth, so should know how to clean up after itself, tho with migration i'm not positive yet if that's the way to do it. if they stop using an application, another factor is user preference on what to do with the data, see #186. > LDAP would decouple creation of buckets from deploying of services IIUC? It would basically store "Application X needs buckets Y and Z," the Garage host would then make sure it creates the required buckets (how is this triggered, polling?). i'd have imagined the CRUD things to be closer to terraform's thing than to LDAP's, which i would expect more like to handle the authorization bits

kiara referenced this pull request 2026-05-19 20:50:04 +02:00

add queue #984

kiara referenced this pull request 2026-05-21 12:19:22 +02:00

unit-tested functionality integrated in deployment pipelines #833

kiara referenced this pull request 2026-06-21 17:09:20 +02:00

hosting provider has admin panel #1068

kiara referenced this pull request 2026-06-23 00:38:12 +02:00

centralize operator redis onto a shared OpenBao-brokered valkey #1124

kiara referenced this pull request 2026-06-23 00:38:49 +02:00

centralize operator redis onto a shared OpenBao-brokered valkey #1124

kiara commented 2026-06-25 22:11:39 +02:00

Author

Owner

Copy link

superseded by #1124.

superseded by #1124.

kiara closed this pull request 2026-06-25 22:11:39 +02:00

kiara referenced this pull request from a commit 2026-06-27 21:35:52 +02:00

review: correct stale TLS + redis-backup comments on central-valkey

All checks were successful

Hide checks

Nix flake completeness check / _complete (pull_request) Successful in 9s

Details

checks-deployment-nixos-hosts / deployment-nixos-hosts (pull_request) Successful in 11s

Details

checks-deployment-ssh-hosts / deployment-ssh-hosts (pull_request) Successful in 1m34s

Details

checks-pre-commit / pre-commit (pull_request) Successful in 5s

Details

checks-test-mastodon-service / test-mastodon-service (pull_request) Successful in 8s

Details

checks-test-peertube-service / test-peertube-service (pull_request) Successful in 10s

Details

checks-test-pixelfed-service / test-pixelfed-service (pull_request) Successful in 10s

Details

devShells-default / default (pull_request) Successful in 6s

Details

checks-deployment-tf-hosts / deployment-tf-hosts (pull_request) Successful in 2m31s

Details

nix-unit-effects-common-lib / effects-common-lib (pull_request) Successful in 8s

Details

nix-unit-effects-tf-common-conversion / effects-tf-common-conversion (pull_request) Successful in 8s

Details

flake-show / apps-ssh (pull_request) Successful in 32s

Details

nix-unit-lib-data-model / lib-data-model (pull_request) Successful in 7s

Details

nix-unit-lib-function / lib-function (pull_request) Successful in 7s

Details

nixosConfigurations-dev-combined / dev-combined (pull_request) Successful in 2s

Details

nix-unit-lib-lib / lib-lib (pull_request) Successful in 7s

Details

nixosConfigurations-dev-forgejo-actions-runner / dev-forgejo-actions-runner (pull_request) Successful in 2s

Details

nixosConfigurations-dev-forgejo / dev-forgejo (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-combined / hosting-provider-combined (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-garage / hosting-provider-garage (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-incus / hosting-provider-incus (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-netbox / hosting-provider-netbox (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-openbao / hosting-provider-openbao (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-panel / hosting-provider-panel (pull_request) Successful in 2s

Details

nixosConfigurations-hosting-provider-terraform-backend / hosting-provider-terraform-backend (pull_request) Successful in 2s

Details

nixosConfigurations-operator-authelia / operator-authelia (pull_request) Successful in 2s

Details

nixosConfigurations-operator-combined / operator-combined (pull_request) Successful in 2s

Details

nixosConfigurations-operator-mastodon / operator-mastodon (pull_request) Successful in 2s

Details

nixosConfigurations-operator-peertube / operator-peertube (pull_request) Successful in 2s

Details

nixosConfigurations-operator-pixelfed / operator-pixelfed (pull_request) Successful in 2s

Details

checks-deployment-tf-incus-hosts / deployment-tf-incus-hosts (pull_request) Successful in 9m53s

Details

checks-apps-tf / apps-tf (pull_request) Successful in 10m18s

Details

checks-apps-tf-incus / apps-tf-incus (pull_request) Successful in 12m17s

Details

checks-apps-panel / apps-panel (pull_request) Successful in 25m29s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

0 points

fibonacci complexity: 0 points

  

0.5 points

fibonacci complexity: 0.5 points

  

1 point

fibonacci complexity: 1 point

  

13 points

fibonacci complexity: 13 points

  

2 points

fibonacci complexity: 2 points

  

21 points

fibonacci complexity: 21 points

  

3 points

fibonacci complexity: 3 points

  

34 points

fibonacci complexity: 34 points

  

5 points

fibonacci complexity: 5 points

  

55 points

fibonacci complexity: 55 points

  

8 points

fibonacci complexity: 8 points

  

ambition

application-offering

  

ambition

configure-applications

  

ambition

front-end

  

ambition/install-applications

  

ambition

security

  

ambition

switch-host

  

ambition

update-applications

  

ambition

user-management

  

api service

features that may relate more to a (to be created) API service

  

blocked

  

component: fediversity panel

  

component: nixops4

  

documentation

  

estimation high: >3d

  

estimation low: <2h

  

estimation mid: <8h

  

infinite points

fibonacci complexity: infinite points

  

productisation

features out-scoped with the decoupling of the productisation effort

  

project-management

Project management related tasks

  

question

  

role: application developer

  

role: application operator

  

role: hosting provider

  

role: maintainer

  

security

  

technical debt

  

testing

  

type unclear

  

type: bug

Anything that doesn't work as advertised

  

type: deliverable

  

type: key result

  

type: objective

  

type: task

  

type: user story

  

user experience

No labels

0 points

0.5 points

1 point

13 points

2 points

21 points

3 points

34 points

5 points

55 points

8 points

ambition

application-offering

ambition

configure-applications

ambition

front-end

ambition/install-applications

ambition

security

ambition

switch-host

ambition

update-applications

ambition

user-management

api service

blocked

component: fediversity panel

component: nixops4

documentation

estimation high: >3d

estimation low: <2h

estimation mid: <8h

infinite points

productisation

project-management

question

role: application developer

role: application operator

role: hosting provider

role: maintainer

security

technical debt

testing

type unclear

type: bug

type: deliverable

type: key result

type: objective

type: task

type: user story

user experience

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

fediversity/fediversity!911

No description provided.