Compare commits

...
Sign in to create a new pull request.

56 commits

Author SHA1 Message Date
3b719cea53 WIP: expose service tests in flake 2025-07-14 13:55:19 +02:00
aef414ffe8 resolve regressions from recent qemu files (#432)
- move import to match module classes
- manually import sources to resolve infinite recursion

closes #431.

Reviewed-on: Fediversity/Fediversity#432
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-11 16:09:27 +02:00
6d74112518 ditch sources arg in fedi201, fixing infinite recursion error (#454)
c.f. #432.

closes #453.

Reviewed-on: Fediversity/Fediversity#454
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-11 16:06:15 +02:00
2b2fb059fd fix cd command (#455)
Reviewed-on: Fediversity/Fediversity#455
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-11 11:07:03 +02:00
66ceb66382 add deployment pipeline (#452)
part of #177

Reviewed-on: Fediversity/Fediversity#452
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-10 16:45:46 +02:00
ad9c61a3db docs: fix typos 2025-07-10 00:37:27 +02:00
b4e1c5b5b3 Restrict fileset necessary for deployment tests (#450)
Now that we won't depend on the flake.nix anymore, we won't depend on all the flake-part.nix files (necessary to evaluate flake.nix) and all the files they depend on etc., so the Nix dependencies of the tests will be drastically reduced, and I will be able to leverage that by introducing a more subtle src. This will make the test not need to re-run if only things outside that reduced src changed (and the previous run is in the Nix store).

Reviewed-on: Fediversity/Fediversity#450
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-09 22:57:52 +02:00
de38611572 Unflakify deployment tests (#449)
This PR builds on top of #447 and #448. Since these might be rejected, there will be some changes needed for this PR as well. Let's see how the discussions go in #447.

In the meantime, @fricklerhandwerk, would you mind (in)validating the core idea of this PR? You only need to look at 7cf43c4041, really.

Reviewed-on: Fediversity/Fediversity#449
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-09 15:07:02 +02:00
1d40dcfc0e Grab git-hooks from npins (#448)
This PR builds on top of #447 and will be subject to the same discussion. Let's discuss there whether it makes sense to get rid of the `flake-parts` and `git-hooks` flake inputs.

Reviewed-on: Fediversity/Fediversity#448
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-09 13:21:48 +02:00
c3bf158130 Note on extracting mkFlake to an external library (#451)
follow-up on Fediversity/Fediversity#447 (comment)

Reviewed-on: Fediversity/Fediversity#451
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-09 12:34:43 +02:00
48c6a1f22b Extract mkFlake to own file - get flake-parts from npins (#447)
The goal is to contain the “`mkFlake` hack” to a file that we can heavily document but otherwise ignore. This also will allow me to reuse it in the “flake under test” of the deployment tests.

Reviewed-on: Fediversity/Fediversity#447
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-09 10:12:47 +02:00
8a7984933d reinstate acme settings needed by applications (#434)
closes #417

Reviewed-on: Fediversity/Fediversity#434
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-08 10:02:13 +02:00
5520fa721b gitea PR unpruned (#445)
see #65

Reviewed-on: Fediversity/Fediversity#445
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-07 13:38:09 +02:00
eabfc228c5 updater: try the first upstream commit without git remote prune (#444)
Reviewed-on: Fediversity/Fediversity#444
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-07 13:23:08 +02:00
3f923532a2 updater: fully qualify github domain in uses (#443)
part of #65.
succeeds #442.

Reviewed-on: Fediversity/Fediversity#443
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-07 12:50:55 +02:00
37d4fc5a42 un-qualify github.com domain in updater uses, which resolved to data.forgejo.org/github.com (#442)
attempt to address https://git.fediversity.eu/Fediversity/Fediversity/actions/runs/920.
part of #65.

Reviewed-on: Fediversity/Fediversity#442
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-07 12:47:06 +02:00
c7b05bb473 deduplicate import-flake, fixing test regression from git merge (#441)
Reviewed-on: Fediversity/Fediversity#441
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-07 10:58:10 +02:00
5345860006 data model: add run-time configuration (#437)
based on @fricklerhandwerk's work at https://git.fediversity.eu/fricklerhandwerk/Fediversity/compare/main...data-model-as-diagram part of #103.
supersedes #402.

on the application model, feedback on a [sample implementation](kiara/Fediversity#2) welcome as well.

Reviewed-on: Fediversity/Fediversity#437
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-06 21:42:25 +02:00
fbb0806072 in update workflow use PR action actually meant for gitea (#438)
see #65

Reviewed-on: Fediversity/Fediversity#438
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-06 21:39:26 +02:00
b0848727fd clarify the use of isNormalUser (#436)
Reviewed-on: Fediversity/Fediversity#436
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-06 09:09:43 +02:00
a882de0b8e address linter gripes in nix files (#430)
Reviewed-on: Fediversity/Fediversity#430
Reviewed-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-03 15:56:16 +02:00
05572ff69e remove string interpolation from imports (#429)
Reviewed-on: Fediversity/Fediversity#429
Reviewed-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-03 15:55:39 +02:00
13c92280ab Clean up lib in forgejo-ci and extend on the .ssh/config comment (#428)
Reviewed-on: Fediversity/Fediversity#428
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-02 17:49:44 +02:00
871672d447 Add forgejo-ci machine to our infrastructure (#389)
picked up from https://git.fediversity.eu/Fediversity/Fediversity/compare/main...niols:forgejo-ci.

closes #356.

Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Reviewed-on: Fediversity/Fediversity#389
Reviewed-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-02 17:43:09 +02:00
6da42936e7 add missing home-manager import to fedipanel VM (#425)
Reviewed-on: Fediversity/Fediversity#425
Reviewed-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-02 17:32:38 +02:00
8df70a2ff0 classify recent flake-parts files 2025-07-02 13:25:23 +02:00
5a92c2c0bc docs: fix links to machines (#426)
Reviewed-on: Fediversity/Fediversity#426
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-02 13:16:38 +02:00
1c92009879
Do not force QEMU options onto machines 2025-07-01 23:55:33 +02:00
a791ad41ec Inject sources, secrets and keys via module system - avoid import ../ (#421)
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Reviewed-on: Fediversity/Fediversity#421
Reviewed-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-01 21:08:15 +02:00
c1dc0fef01 Split nameservers between IPv4 and IPv6 (#420)
Reviewed-on: Fediversity/Fediversity#420
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-07-01 13:09:06 +02:00
5a3cbe4d83 fix agenix package in shell (#422)
as per Fediversity/Fediversity#419 (comment)

Reviewed-on: Fediversity/Fediversity#422
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-07-01 11:29:38 +02:00
fd1d55df5f
move shell from flake 2025-07-01 10:22:58 +02:00
0c23115cff allow configuring network interface (#413)
Reviewed-on: Fediversity/Fediversity#413
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-30 19:23:51 +02:00
3f1c8a9bb7
Document why Nix and OpenSSH
lost in #412. Alternatively, we could have a comment on both lines
saying eg. “for NixOps4”
2025-06-30 14:30:29 +02:00
737aecaba6 set default value for nixops4Package (#412)
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Reviewed-on: Fediversity/Fediversity#412
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-30 12:44:53 +02:00
d7dbdd923c make CI test invocations idempotent to better facilitate manual use (#416)
Reviewed-on: Fediversity/Fediversity#416
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-30 12:42:19 +02:00
1c44004cfe update documentation for #375 (#406)
Reviewed-on: Fediversity/Fediversity#406
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-27 14:40:40 +02:00
ae444d5352 simplify imports (#415)
Reviewed-on: Fediversity/Fediversity#415
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-27 14:01:41 +02:00
e77fdd9eec expose nixops4 in nix-shell (#411)
Instead of Fediversity/Fediversity#406

Eventually we should merge `//panel/default.nix` with `//default.nix` of course.

Co-authored-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Reviewed-on: Fediversity/Fediversity#411
Reviewed-by: Nicolas Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-committed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2025-06-27 12:00:47 +02:00
1f1cf0d516 unset class, fixing #408 (#410)
Reviewed-on: Fediversity/Fediversity#410
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-27 11:53:17 +02:00
f94eac698a
disable updater schedule while it hangs 2025-06-26 17:01:40 +02:00
46182e7512 fix workflow step title (#404)
split off from #399

Reviewed-on: Fediversity/Fediversity#404
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-24 08:58:01 +02:00
9a25a04bfa specify _class module attributes to explicitly declare module types (#398)
closes #93.

note that this includes classes:

- `nixos`
- `nixosTest`
- `nixops4Resource`
- `nixops4Deployment`

.. and my (made-up, as per the [docs](https://ryantm.github.io/nixpkgs/module-system/module-system/#module-system-lib-evalModules-param-class)):

- `nix-unit`
- `package`

.. while i did not manage to cover:

- service tests, given `pkgs.nixosTest` seemed to not actually like `_class = "nixosTest"` (?!)

... nor #93's mentioned destructured arguments for that matter, as per Fediversity/Fediversity#93 (comment) - let me know if that is still desired as well.

Reviewed-on: Fediversity/Fediversity#398
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-23 17:24:54 +02:00
c1b33121b6 expose npins in shell for CI (#403)
Should fix Fediversity/Fediversity#65 (comment)

Reviewed-on: Fediversity/Fediversity#403
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-committed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2025-06-23 17:21:32 +02:00
d073bd706d
cast DEPLOYMENT_FLAKE to string 2025-06-22 14:42:14 +02:00
486b316885 run updater natively (#394)
see Fediversity/Fediversity#65 (comment).

closes #65.

Reviewed-on: Fediversity/Fediversity#394
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-20 09:41:38 +02:00
611c961dcf separate test declarations from invocations (#396)
see Fediversity/Fediversity#395 (comment)

Reviewed-on: Fediversity/Fediversity#396
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-19 18:11:08 +02:00
d67f533948 fix running nixops4 apply test (#391)
Closes #390

Reviewed-on: Fediversity/Fediversity#391
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-committed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2025-06-19 08:26:20 +02:00
bd1cfd7a7c Introduce test for deploying all services via FediPanel (#361)
Closes #277

Same as #329 but where we run the FediPanel and interact with it via a browser
instead of running NixOps4 directly.

Reviewed-on: Fediversity/Fediversity#361
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-06-18 12:37:47 +02:00
939f9d961d add data model entity: application (#387)
part of #103.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Reviewed-on: Fediversity/Fediversity#387
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-17 17:11:52 +02:00
4801433ae0 Get rid of the need for deployer.pub (#385)
The tests still work because we manually write the deployer's public key in `/root/.ssh/authorized_keys` on the target machines. In itself, however, the configuration that we push does not allow the deployer to push anything on the target machines.

Context: Fediversity/Fediversity#361 (comment)
Reviewed-on: Fediversity/Fediversity#385
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-06-17 16:34:29 +02:00
3a3a083793 FediPanel: allow configuring flake and deployment (#376)
Last part of #361.

Builds on top of #375.

Reviewed-on: Fediversity/Fediversity#376
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-06-15 16:55:19 +02:00
ace56e754e FediPanel: do not call nix develop (#375)
Yet another piece of #361.

Reviewed-on: Fediversity/Fediversity#375
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
2025-06-15 15:06:23 +02:00
dbb4ce67fc move machines to reflect a semantic structure (#367)
later we may want to distinguish dev vs host as well, tho eventually we expect not to have hard-coded machines anyway.

split off from #319.

Reviewed-on: Fediversity/Fediversity#367
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
2025-06-15 15:01:56 +02:00
5a514b96e9 use deployed environment for launching nixops4 from the panel 2025-06-13 16:39:34 +02:00
1b832c1f5b bypass native flake input for Nixpkgs (#374)
@Niols the sheer amount of hassle and noise indicates that it may be better to first split out a `flake.nix` just for the tests. And all this clutter doesn't even explain yet *why* we thought it needs to be there.

closes #279.

Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Reviewed-on: Fediversity/Fediversity#374
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-committed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2025-06-12 13:05:11 +02:00
137 changed files with 1777 additions and 599 deletions

View file

@ -0,0 +1,24 @@
name: deploy-infra
on:
workflow_dispatch: # allows manual triggering
push:
branches:
# - main
jobs:
deploy:
runs-on: native
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up SSH key to access age secrets
run: |
env
mkdir -p ~/.ssh
echo "${{ secrets.CD_SSH_KEY }}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
- name: Deploy
run: nix-shell --run 'nixops4 apply default'

View file

@ -15,17 +15,23 @@ jobs:
- uses: actions/checkout@v4
- run: nix-build -A tests
check-data-model:
runs-on: native
steps:
- uses: actions/checkout@v4
- run: nix-shell --run 'nix-unit ./deployment/data-model-test.nix'
check-peertube:
runs-on: native
steps:
- uses: actions/checkout@v4
- run: cd services && nix-build -A tests.peertube
- run: nix-build services -A tests.peertube
check-panel:
runs-on: native
steps:
- uses: actions/checkout@v4
- run: cd panel && nix-build -A tests
- run: nix-build panel -A tests
check-deployment-basic:
runs-on: native
@ -38,3 +44,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: nix build .#checks.x86_64-linux.deployment-cli -L
check-deployment-panel:
runs-on: native
steps:
- uses: actions/checkout@v4
- run: nix build .#checks.x86_64-linux.deployment-panel -L

View file

@ -2,20 +2,23 @@ name: update-dependencies
on:
workflow_dispatch: # allows manual triggering
schedule:
- cron: '0 0 1 * *' # monthly
# FIXME: re-enable when manual run works
# schedule:
# - cron: '0 0 1 * *' # monthly
jobs:
lockfile:
runs-on: ubuntu-latest
runs-on: native
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install Nix
uses: cachix/install-nix-action@v31
- name: Install npins
run: nix profile install 'nixpkgs#npins'
- name: Update npins sources
uses: getchoo/update-npins@v0
- name: Update pins
run: nix-shell --run "npins update"
- name: Create PR
uses: https://github.com/KiaraGrouwstra/gitea-create-pull-request@f9f80aa5134bc5c03c38f5aaa95053492885b397
with:
remote-instance-api-version: v1
token: "${{ secrets.DEPLOY_KEY }}"
branch: npins-update
commit-message: "npins: update sources"
title: "npins: update sources"

View file

@ -154,6 +154,3 @@ details as to what they are for. As an overview:
- [`services/`](./services) contains our effort to make Fediverse applications
work seemlessly together in our specific setting.
- [`website/`](./website) contains the framework and the content of [the
Fediversity website](https://fediversity.eu/)

View file

@ -10,6 +10,8 @@ let
gitignore
;
inherit (pkgs) lib;
inherit (import sources.flake-inputs) import-flake;
inherit ((import-flake { src = ./.; }).inputs) nixops4;
pre-commit-check =
(import "${git-hooks}/nix" {
inherit nixpkgs system;
@ -41,6 +43,30 @@ in
shell = pkgs.mkShellNoCC {
inherit (pre-commit-check) shellHook;
buildInputs = pre-commit-check.enabledPackages;
packages =
let
test-loop = pkgs.writeShellApplication {
name = "test-loop";
runtimeInputs = [
pkgs.watchexec
pkgs.nix-unit
];
text = ''
watchexec -w ${builtins.toString ./.} -- nix-unit ${builtins.toString ./deployment/data-model-test.nix} "$@"
'';
};
in
[
pkgs.npins
pkgs.nil
(pkgs.callPackage "${sources.agenix}/pkgs/agenix.nix" { })
pkgs.openssh
pkgs.httpie
pkgs.jq
pkgs.nix-unit
test-loop
nixops4.packages.${system}.default
];
};
tests = {

View file

@ -3,6 +3,13 @@
This directory contains work to generate a full Fediversity deployment from a minimal configuration.
This is different from [`../services/`](../services) that focuses on one machine, providing a polished and unified interface to different Fediverse services.
## Data model
The core piece of the project is the [Fediversity data model](./data-model.nix), which describes all entities and their interactions.
What can be done with it is exemplified in the [evaluation tests](./data-model-test.nix).
Run `test-loop` in the development environment when hacking on the data model or adding tests.
## Checks
There are three levels of deployment checks: `basic`, `cli`, `panel`.
@ -109,8 +116,8 @@ flowchart LR
target_machines -->|get certs| acme
```
### [WIP] Service deployment check from the panel
### Service deployment check from the FediPanel
This is a full deployment check running the panel on the deployer machine, deploying some services through the panel and checking that they are indeed on the target machines, then cleans them up and checks whether that works, too.
This is a full deployment check running the [FediPanel](../panel) on the deployer machine, deploying some services through it and checking that they are indeed on the target machines, then cleans them up and checks whether that works, too.
It builds upon the basic and CLI deployment checks.
It builds upon the basic and CLI deployment checks, the only difference being that `deployer` runs NixOps4 only indirectly via the panel, and the `client` node is the one that triggers the deployment via a browser, the way a human would.

View file

@ -0,0 +1,8 @@
{
targetMachines = [
"hello"
"cowsay"
];
pathToRoot = ../../..;
pathFromRoot = ./.;
}

View file

@ -0,0 +1,14 @@
{
runNixOSTest,
inputs,
sources,
}:
runNixOSTest {
imports = [
../common/nixosTest.nix
./nixosTest.nix
];
_module.args = { inherit inputs sources; };
inherit (import ./constants.nix) targetMachines pathToRoot pathFromRoot;
}

View file

@ -0,0 +1,36 @@
{
inputs,
sources,
lib,
providers,
...
}:
let
inherit (import ./constants.nix) targetMachines pathToRoot pathFromRoot;
in
{
providers = {
inherit (inputs.nixops4.modules.nixops4Provider) local;
};
resources = lib.genAttrs targetMachines (nodeName: {
type = providers.local.exec;
imports = [
inputs.nixops4-nixos.modules.nixops4Resource.nixos
../common/targetResource.nix
];
_module.args = { inherit inputs sources; };
inherit nodeName pathToRoot pathFromRoot;
nixos.module =
{ pkgs, ... }:
{
environment.systemPackages = [ pkgs.${nodeName} ];
};
});
}

View file

@ -1,54 +0,0 @@
{
self,
inputs,
lib,
...
}:
let
inherit (lib) genAttrs;
targetMachines = [
"hello"
"cowsay"
];
pathToRoot = /. + (builtins.unsafeDiscardStringContext self);
pathFromRoot = ./.;
in
{
perSystem =
{ pkgs, ... }:
{
checks.deployment-basic = pkgs.testers.runNixOSTest {
imports = [
../common/nixosTest.nix
./nixosTest.nix
];
_module.args.inputs = inputs;
inherit targetMachines pathToRoot pathFromRoot;
};
};
nixops4Deployments.check-deployment-basic =
{ providers, ... }:
{
providers = {
inherit (inputs.nixops4.modules.nixops4Provider) local;
};
resources = genAttrs targetMachines (nodeName: {
type = providers.local.exec;
imports = [
inputs.nixops4-nixos.modules.nixops4Resource.nixos
../common/targetResource.nix
];
_module.args.inputs = inputs;
inherit nodeName pathToRoot pathFromRoot;
nixos.module =
{ pkgs, ... }:
{
environment.systemPackages = [ pkgs.${nodeName} ];
};
});
};
}

View file

@ -0,0 +1,22 @@
{
inputs = {
nixops4.follows = "nixops4-nixos/nixops4";
nixops4-nixos.url = "github:nixops4/nixops4-nixos";
};
outputs =
inputs:
import ./mkFlake.nix inputs (
{ inputs, sources, ... }:
{
imports = [
inputs.nixops4.modules.flake.default
];
nixops4Deployments.check-deployment-basic = {
imports = [ ./deployment/check/basic/deployment.nix ];
_module.args = { inherit inputs sources; };
};
}
);
}

View file

@ -1,8 +1,15 @@
{ inputs, ... }:
{ inputs, lib, ... }:
{
_class = "nixosTest";
name = "deployment-basic";
sourceFileset = lib.fileset.unions [
./constants.nix
./deployment.nix
];
nodes.deployer =
{ pkgs, ... }:
{
@ -10,6 +17,12 @@
inputs.nixops4.packages.${pkgs.system}.default
];
# FIXME: sad times
system.extraDependencies = with pkgs; [
jq
jq.inputDerivation
];
system.extraDependenciesFromModule =
{ pkgs, ... }:
{

View file

@ -0,0 +1,11 @@
{
targetMachines = [
"garage"
"mastodon"
"peertube"
"pixelfed"
];
pathToRoot = ../../..;
pathFromRoot = ./.;
enableAcme = true;
}

View file

@ -0,0 +1,19 @@
{
runNixOSTest,
inputs,
sources,
}:
runNixOSTest {
imports = [
../common/nixosTest.nix
./nixosTest.nix
];
_module.args = { inherit inputs sources; };
inherit (import ./constants.nix)
targetMachines
pathToRoot
pathFromRoot
enableAcme
;
}

View file

@ -1 +0,0 @@
## This is a placeholder file. It will be overwritten by the test.

View file

@ -0,0 +1,59 @@
{
inputs,
sources,
lib,
}:
let
inherit (builtins) fromJSON readFile listToAttrs;
inherit (import ./constants.nix)
targetMachines
pathToRoot
pathFromRoot
enableAcme
;
makeTargetResource = nodeName: {
imports = [ ../common/targetResource.nix ];
_module.args = { inherit inputs sources; };
inherit
nodeName
pathToRoot
pathFromRoot
enableAcme
;
};
## The deployment function - what we are here to test!
##
## TODO: Modularise `deployment/default.nix` to get rid of the nested
## function calls.
makeTestDeployment =
args:
(import ../..)
{
inherit lib;
inherit (inputs) nixops4 nixops4-nixos;
fediversity = import ../../../services/fediversity;
}
(listToAttrs (
map (nodeName: {
name = "${nodeName}ConfigurationResource";
value = makeTargetResource nodeName;
}) targetMachines
))
(fromJSON (readFile ../../configuration.sample.json) // args);
in
{
check-deployment-cli-nothing = makeTestDeployment { };
check-deployment-cli-mastodon-pixelfed = makeTestDeployment {
mastodon.enable = true;
pixelfed.enable = true;
};
check-deployment-cli-peertube = makeTestDeployment {
peertube.enable = true;
};
}

View file

@ -1,87 +0,0 @@
{
self,
inputs,
lib,
...
}:
let
inherit (builtins) fromJSON readFile listToAttrs;
targetMachines = [
"garage"
"mastodon"
"peertube"
"pixelfed"
];
pathToRoot = /. + (builtins.unsafeDiscardStringContext self);
pathFromRoot = ./.;
enableAcme = true;
in
{
perSystem =
{ pkgs, ... }:
{
checks.deployment-cli = pkgs.testers.runNixOSTest {
imports = [
../common/nixosTest.nix
./nixosTest.nix
];
_module.args.inputs = inputs;
inherit
targetMachines
pathToRoot
pathFromRoot
enableAcme
;
};
};
nixops4Deployments =
let
makeTargetResource = nodeName: {
imports = [ ../common/targetResource.nix ];
_module.args.inputs = inputs;
inherit
nodeName
pathToRoot
pathFromRoot
enableAcme
;
};
## The deployment function - what we are here to test!
##
## TODO: Modularise `deployment/default.nix` to get rid of the nested
## function calls.
makeTestDeployment =
args:
(import ../..)
{
inherit lib;
inherit (inputs) nixops4 nixops4-nixos;
fediversity = import ../../../services/fediversity;
}
(listToAttrs (
map (nodeName: {
name = "${nodeName}ConfigurationResource";
value = makeTargetResource nodeName;
}) targetMachines
))
(fromJSON (readFile ../../configuration.sample.json) // args);
in
{
check-deployment-cli-nothing = makeTestDeployment { };
check-deployment-cli-mastodon-pixelfed = makeTestDeployment {
mastodon.enable = true;
pixelfed.enable = true;
};
check-deployment-cli-peertube = makeTestDeployment {
peertube.enable = true;
};
};
}

View file

@ -0,0 +1,26 @@
{
inputs = {
nixops4.follows = "nixops4-nixos/nixops4";
nixops4-nixos.url = "github:nixops4/nixops4-nixos";
};
outputs =
inputs:
import ./mkFlake.nix inputs (
{
inputs,
sources,
lib,
...
}:
{
imports = [
inputs.nixops4.modules.flake.default
];
nixops4Deployments = import ./deployment/check/cli/deployments.nix {
inherit inputs sources lib;
};
}
);
}

View file

@ -1,4 +1,9 @@
{ inputs, hostPkgs, ... }:
{
inputs,
hostPkgs,
lib,
...
}:
let
## Some places need a dummy file that will in fact never be used. We create
@ -7,8 +12,25 @@ let
in
{
_class = "nixosTest";
name = "deployment-cli";
sourceFileset = lib.fileset.unions [
./constants.nix
./deployments.nix
# REVIEW: I would like to be able to grab all of `/deployment` minus
# `/deployment/check`, but I can't because there is a bunch of other files
# in `/deployment`. Maybe we can think of a reorg making things more robust
# here? (comment also in panel test)
../../default.nix
../../options.nix
../../configuration.sample.json
../../../services/fediversity
];
nodes.deployer =
{ pkgs, ... }:
{

View file

@ -3,6 +3,7 @@
lib,
pkgs,
config,
sources,
...
}:
@ -16,6 +17,8 @@ let
in
{
_class = "nixos";
imports = [ ./sharedOptions.nix ];
options.system.extraDependenciesFromModule = mkOption {
@ -51,11 +54,13 @@ in
system.extraDependencies =
[
"${inputs.flake-parts}"
"${inputs.flake-parts.inputs.nixpkgs-lib}"
"${inputs.nixops4}"
"${inputs.nixops4-nixos}"
"${inputs.nixpkgs}"
inputs.nixops4
inputs.nixops4-nixos
inputs.nixpkgs
sources.flake-parts
sources.flake-inputs
sources.git-hooks
pkgs.stdenv
pkgs.stdenvNoCC
@ -72,7 +77,7 @@ in
config.system.extraDependenciesFromModule
{
nixpkgs.hostPlatform = "x86_64-linux";
_module.args.inputs = inputs;
_module.args = { inherit inputs sources; };
enableAcme = config.enableAcme;
acmeNodeIP = config.acmeNodeIP;
}

View file

@ -3,6 +3,7 @@
lib,
config,
hostPkgs,
sources,
...
}:
@ -12,6 +13,7 @@ let
toJSON
;
inherit (lib)
types
fileset
mkOption
genAttrs
@ -26,14 +28,6 @@ let
forConcat = xs: f: concatStringsSep "\n" (map f xs);
## The whole repository, with the flake at its root.
## FIXME: We could probably have fileset be the union of ./. with flake.nix
## and flake.lock - I doubt we need anything else.
src = fileset.toSource {
fileset = config.pathToRoot;
root = config.pathToRoot;
};
## We will need to override some inputs by the empty flake, so we make one.
emptyFlake = runCommandNoCC "empty-flake" { } ''
mkdir $out
@ -42,6 +36,8 @@ let
in
{
_class = "nixosTest";
imports = [
./sharedOptions.nix
];
@ -50,16 +46,46 @@ in
## FIXME: I wish I could just use `testScript` but with something like
## `mkOrder` to put this module's string before something else.
extraTestScript = mkOption { };
sourceFileset = mkOption {
## REVIEW: Upstream to nixpkgs?
type = types.mkOptionType {
name = "fileset";
description = "fileset";
descriptionClass = "noun";
check = (x: (builtins.tryEval (fileset.unions [ x ])).success);
merge = (_: defs: fileset.unions (map (x: x.value) defs));
};
description = ''
A fileset that will be copied to the deployer node in the current
working directory. This should contain all the files that are
necessary to run that particular test, such as the NixOS
modules necessary to evaluate a deployment.
'';
};
};
config = {
sourceFileset = fileset.unions [
# NOTE: not the flake itself; it will be overridden.
../../../mkFlake.nix
../../../flake.lock
../../../npins
./sharedOptions.nix
./targetNode.nix
./targetResource.nix
(config.pathToCwd + "/flake-under-test.nix")
];
acmeNodeIP = config.nodes.acme.networking.primaryIPAddress;
nodes =
{
deployer = {
imports = [ ./deployerNode.nix ];
_module.args.inputs = inputs;
_module.args = { inherit inputs sources; };
enableAcme = config.enableAcme;
acmeNodeIP = config.nodes.acme.networking.primaryIPAddress;
};
@ -86,7 +112,7 @@ in
genAttrs config.targetMachines (_: {
imports = [ ./targetNode.nix ];
_module.args.inputs = inputs;
_module.args = { inherit inputs sources; };
enableAcme = config.enableAcme;
acmeNodeIP = if config.enableAcme then config.nodes.acme.networking.primaryIPAddress else null;
});
@ -100,8 +126,16 @@ in
${n}.wait_for_unit("multi-user.target")
'')}
## A subset of the repository that is necessary for this test. It will be
## copied inside the test. The smaller this set, the faster our CI, because we
## won't need to re-run when things change outside of it.
with subtest("Unpacking"):
deployer.succeed("cp -r --no-preserve=mode ${src}/* .")
deployer.succeed("cp -r --no-preserve=mode ${
fileset.toSource {
root = ../../..;
fileset = config.sourceFileset;
}
}/* .")
with subtest("Configure the network"):
${forConcat config.targetMachines (
@ -119,7 +153,6 @@ in
with subtest("Configure the deployer key"):
deployer.succeed("""mkdir -p ~/.ssh && ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa""")
deployer_key = deployer.succeed("cat ~/.ssh/id_rsa.pub").strip()
deployer.succeed(f"echo '{deployer_key}' > ${config.pathFromRoot}/deployer.pub")
${forConcat config.targetMachines (tm: ''
${tm}.succeed(f"mkdir -p /root/.ssh && echo '{deployer_key}' >> /root/.ssh/authorized_keys")
'')}
@ -132,11 +165,16 @@ in
## NOTE: This is super slow. It could probably be optimised in Nix, for
## instance by allowing to grab things directly from the host's store.
with subtest("Override the lock"):
##
## NOTE: We use the repository as-is (cf `src` above), overriding only
## `flake.nix` by our `flake-under-test.nix`. We also override the flake
## lock file to use locally available inputs, as we cannot download them.
##
with subtest("Override the flake and its lock"):
deployer.succeed("cp ${config.pathFromRoot}/flake-under-test.nix flake.nix")
deployer.succeed("""
nix flake lock --extra-experimental-features 'flakes nix-command' \
--offline -v \
--override-input flake-parts ${inputs.flake-parts} \
--override-input nixops4 ${inputs.nixops4.packages.${system}.flake-in-a-bottle} \
\
--override-input nixops4-nixos ${inputs.nixops4-nixos} \
@ -148,9 +186,6 @@ in
inputs.nixops4-nixos.inputs.nixops4.packages.${system}.flake-in-a-bottle
} \
--override-input nixops4-nixos/git-hooks-nix ${emptyFlake} \
\
--override-input nixpkgs ${inputs.nixpkgs} \
--override-input git-hooks ${inputs.git-hooks} \
;
""")

View file

@ -11,6 +11,7 @@ let
inherit (lib) mkOption types;
in
# `config` not set and imported from multiple places: no fixed module class
{
options = {
targetMachines = mkOption {

View file

@ -12,6 +12,8 @@ let
in
{
_class = "nixos";
imports = [
(modulesPath + "/profiles/qemu-guest.nix")
(modulesPath + "/../lib/testing/nixos-test-base.nix")

View file

@ -2,6 +2,7 @@
inputs,
lib,
config,
sources,
...
}:
@ -12,6 +13,8 @@ let
in
{
_class = "nixops4Resource";
imports = [ ./sharedOptions.nix ];
options = {
@ -38,7 +41,7 @@ in
(lib.modules.importJSON (config.pathToCwd + "/${config.nodeName}-network.json"))
];
_module.args.inputs = inputs;
_module.args = { inherit inputs sources; };
enableAcme = config.enableAcme;
acmeNodeIP = trim (readFile (config.pathToCwd + "/acme_server_ip"));

View file

@ -0,0 +1,11 @@
{
targetMachines = [
"garage"
"mastodon"
"peertube"
"pixelfed"
];
pathToRoot = ../../..;
pathFromRoot = ./.;
enableAcme = true;
}

View file

@ -0,0 +1,19 @@
{
runNixOSTest,
inputs,
sources,
}:
runNixOSTest {
imports = [
../common/nixosTest.nix
./nixosTest.nix
];
_module.args = { inherit inputs sources; };
inherit (import ./constants.nix)
targetMachines
pathToRoot
pathFromRoot
enableAcme
;
}

View file

@ -0,0 +1,58 @@
{
inputs,
sources,
lib,
}:
let
inherit (builtins) fromJSON listToAttrs;
inherit (import ./constants.nix)
targetMachines
pathToRoot
pathFromRoot
enableAcme
;
makeTargetResource = nodeName: {
imports = [ ../common/targetResource.nix ];
_module.args = { inherit inputs sources; };
inherit
nodeName
pathToRoot
pathFromRoot
enableAcme
;
};
## The deployment function - what we are here to test!
##
## TODO: Modularise `deployment/default.nix` to get rid of the nested
## function calls.
makeTestDeployment =
args:
(import ../..)
{
inherit lib;
inherit (inputs) nixops4 nixops4-nixos;
fediversity = import ../../../services/fediversity;
}
(listToAttrs (
map (nodeName: {
name = "${nodeName}ConfigurationResource";
value = makeTargetResource nodeName;
}) targetMachines
))
args;
in
makeTestDeployment (
fromJSON (
let
env = builtins.getEnv "DEPLOYMENT";
in
if env == "" then
throw "The DEPLOYMENT environment needs to be set. You do not want to use this deployment unless in the `deployment-panel` NixOS test."
else
env
)
)

View file

@ -0,0 +1,26 @@
{
inputs = {
nixops4.follows = "nixops4-nixos/nixops4";
nixops4-nixos.url = "github:nixops4/nixops4-nixos";
};
outputs =
inputs:
import ./mkFlake.nix inputs (
{
inputs,
sources,
lib,
...
}:
{
imports = [
inputs.nixops4.modules.flake.default
];
nixops4Deployments.check-deployment-panel = import ./deployment/check/panel/deployment.nix {
inherit inputs sources lib;
};
}
);
}

View file

@ -0,0 +1,377 @@
{
inputs,
lib,
hostPkgs,
config,
...
}:
let
inherit (lib)
getExe
;
## Some places need a dummy file that will in fact never be used. We create
## it here.
dummyFile = hostPkgs.writeText "dummy" "dummy";
panelPort = 8000;
panelUser = "test";
panelEmail = "test@test.com";
panelPassword = "ouiprdaaa43"; # panel's manager complains if too close to username or email
fediUser = "test";
fediEmail = "test@test.com";
fediPassword = "testtest";
fediName = "Testy McTestface";
toPythonBool = b: if b then "True" else "False";
interactWithPanel =
{
baseUri,
enableMastodon,
enablePeertube,
enablePixelfed,
}:
hostPkgs.writers.writePython3Bin "interact-with-panel"
{
libraries = with hostPkgs.python3Packages; [ selenium ];
flakeIgnore = [
"E302" # expected 2 blank lines, found 0
"E303" # too many blank lines
"E305" # expected 2 blank lines after end of function or class
"E501" # line too long (> 79 characters)
"E731" # do not assign lambda expression, use a def
];
}
''
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
print("Create and configure driver...")
options = Options()
options.add_argument("--headless")
options.binary_location = "${getExe hostPkgs.firefox-unwrapped}"
service = webdriver.FirefoxService(executable_path="${getExe hostPkgs.geckodriver}")
driver = webdriver.Firefox(options=options, service=service)
driver.set_window_size(1280, 960)
driver.implicitly_wait(360)
driver.command_executor.set_timeout(3600)
print("Open login page...")
driver.get("${baseUri}/login/")
print("Enter username...")
driver.find_element(By.XPATH, "//input[@name = 'username']").send_keys("${panelUser}")
print("Enter password...")
driver.find_element(By.XPATH, "//input[@name = 'password']").send_keys("${panelPassword}")
print("Click Login button...")
driver.find_element(By.XPATH, "//button[normalize-space() = 'Login']").click()
print("Open configuration page...")
driver.get("${baseUri}/configuration/")
# Helpers to actually set and not add or switch input values.
def input_set(elt, keys):
elt.clear()
elt.send_keys(keys)
def checkbox_set(elt, new_value):
if new_value != elt.is_selected():
elt.click()
print("Enable Fediversity...")
checkbox_set(driver.find_element(By.XPATH, "//input[@name = 'enable']"), True)
print("Fill in initialUser info...")
input_set(driver.find_element(By.XPATH, "//input[@name = 'initialUser.username']"), "${fediUser}")
input_set(driver.find_element(By.XPATH, "//input[@name = 'initialUser.password']"), "${fediPassword}")
input_set(driver.find_element(By.XPATH, "//input[@name = 'initialUser.email']"), "${fediEmail}")
input_set(driver.find_element(By.XPATH, "//input[@name = 'initialUser.displayName']"), "${fediName}")
print("Enable services...")
checkbox_set(driver.find_element(By.XPATH, "//input[@name = 'mastodon.enable']"), ${toPythonBool enableMastodon})
checkbox_set(driver.find_element(By.XPATH, "//input[@name = 'peertube.enable']"), ${toPythonBool enablePeertube})
checkbox_set(driver.find_element(By.XPATH, "//input[@name = 'pixelfed.enable']"), ${toPythonBool enablePixelfed})
print("Start deployment...")
driver.find_element(By.XPATH, "//button[@id = 'deploy-button']").click()
print("Wait for deployment status to show up...")
get_deployment_result = lambda d: d.find_element(By.XPATH, "//div[@id = 'deployment-result']//p")
WebDriverWait(driver, timeout=3660, poll_frequency=10).until(get_deployment_result)
deployment_result = get_deployment_result(driver).get_attribute('innerHTML')
print("Quit...")
driver.quit()
match deployment_result:
case 'Deployment Succeeded':
print("Deployment has succeeded; exiting normally")
exit(0)
case 'Deployment Failed':
print("Deployment has failed; exiting with return code `1`")
exit(1)
case _:
print(f"Unexpected deployment result: {deployment_result}; exiting with return code `2`")
exit(2)
'';
in
{
_class = "nixosTest";
name = "deployment-panel";
sourceFileset = lib.fileset.unions [
./constants.nix
./deployment.nix
# REVIEW: I would like to be able to grab all of `/deployment` minus
# `/deployment/check`, but I can't because there is a bunch of other files
# in `/deployment`. Maybe we can think of a reorg making things more robust
# here? (comment also in CLI test)
../../default.nix
../../options.nix
../../../services/fediversity
];
## The panel's module sets `nixpkgs.overlays` which clashes with
## `pkgsReadOnly`. We disable it here.
node.pkgsReadOnly = false;
nodes.deployer =
{ pkgs, ... }:
{
imports = [
(import ../../../panel { }).module
];
## FIXME: This should be in the common stuff.
security.acme = {
acceptTerms = true;
defaults.email = "test@test.com";
defaults.server = "https://acme.test/dir";
};
security.pki.certificateFiles = [
(import "${inputs.nixpkgs}/nixos/tests/common/acme/server/snakeoil-certs.nix").ca.cert
];
networking.extraHosts = "${config.acmeNodeIP} acme.test";
services.panel = {
enable = true;
production = true;
domain = "deployer";
secrets = {
SECRET_KEY = dummyFile;
};
port = panelPort;
deployment = {
flake = "/run/fedipanel/flake";
name = "check-deployment-panel";
};
};
environment.systemPackages = [ pkgs.expect ];
## FIXME: The following dependencies are necessary but I do not
## understand why they are not covered by the fake node.
system.extraDependencies = with pkgs; [
peertube
peertube.inputDerivation
gixy # a configuration checker for nginx
gixy.inputDerivation
];
system.extraDependenciesFromModule = {
imports = [ ../../../services/fediversity ];
fediversity = {
domain = "fediversity.net"; # would write `dummy` but that would not type
garage.enable = true;
mastodon = {
enable = true;
s3AccessKeyFile = dummyFile;
s3SecretKeyFile = dummyFile;
};
peertube = {
enable = true;
secretsFile = dummyFile;
s3AccessKeyFile = dummyFile;
s3SecretKeyFile = dummyFile;
};
pixelfed = {
enable = true;
s3AccessKeyFile = dummyFile;
s3SecretKeyFile = dummyFile;
};
temp.cores = 1;
temp.initialUser = {
username = "dummy";
displayName = "dummy";
email = "dummy";
passwordFile = dummyFile;
};
};
};
};
nodes.client =
{ pkgs, ... }:
{
environment.systemPackages = with pkgs; [
httpie
dnsutils # for `dig`
openssl
cacert
wget
python3
python3Packages.selenium
firefox-unwrapped
geckodriver
];
security.pki.certificateFiles = [
config.nodes.acme.test-support.acme.caCert
];
networking.extraHosts = "${config.acmeNodeIP} acme.test";
};
## NOTE: The target machines may need more RAM than the default to handle
## being deployed to, otherwise we get something like:
##
## pixelfed # [ 616.785499 ] sshd-session[1167]: Conection closed by 2001:db8:1::2 port 45004
## deployer # error: writing to file: No space left on device
## pixelfed # [ 616.788538 ] sshd-session[1151]: pam_unix(sshd:session): session closed for user port
## pixelfed # [ 616.793929 ] systemd-logind[719]: Session 4 logged out. Waiting for processes to exit.
## deployer # Error: Could not create resource
##
## These values have been trimmed down to the gigabyte.
nodes.mastodon.virtualisation.memorySize = 4 * 1024;
nodes.pixelfed.virtualisation.memorySize = 4 * 1024;
nodes.peertube.virtualisation.memorySize = 5 * 1024;
## FIXME: The test of presence of the services are very simple: we only
## check that there is a systemd service of the expected name on the
## machine. This proves at least that NixOps4 did something, and we cannot
## really do more for now because the services aren't actually working
## properly, in particular because of DNS issues. We should fix the services
## and check that they are working properly.
extraTestScript = ''
## TODO: We want a nicer way to control where the FediPanel consumes its
## flake, which can default to the store but could also be somewhere else if
## someone wanted to change the code of the flake.
##
with subtest("Give the panel access to the flake"):
deployer.succeed("mkdir /run/fedipanel /run/fedipanel/flake >&2")
deployer.succeed("cp -R . /run/fedipanel/flake >&2")
deployer.succeed("chown -R panel:panel /run/fedipanel >&2")
## TODO: I want a programmatic way to provide an SSH key to the panel (and
## therefore NixOps4). This should happen either in the Python code, but
## maybe it is fair that that one picks up on the user's key? It could
## also be in the Nix packaging.
##
with subtest("Set up the panel's SSH keys"):
deployer.succeed("mkdir /home/panel/.ssh >&2")
deployer.succeed("cp -R /root/.ssh/* /home/panel/.ssh >&2")
deployer.succeed("chown -R panel:panel /home/panel/.ssh >&2")
deployer.succeed("chmod 600 /home/panel/.ssh/* >&2")
## TODO: This is a hack to accept the root CA used by Pebble on the client
## machine. Pebble randomizes everything, so the only way to get it is to
## call the /roots/0 endpoint at runtime, leaving not much margin for a nice
## Nixy way of adding the certificate. There is no way around it as this is
## by design in Pebble, showing in fact that Pebble was not the appropriate
## tool for our use and that nixpkgs does not in fact provide an easy way to
## generate _usable_ certificates in NixOS tests. I suggest we merge this,
## and track the task to set it up in a cleaner way. I would tackle this in
## a subsequent PR, and hopefully even contribute this BetterWay(tm) to
## nixpkgs. — Niols
##
with subtest("Set up ACME root CA on client"):
client.succeed("""
cd /etc/ssl/certs
curl -o pebble-root-ca.pem https://acme.test:15000/roots/0
curl -o pebble-intermediate-ca.pem https://acme.test:15000/intermediates/0
{ cat ca-bundle.crt
cat pebble-root-ca.pem
cat pebble-intermediate-ca.pem
} > new-ca-bundle.crt
rm ca-bundle.crt ca-certificates.crt
mv new-ca-bundle.crt ca-bundle.crt
ln -s ca-bundle.crt ca-certificates.crt
""")
## TODO: I would hope for a more declarative way to add users. This should
## be handled by the Nix packaging of the FediPanel. — Niols
##
with subtest("Create panel user"):
deployer.succeed("""
expect -c '
spawn manage createsuperuser --username ${panelUser} --email ${panelEmail}
expect "Password: "; send "${panelPassword}\\n";
expect "Password (again): "; send "${panelPassword}\\n"
interact
' >&2
""")
with subtest("Check the status of the services - there should be none"):
garage.fail("systemctl status garage.service")
mastodon.fail("systemctl status mastodon-web.service")
peertube.fail("systemctl status peertube.service")
pixelfed.fail("systemctl status phpfpm-pixelfed.service")
with subtest("Run deployment with no services enabled"):
client.succeed("${
interactWithPanel {
baseUri = "https://deployer";
enableMastodon = false;
enablePeertube = false;
enablePixelfed = false;
}
}/bin/interact-with-panel >&2")
with subtest("Check the status of the services - there should still be none"):
garage.fail("systemctl status garage.service")
mastodon.fail("systemctl status mastodon-web.service")
peertube.fail("systemctl status peertube.service")
pixelfed.fail("systemctl status phpfpm-pixelfed.service")
with subtest("Run deployment with Mastodon and Pixelfed enabled"):
client.succeed("${
interactWithPanel {
baseUri = "https://deployer";
enableMastodon = true;
enablePeertube = false;
enablePixelfed = true;
}
}/bin/interact-with-panel >&2")
with subtest("Check the status of the services - expecting Garage, Mastodon and Pixelfed"):
garage.succeed("systemctl status garage.service")
mastodon.succeed("systemctl status mastodon-web.service")
peertube.fail("systemctl status peertube.service")
pixelfed.succeed("systemctl status phpfpm-pixelfed.service")
with subtest("Run deployment with only Peertube enabled"):
client.succeed("${
interactWithPanel {
baseUri = "https://deployer";
enableMastodon = false;
enablePeertube = true;
enablePixelfed = false;
}
}/bin/interact-with-panel >&2")
with subtest("Check the status of the services - expecting Garage and Peertube"):
garage.succeed("systemctl status garage.service")
mastodon.fail("systemctl status mastodon-web.service")
peertube.succeed("systemctl status peertube.service")
pixelfed.fail("systemctl status phpfpm-pixelfed.service")
'';
}

View file

@ -0,0 +1,70 @@
let
inherit (import ../default.nix { }) pkgs inputs;
inherit (pkgs) lib;
inherit (lib) mkOption;
eval =
module:
(lib.evalModules {
specialArgs = {
inherit inputs;
};
modules = [
module
./data-model.nix
];
}).config;
in
{
_class = "nix-unit";
test-eval = {
expr =
let
fediversity = eval (
{ config, ... }:
{
config = {
applications.hello =
{ ... }:
{
description = ''Command-line tool that will print "Hello, world!" on the terminal'';
module =
{ ... }:
{
options = {
enable = lib.mkEnableOption "Hello in the shell";
};
};
implementation =
cfg:
lib.optionalAttrs cfg.enable {
dummy.login-shell.packages.hello = pkgs.hello;
};
};
};
options = {
example-configuration = mkOption {
type = config.configuration;
readOnly = true;
default = {
enable = true;
applications.hello.enable = true;
};
};
};
}
);
in
{
inherit (fediversity)
example-configuration
;
};
expected = {
example-configuration = {
enable = true;
applications.hello.enable = true;
};
};
};
}

89
deployment/data-model.nix Normal file
View file

@ -0,0 +1,89 @@
{
lib,
config,
...
}:
let
inherit (lib) mkOption types;
inherit (lib.types)
attrsOf
attrTag
deferredModuleWith
submodule
optionType
functionTo
;
functionType = import ./function.nix;
application-resources = {
options.resources = mkOption {
# TODO: maybe transpose, and group the resources by type instead
type = attrsOf (
attrTag (lib.mapAttrs (_name: resource: mkOption { type = resource.request; }) config.resources)
);
};
};
in
{
_class = "nixops4Deployment";
options = {
applications = mkOption {
description = "Collection of Fediversity applications";
type = attrsOf (
submodule (application: {
_class = "fediversity-application";
options = {
description = mkOption {
description = "Description to be shown in the application overview";
type = types.str;
};
module = mkOption {
description = "Operator-facing configuration options for the application";
type = deferredModuleWith { staticModules = [ { _class = "fediversity-application-config"; } ]; };
};
implementation = mkOption {
description = "Mapping of application configuration to deployment resources, a description of what an application needs to run";
type = application.config.config-mapping.function-type;
};
resources = mkOption {
description = "Compute resources required by an application";
type = functionTo application.config.config-mapping.output-type;
readOnly = true;
default = input: (application.config.implementation input).output;
};
config-mapping = mkOption {
description = "Function type for the mapping from application configuration to required resources";
type = submodule functionType;
readOnly = true;
default = {
input-type = application.config.module;
output-type = application-resources;
};
};
};
})
);
};
configuration = mkOption {
description = "Configuration type declaring options to be set by operators";
type = optionType;
readOnly = true;
default = submodule {
options = {
enable = lib.mkEnableOption {
description = "your Fediversity configuration";
};
applications = lib.mapAttrs (
_name: application:
mkOption {
description = application.description;
type = submodule application.module;
default = { };
}
) config.applications;
};
};
};
};
}

View file

@ -65,6 +65,8 @@ let
cfg = config.deployment;
in
{
_class = "nixops4Deployment";
options = {
deployment = lib.mkOption {
description = ''

View file

@ -1,6 +1,26 @@
{ inputs, sources, ... }:
{
imports = [
./check/basic/flake-part.nix
./check/cli/flake-part.nix
];
_class = "flake";
perSystem =
{ pkgs, ... }:
{
checks = {
deployment-basic = import ./check/basic {
inherit (pkgs.testers) runNixOSTest;
inherit inputs sources;
};
deployment-cli = import ./check/cli {
inherit (pkgs.testers) runNixOSTest;
inherit inputs sources;
};
deployment-panel = import ./check/panel {
inherit (pkgs.testers) runNixOSTest;
inherit inputs sources;
};
};
};
}

37
deployment/function.nix Normal file
View file

@ -0,0 +1,37 @@
/**
Modular function type
*/
{ config, lib, ... }:
let
inherit (lib) mkOption types;
inherit (types)
deferredModule
submodule
functionTo
optionType
;
in
{
options = {
input-type = mkOption {
type = deferredModule;
};
output-type = mkOption {
type = deferredModule;
};
function-type = mkOption {
type = optionType;
readOnly = true;
default = functionTo (submodule {
options = {
input = mkOption {
type = submodule config.input-type;
};
output = mkOption {
type = submodule config.output-type;
};
};
});
};
};
}

View file

@ -17,6 +17,8 @@ let
inherit (lib) types mkOption;
in
{
_class = "nixops4Deployment";
options = {
enable = lib.mkEnableOption "Fediversity configuration";
domain = mkOption {

140
flake.lock generated
View file

@ -59,22 +59,6 @@
}
},
"flake-compat_2": {
"flake": false,
"locked": {
"lastModified": 1696426674,
"narHash": "sha256-kvjfFW7WAETZlt09AgDn1MrtKzP7t90Vf7vypd3OL1U=",
"owner": "edolstra",
"repo": "flake-compat",
"rev": "0f9255e01c2351cc7d116c072cb317785dd33b33",
"type": "github"
},
"original": {
"owner": "edolstra",
"repo": "flake-compat",
"type": "github"
}
},
"flake-compat_3": {
"flake": false,
"locked": {
"lastModified": 1733328505,
@ -90,7 +74,7 @@
"type": "github"
}
},
"flake-compat_4": {
"flake-compat_3": {
"flake": false,
"locked": {
"lastModified": 1696426674,
@ -143,24 +127,6 @@
}
},
"flake-parts_3": {
"inputs": {
"nixpkgs-lib": "nixpkgs-lib_3"
},
"locked": {
"lastModified": 1738453229,
"narHash": "sha256-7H9XgNiGLKN1G1CgRh0vUL4AheZSYzPm+zmZ7vxbJdo=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "32ea77a06711b758da0ad9bd6a844c5740a87abd",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"flake-parts_4": {
"inputs": {
"nixpkgs-lib": [
"nixops4-nixos",
@ -201,32 +167,12 @@
"type": "github"
}
},
"git-hooks": {
"git-hooks-nix": {
"inputs": {
"flake-compat": "flake-compat",
"gitignore": "gitignore",
"nixpkgs": "nixpkgs"
},
"locked": {
"lastModified": 1742649964,
"narHash": "sha256-DwOTp7nvfi8mRfuL1escHDXabVXFGT1VlPD1JHrtrco=",
"owner": "cachix",
"repo": "git-hooks.nix",
"rev": "dcf5072734cb576d2b0c59b2ac44f5050b5eac82",
"type": "github"
},
"original": {
"owner": "cachix",
"repo": "git-hooks.nix",
"type": "github"
}
},
"git-hooks-nix": {
"inputs": {
"flake-compat": "flake-compat_2",
"gitignore": "gitignore_2",
"nixpkgs": "nixpkgs_2"
},
"locked": {
"lastModified": 1737465171,
"narHash": "sha256-R10v2hoJRLq8jcL4syVFag7nIGE7m13qO48wRIukWNg=",
@ -281,27 +227,6 @@
}
},
"gitignore": {
"inputs": {
"nixpkgs": [
"git-hooks",
"nixpkgs"
]
},
"locked": {
"lastModified": 1709087332,
"narHash": "sha256-HG2cCnktfHsKV0s4XW83gU3F57gaTljL9KNSuG6bnQs=",
"owner": "hercules-ci",
"repo": "gitignore.nix",
"rev": "637db329424fd7e46cf4185293b9cc8c88c95394",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "gitignore.nix",
"type": "github"
}
},
"gitignore_2": {
"inputs": {
"nixpkgs": [
"nixops4-nixos",
@ -341,8 +266,8 @@
},
"nix": {
"inputs": {
"flake-compat": "flake-compat_3",
"flake-parts": "flake-parts_4",
"flake-compat": "flake-compat_2",
"flake-parts": "flake-parts_3",
"git-hooks-nix": "git-hooks-nix_2",
"nixfmt": "nixfmt",
"nixpkgs": [
@ -416,10 +341,10 @@
},
"nixops4": {
"inputs": {
"flake-parts": "flake-parts_3",
"flake-parts": "flake-parts_2",
"nix": "nix",
"nix-cargo-integration": "nix-cargo-integration",
"nixpkgs": "nixpkgs_3",
"nixpkgs": "nixpkgs_2",
"nixpkgs-old": "nixpkgs-old"
},
"locked": {
@ -438,7 +363,7 @@
},
"nixops4-nixos": {
"inputs": {
"flake-parts": "flake-parts_2",
"flake-parts": "flake-parts",
"git-hooks-nix": "git-hooks-nix",
"nixops4": "nixops4",
"nixops4-nixos": [
@ -520,18 +445,6 @@
"url": "https://github.com/NixOS/nixpkgs/archive/072a6db25e947df2f31aab9eccd0ab75d5b2da11.tar.gz"
}
},
"nixpkgs-lib_3": {
"locked": {
"lastModified": 1738452942,
"narHash": "sha256-vJzFZGaCpnmo7I6i416HaBLpC+hvcURh/BQwROcGIp8=",
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/072a6db25e947df2f31aab9eccd0ab75d5b2da11.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/072a6db25e947df2f31aab9eccd0ab75d5b2da11.tar.gz"
}
},
"nixpkgs-old": {
"locked": {
"lastModified": 1735563628,
@ -565,22 +478,6 @@
}
},
"nixpkgs_2": {
"locked": {
"lastModified": 1730768919,
"narHash": "sha256-8AKquNnnSaJRXZxc5YmF/WfmxiHX6MMZZasRP6RRQkE=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "a04d33c0c3f1a59a2c1cb0c6e34cd24500e5a1dc",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixpkgs-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"nixpkgs_3": {
"locked": {
"lastModified": 1738410390,
"narHash": "sha256-xvTo0Aw0+veek7hvEVLzErmJyQkEcRk6PSR4zsRQFEc=",
@ -596,22 +493,6 @@
"type": "github"
}
},
"nixpkgs_4": {
"locked": {
"lastModified": 1740463929,
"narHash": "sha256-4Xhu/3aUdCKeLfdteEHMegx5ooKQvwPHNkOgNCXQrvc=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "5d7db4668d7a0c6cc5fc8cf6ef33b008b2b1ed8b",
"type": "github"
},
"original": {
"owner": "nixos",
"ref": "nixos-24.11",
"repo": "nixpkgs",
"type": "github"
}
},
"parts": {
"inputs": {
"nixpkgs-lib": [
@ -637,7 +518,7 @@
},
"purescript-overlay": {
"inputs": {
"flake-compat": "flake-compat_4",
"flake-compat": "flake-compat_3",
"nixpkgs": [
"nixops4-nixos",
"nixops4",
@ -680,14 +561,11 @@
},
"root": {
"inputs": {
"flake-parts": "flake-parts",
"git-hooks": "git-hooks",
"nixops4": [
"nixops4-nixos",
"nixops4"
],
"nixops4-nixos": "nixops4-nixos",
"nixpkgs": "nixpkgs_4"
"nixops4-nixos": "nixops4-nixos"
}
},
"rust-overlay": {

View file

@ -1,72 +1,49 @@
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-24.11"; # consumed by flake-parts
flake-parts.url = "github:hercules-ci/flake-parts";
git-hooks.url = "github:cachix/git-hooks.nix";
nixops4.follows = "nixops4-nixos/nixops4";
nixops4-nixos.url = "github:nixops4/nixops4-nixos";
};
outputs =
inputs@{ flake-parts, ... }:
let
sources = import ./npins;
inherit (sources) git-hooks agenix;
in
flake-parts.lib.mkFlake { inherit inputs; } {
systems = [
"x86_64-linux"
"aarch64-linux"
"x86_64-darwin"
"aarch64-darwin"
];
inputs:
import ./mkFlake.nix inputs (
{ inputs, sources, ... }:
{
imports = [
"${sources.git-hooks}/flake-module.nix"
inputs.nixops4.modules.flake.default
imports = [
(import "${git-hooks}/flake-module.nix")
inputs.nixops4.modules.flake.default
./deployment/flake-part.nix
./infra/flake-part.nix
./keys/flake-part.nix
./secrets/flake-part.nix
./secrets/flake-part.nix
./services/tests/flake-part.nix
];
./deployment/flake-part.nix
./infra/flake-part.nix
];
perSystem =
{
pkgs,
lib,
...
}:
{
formatter = pkgs.nixfmt-rfc-style;
perSystem =
{
pkgs,
lib,
inputs',
...
}:
{
formatter = pkgs.nixfmt-rfc-style;
pre-commit.settings.hooks =
let
## Add a directory here if pre-commit hooks shouldn't apply to it.
optout = [ "npins" ];
excludes = map (dir: "^${dir}/") optout;
addExcludes = lib.mapAttrs (_: c: c // { inherit excludes; });
in
addExcludes {
nixfmt-rfc-style.enable = true;
deadnix.enable = true;
trim-trailing-whitespace.enable = true;
shellcheck.enable = true;
};
devShells.default = pkgs.mkShell {
packages = [
pkgs.npins
pkgs.nil
(pkgs.callPackage "${agenix}/pkgs/agenix.nix" { })
pkgs.openssh
pkgs.httpie
pkgs.jq
# exposing this env var as a hack to pass info in from form
(inputs'.nixops4.packages.default.overrideAttrs {
impureEnvVars = [ "DEPLOYMENT" ];
})
];
pre-commit.settings.hooks =
let
## Add a directory here if pre-commit hooks shouldn't apply to it.
optout = [ "npins" ];
excludes = map (dir: "^${dir}/") optout;
addExcludes = lib.mapAttrs (_: c: c // { inherit excludes; });
in
addExcludes {
nixfmt-rfc-style.enable = true;
deadnix.enable = true;
trim-trailing-whitespace.enable = true;
shellcheck.enable = true;
};
};
};
};
}
);
}

View file

@ -14,7 +14,7 @@ everything will become much cleaner.
above 100. For instance, `fedi117`.
2. Add a basic configuration for the machine. These typically go in
`infra/machines/<name>/default.nix`. You can look at other `fediXXX` VMs to
`machines/dev/<name>/default.nix`. You can look at other `fediXXX` VMs to
find inspiration. You probably do not need a `nixos.module` option at this
point.
@ -48,7 +48,7 @@ everything will become much cleaner.
7. Regenerate the list of machines:
```
sh infra/machines.md.sh
sh machines/machines.md.sh
```
Commit it with the machine's configuration, public key, etc.

View file

@ -5,8 +5,9 @@ let
in
{
_class = "nixos";
imports = [
./hardware.nix
./networking.nix
./users.nix
];
@ -22,4 +23,9 @@ in
nix.extraOptions = ''
experimental-features = nix-command flakes
'';
boot.loader = {
systemd-boot.enable = true;
efi.canTouchEfiVariables = true;
};
}

View file

@ -1,63 +1,64 @@
{ config, lib, ... }:
let
inherit (lib) mkDefault;
inherit (lib) mkDefault mkIf mkMerge;
in
{
_class = "nixos";
config = {
services.openssh = {
enable = true;
settings.PasswordAuthentication = false;
};
networking = {
hostName = config.fediversityVm.name;
domain = config.fediversityVm.domain;
networking = mkMerge [
{
hostName = config.fediversityVm.name;
domain = config.fediversityVm.domain;
## REVIEW: Do we actually need that, considering that we have static IPs?
useDHCP = mkDefault true;
## REVIEW: Do we actually need that, considering that we have static IPs?
useDHCP = mkDefault true;
interfaces = {
eth0 = {
ipv4 = {
addresses = [
{
inherit (config.fediversityVm.ipv4) address prefixLength;
}
];
};
ipv6 = {
addresses = [
{
inherit (config.fediversityVm.ipv6) address prefixLength;
}
];
};
## Disable the default firewall and use nftables instead, with a custom
## Procolix-made ruleset.
firewall.enable = false;
nftables = {
enable = true;
rulesetFile = ./nftables-ruleset.nft;
};
};
}
defaultGateway = {
address = config.fediversityVm.ipv4.gateway;
interface = "eth0";
};
defaultGateway6 = {
address = config.fediversityVm.ipv6.gateway;
interface = "eth0";
};
## IPv4
(mkIf config.fediversityVm.ipv4.enable {
interfaces.${config.fediversityVm.ipv4.interface}.ipv4.addresses = [
{ inherit (config.fediversityVm.ipv4) address prefixLength; }
];
defaultGateway = {
address = config.fediversityVm.ipv4.gateway;
interface = config.fediversityVm.ipv4.interface;
};
nameservers = [
"95.215.185.6"
"95.215.185.7"
];
})
nameservers = [
"95.215.185.6"
"95.215.185.7"
"2a00:51c0::5fd7:b906"
"2a00:51c0::5fd7:b907"
];
firewall.enable = false;
nftables = {
enable = true;
rulesetFile = ./nftables-ruleset.nft;
};
};
## IPv6
(mkIf config.fediversityVm.ipv6.enable {
interfaces.${config.fediversityVm.ipv6.interface}.ipv6.addresses = [
{ inherit (config.fediversityVm.ipv6) address prefixLength; }
];
defaultGateway6 = {
address = config.fediversityVm.ipv6.gateway;
interface = config.fediversityVm.ipv6.interface;
};
nameservers = [
"2a00:51c0::5fd7:b906"
"2a00:51c0::5fd7:b907"
];
})
];
};
}

View file

@ -1,4 +1,6 @@
{
_class = "nixos";
users.users = {
procolix = {
isNormalUser = true;

View file

@ -6,6 +6,8 @@ let
in
{
# `config` not set and imported from multiple places: no fixed module class
options.fediversityVm = {
##########################################################################
@ -89,6 +91,17 @@ in
};
ipv4 = {
enable = mkOption {
default = true;
};
interface = mkOption {
description = ''
The interface that carries the machine's IPv4 network.
'';
default = "eth0";
};
address = mkOption {
description = ''
The IP address of the machine, version 4. It will be injected as a
@ -114,6 +127,17 @@ in
};
ipv6 = {
enable = mkOption {
default = true;
};
interface = mkOption {
description = ''
The interface that carries the machine's IPv6 network.
'';
default = "eth0";
};
address = mkOption {
description = ''
The IP address of the machine, version 6. It will be injected as a

View file

@ -1,20 +1,27 @@
{ modulesPath, ... }:
let
# pulling this in manually over from module args resolves an infinite recursion.
# FIXME: instead untangle `//infra/flake-part.nix` and make it stop passing wild functions.
# move moving towards a portable-services-like pattern where some things are submodules.
# Right now those wild functions are for parameterising a bunch of things,
# and the modular way to do that would be options --
# obviously you can't use those for `imports`,
# so one way to decouple fixpoints is to isolate them into submodules.
# Therefore one approach would be to try to go down the call graph,
# and see where what's currently a function could be a `submodule` field of something else.
sources = import ../../npins;
in
{
imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
_class = "nixos";
imports = [
"${sources.nixpkgs}/nixos/modules/profiles/qemu-guest.nix"
];
boot = {
loader = {
systemd-boot.enable = true;
efi.canTouchEfiVariables = true;
};
initrd = {
availableKernelModules = [
"ata_piix"
"uhci_hcd"
"virtio_pci"
"virtio_scsi"
"sd_mod"
"sr_mod"
];

View file

@ -1,6 +1,10 @@
{
inputs,
lib,
config,
sources,
keys,
secrets,
...
}:
@ -8,15 +12,11 @@ let
inherit (lib) attrValues elem mkDefault;
inherit (lib.attrsets) concatMapAttrs optionalAttrs;
inherit (lib.strings) removeSuffix;
sources = import ../../npins;
inherit (sources) nixpkgs agenix disko;
secretsPrefix = ../../secrets;
secrets = import (secretsPrefix + "/secrets.nix");
keys = import ../../keys;
in
{
_class = "nixops4Resource";
imports = [ ./options.nix ];
fediversityVm.hostPublicKey = mkDefault keys.systems.${config.fediversityVm.name};
@ -26,15 +26,15 @@ in
hostPublicKey = config.fediversityVm.hostPublicKey;
};
inherit nixpkgs;
inherit (inputs) nixpkgs;
## The configuration of the machine. We strive to keep in this file only the
## options that really need to be injected from the resource. Everything else
## should go into the `./nixos` subdirectory.
nixos.module = {
imports = [
(import "${agenix}/modules/age.nix")
(import "${disko}/module.nix")
"${sources.agenix}/modules/age.nix"
"${sources.disko}/module.nix"
./options.nix
./nixos
];
@ -43,15 +43,15 @@ in
## configuration.
fediversityVm = config.fediversityVm;
## Read all the secrets, filter the ones that are supposed to be readable
## with this host's public key, and add them correctly to the configuration
## as `age.secrets.<name>.file`.
## Read all the secrets, filter the ones that are supposed to be readable with
## public key, and create a mapping from `<name>.file` to the absolute path of
## the secret's file.
age.secrets = concatMapAttrs (
name: secret:
optionalAttrs (elem config.fediversityVm.hostPublicKey secret.publicKeys) ({
${removeSuffix ".age" name}.file = secretsPrefix + "/${name}";
})
) secrets;
optionalAttrs (elem config.fediversityVm.hostPublicKey secret.publicKeys) {
${removeSuffix ".age" name}.file = secrets.rootPath + "/${name}";
}
) secrets.mapping;
## FIXME: Remove direct root authentication once the NixOps4 NixOS provider
## supports users with password-less sudo.

View file

@ -1,6 +1,9 @@
{
inputs,
lib,
sources,
keys,
secrets,
...
}:
@ -13,7 +16,6 @@ let
filterAttrs
;
inherit (lib.attrsets) genAttrs;
sources = import ../../npins;
## Given a machine's name and whether it is a test VM, make a resource module,
## except for its missing provider. (Depending on the use of that resource, we
@ -21,6 +23,20 @@ let
makeResourceModule =
{ vmName, isTestVm }:
{
# TODO(@fricklerhandwerk): this is terrible but IMO we should just ditch flake-parts and have our own data model for how the project is organised internally
_module.args = {
inherit
inputs
sources
keys
secrets
;
};
nixos.module.imports = [
./common/proxmox-qemu-vm.nix
];
imports =
[
./common/resource.nix
@ -28,17 +44,17 @@ let
++ (
if isTestVm then
[
./test-machines/${vmName}
../machines/operator/${vmName}
{
nixos.module.users.users.root.openssh.authorizedKeys.keys = [
# allow our panel vm access to the test machines
(import ../keys).panel
keys.panel
];
}
]
else
[
./machines/${vmName}
../machines/dev/${vmName}
]
);
fediversityVm.name = vmName;
@ -147,11 +163,13 @@ let
listSubdirectories = path: attrNames (filterAttrs (_: type: type == "directory") (readDir path));
machines = listSubdirectories ./machines;
testMachines = listSubdirectories ./test-machines;
machines = listSubdirectories ../machines/dev;
testMachines = listSubdirectories ../machines/operator;
in
{
_class = "flake";
## - Each normal or test machine gets a NixOS configuration.
## - Each normal or test machine gets a VM options entry.
## - Each normal machine gets a deployment.

View file

@ -15,7 +15,6 @@ let
installer =
{
config,
pkgs,
lib,
...

1
keys/cd-ssh-key.pub Normal file
View file

@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMlsYTtMx3hFO8B5B8iHaXL2JKj9izHeC+/AMhIWXBPs cd-age

View file

@ -35,4 +35,5 @@ in
contributors = collectKeys ./contributors;
systems = collectKeys ./systems;
panel = removeTrailingWhitespace (readFile ./panel-ssh-key.pub);
cd = removeTrailingWhitespace (readFile ./cd-ssh-key.pub);
}

5
keys/flake-part.nix Normal file
View file

@ -0,0 +1,5 @@
{
_class = "flake";
_module.args.keys = import ./.;
}

View file

@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIFXQW5fxJoNY9wtTMsNExgbAbvyljIRGBLjY+USh/0A

4
machines/README.md Normal file
View file

@ -0,0 +1,4 @@
# Machines
This directory contains the definition of [the VMs](machines.md) that host our
infrastructure.

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 200;
proxmox = "fediversity";
@ -14,4 +16,10 @@
gateway = "2a00:51c0:13:1305::1";
};
};
nixos.module = {
imports = [
../../../infra/common/proxmox-qemu-vm.nix
];
};
}

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 201;
proxmox = "fediversity";
@ -17,6 +19,7 @@
nixos.module = {
imports = [
../../../infra/common/proxmox-qemu-vm.nix
./fedipanel.nix
];
};

View file

@ -4,10 +4,14 @@
}:
let
name = "panel";
sources = import ../../../npins;
in
{
_class = "nixos";
imports = [
(import ../../../panel { }).module
(import "${sources.home-manager}/nixos")
];
security.acme = {

View file

@ -0,0 +1,70 @@
{ lib, ... }:
let
inherit (lib) mkDefault mkForce;
in
{
_class = "nixops4Resource";
# NOTE: This needs an SSH config entry `forgejo-ci` to locate and access the
# machine. This is because different people access the machine in different
# way (eg. via a proxy vs. via Procolix's VPN). This might look like:
#
# Host forgejo-ci
# HostName 45.142.234.216
# HostKeyAlias forgejo-ci
#
# The `HostKeyAlias` statement is crucial. Without it, deployment will fail
# with the SSH error “Host key verification failed”.
ssh.host = mkForce "forgejo-ci";
fediversityVm = {
domain = "procolix.com";
ipv4 = {
interface = "enp1s0f0";
address = "192.168.201.65";
prefixLength = 24;
gateway = "192.168.201.1";
};
ipv6.enable = false;
};
nixos.module =
{ config, ... }:
{
_class = "nixos";
imports = [
./forgejo-actions-runner.nix
];
hardware.cpu.intel.updateMicrocode = mkDefault config.hardware.enableRedistributableFirmware;
networking = {
nftables.enable = mkForce false;
hostId = "1d6ea552";
};
## NOTE: This is a physical machine, so is not covered by disko
fileSystems."/" = lib.mkForce {
device = "rpool/root";
fsType = "zfs";
};
fileSystems."/home" = {
device = "rpool/home";
fsType = "zfs";
};
fileSystems."/boot" = lib.mkForce {
device = "/dev/disk/by-uuid/50B2-DD3F";
fsType = "vfat";
options = [
"fmask=0077"
"dmask=0077"
];
};
};
}

View file

@ -0,0 +1,47 @@
{ pkgs, config, ... }:
{
_class = "nixos";
services.gitea-actions-runner = {
package = pkgs.forgejo-actions-runner;
instances.default = {
enable = true;
name = config.networking.fqdn;
url = "https://git.fediversity.eu";
tokenFile = config.age.secrets.forgejo-runner-token.path;
settings = {
log.level = "info";
runner = {
file = ".runner";
# Take only 1 job at a time to avoid clashing NixOS tests, see #362
capacity = 1;
timeout = "3h";
insecure = false;
fetch_timeout = "5s";
fetch_interval = "2s";
};
};
## This runner supports Docker (with a default Ubuntu image) and native
## modes. In native mode, it contains a few default packages.
labels = [
"docker:docker://node:16-bullseye"
"native:host"
];
hostPackages = with pkgs; [
bash
git
nix
nodejs
];
};
};
## For the Docker mode of the runner.
virtualisation.docker.enable = true;
}

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 2116;
proxmox = "procolix";
@ -12,6 +14,7 @@
{ lib, ... }:
{
imports = [
../../../infra/common/proxmox-qemu-vm.nix
./forgejo.nix
];

View file

@ -5,6 +5,8 @@ let
in
{
_class = "nixos";
services.forgejo = {
enable = true;

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 2187;
proxmox = "procolix";
@ -12,6 +14,7 @@
{ lib, ... }:
{
imports = [
../../../infra/common/proxmox-qemu-vm.nix
./wiki.nix
];

View file

@ -1,6 +1,8 @@
{ config, ... }:
{
_class = "nixos";
services.phpfpm.pools.mediawiki.phpOptions = ''
upload_max_filesize = 1024M;
post_max_size = 1024M;

View file

@ -7,9 +7,10 @@ Currently, this repository keeps track of the following VMs:
Machine | Proxmox | Description
--------|---------|-------------
[`fedi200`](./fedi200) | fediversity | Testing machine for Hans
[`fedi201`](./fedi201) | fediversity | FediPanel
[`vm02116`](./vm02116) | procolix | Forgejo
[`vm02187`](./vm02187) | procolix | Wiki
[`fedi200`](./dev/fedi200) | fediversity | Testing machine for Hans
[`fedi201`](./dev/fedi201) | fediversity | FediPanel
[`vm02116`](./dev/vm02116) | procolix | Forgejo
[`vm02187`](./dev/vm02187) | procolix | Wiki
| `forgejo-ci` | n/a (physical) | Forgejo actions runner |
This table excludes all machines with names starting with `test`.

View file

@ -20,7 +20,7 @@ vmOptions=$(
cd ..
nix eval \
--impure --raw --expr "
builtins.toJSON (builtins.getFlake (builtins.toString ./.)).vmOptions
builtins.toJSON (builtins.getFlake (builtins.toString ../.)).vmOptions
" \
--log-format raw --quiet
)
@ -32,11 +32,12 @@ for machine in $(echo "$vmOptions" | jq -r 'keys[]'); do
description=$(echo "$vmOptions" | jq -r ".$machine.description" | head -n 1)
# shellcheck disable=SC2016
printf '[`%s`](./%s) | %s | %s\n' "$machine" "$machine" "$proxmox" "$description"
printf '[`%s`](./dev/%s) | %s | %s\n' "$machine" "$machine" "$proxmox" "$description"
fi
done
cat <<\EOF
| `forgejo-ci` | n/a (physical) | Forgejo actions runner |
This table excludes all machines with names starting with `test`.
EOF

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7001;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7002;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7003;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7004;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7005;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7006;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7011;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7012;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7013;
proxmox = "fediversity";

View file

@ -1,4 +1,6 @@
{
_class = "nixops4Resource";
fediversityVm = {
vmId = 7014;
proxmox = "fediversity";

54
mkFlake.nix Normal file
View file

@ -0,0 +1,54 @@
## This file contains a tweak of flake-parts's `mkFlake` function to splice in
## sources taken from npins.
## NOTE: Much of the logic in this file feels like it should be not super
## specific to fediversity. Could it make sense to extract the core of this to
## another place it feels closer to in spirit, such as @fricklerhandwerk's
## flake-inputs (which this code already depends on anyway, and which already
## contained two distinct helpers for migrating away from flakes)? cf
## https://git.fediversity.eu/Fediversity/Fediversity/pulls/447#issuecomment-8671
inputs@{ self, ... }:
let
sources = import ./npins;
inherit (import sources.flake-inputs) import-flake;
# XXX(@fricklerhandwerk): this atrocity is required to splice in a foreign Nixpkgs via flake-parts
# XXX - this is just importing a flake
nixpkgs = import-flake { src = sources.nixpkgs; };
# XXX - this overrides the inputs attached to `self`
inputs' = self.inputs // {
nixpkgs = nixpkgs;
};
self' = self // {
inputs = inputs';
};
flake-parts-lib = import "${sources.flake-parts}/lib.nix" { inherit (nixpkgs) lib; };
in
flakeModule:
flake-parts-lib.mkFlake
{
# XXX - finally we override the overall set of `inputs` -- we need both:
# `flake-parts obtains `nixpkgs` from `self.inputs` and not from `inputs`.
inputs = inputs // {
inherit nixpkgs;
};
self = self';
specialArgs = {
inherit sources;
};
}
{
systems = [
"x86_64-linux"
"aarch64-linux"
"x86_64-darwin"
"aarch64-darwin"
];
imports = [ flakeModule ];
}

View file

@ -25,6 +25,38 @@
"url": null,
"hash": "1w2gsy6qwxa5abkv8clb435237iifndcxq0s79wihqw11a5yb938"
},
"disko": {
"type": "GitRelease",
"repository": {
"type": "GitHub",
"owner": "nix-community",
"repo": "disko"
},
"pre_releases": false,
"version_upper_bound": null,
"release_prefix": null,
"submodules": false,
"version": "v1.12.0",
"revision": "7121f74b976481bc36877abaf52adab2a178fcbe",
"url": "https://api.github.com/repos/nix-community/disko/tarball/v1.12.0",
"hash": "0wbx518d2x54yn4xh98cgm65wvj0gpy6nia6ra7ns4j63hx14fkq"
},
"flake-inputs": {
"type": "GitRelease",
"repository": {
"type": "GitHub",
"owner": "fricklerhandwerk",
"repo": "flake-inputs"
},
"pre_releases": false,
"version_upper_bound": null,
"release_prefix": null,
"submodules": false,
"version": "4.1",
"revision": "ad02792f7543754569fe2fd3d5787ee00ef40be2",
"url": "https://api.github.com/repos/fricklerhandwerk/flake-inputs/tarball/4.1",
"hash": "1j57avx2mqjnhrsgq3xl7ih8v7bdhz1kj3min6364f486ys048bm"
},
"flake-parts": {
"type": "Git",
"repository": {
@ -64,6 +96,19 @@
"url": "https://github.com/hercules-ci/gitignore.nix/archive/637db329424fd7e46cf4185293b9cc8c88c95394.tar.gz",
"hash": "02wxkdpbhlm3yk5mhkhsp3kwakc16xpmsf2baw57nz1dg459qv8w"
},
"home-manager": {
"type": "Git",
"repository": {
"type": "GitHub",
"owner": "nix-community",
"repo": "home-manager"
},
"branch": "master",
"submodules": false,
"revision": "863842639722dd12ae9e37ca83bcb61a63b36f6c",
"url": "https://github.com/nix-community/home-manager/archive/863842639722dd12ae9e37ca83bcb61a63b36f6c.tar.gz",
"hash": "0rw9n8d4v87pzlmw7ws15f0sldb51fd9528skpbzmrzl4pinsgij"
},
"htmx": {
"type": "GitRelease",
"repository": {

View file

@ -20,8 +20,15 @@ in
packages = [
pkgs.npins
manage
# NixOps4 and its dependencies
pkgs.nixops4
pkgs.nix
pkgs.openssh
];
env = import ./env.nix { inherit lib pkgs; } // {
env = {
DEPLOYMENT_FLAKE = toString ../.;
DEPLOYMENT_NAME = "test";
NPINS_DIRECTORY = toString ../npins;
CREDENTIALS_DIRECTORY = toString ./.credentials;
DATABASE_URL = "sqlite:///${toString ./src}/db.sqlite3";

View file

@ -1,18 +0,0 @@
{
lib,
pkgs,
...
}:
let
inherit (builtins) toString;
in
{
REPO_DIR = toString ../.;
# explicitly use nix, as e.g. lix does not have configurable-impure-env
BIN_PATH = lib.makeBinPath [
# explicitly use nix, as e.g. lix does not have configurable-impure-env
pkgs.nix
# nixops error maybe due to our flake git hook: executing 'git': No such file or directory
pkgs.git
];
}

View file

@ -23,7 +23,9 @@ let
cfg = config.services.${name};
package = pkgs.callPackage ./package.nix { };
environment = import ../env.nix { inherit lib pkgs; } // {
environment = {
DEPLOYMENT_FLAKE = cfg.deployment.flake;
DEPLOYMENT_NAME = cfg.deployment.name;
DATABASE_URL = "sqlite:////var/lib/${name}/db.sqlite3";
USER_SETTINGS_FILE = pkgs.concatText "configuration.py" [
((pkgs.formats.pythonVars { }).generate "settings.py" cfg.settings)
@ -74,6 +76,8 @@ in
# https://git.dgnum.eu/mdebray/djangonix/
# unlicensed at the time of writing, but surely worth taking some inspiration from...
{
_class = "nixos";
options.services.${name} = {
enable = mkEnableOption "Service configuration for `${name}`";
production = mkOption {
@ -133,6 +137,35 @@ in
type = types.attrsOf types.path;
default = { };
};
nixops4Package = mkOption {
type = types.package;
description = ''
A package providing NixOps4.
TODO: This should not be at the level of the NixOS module, but instead
at the level of the panel's package. Until one finds a way to grab
NixOps4 from the package's npins-based code, we will have to do with
this workaround.
'';
default = pkgs.nixops4;
};
deployment = {
flake = mkOption {
type = types.path;
default = ../..;
description = ''
The path to the flake containing the deployment. This is used to run the deployment button.
'';
};
name = mkOption {
type = types.str;
default = "test";
description = ''
The name of the deployment within the flake.
'';
};
};
};
config = mkIf cfg.enable {
@ -169,9 +202,8 @@ in
};
};
users.users.${name} = {
isNormalUser = true;
};
# needed to place a config file with home-manager
users.users.${name}.isNormalUser = true;
users.groups.${name} = { };
systemd.services.${name} = {
@ -181,6 +213,11 @@ in
path = [
python-environment
manage-service
## NixOps4 and its dependencies
cfg.nixops4Package
pkgs.nix
pkgs.openssh
];
preStart = ''
# Auto-migrate on first run or if the package has changed

View file

@ -8,4 +8,17 @@ let
in
{
python3 = prev.lib.attrsets.recursiveUpdate prev.python3 { pkgs = extraPython3Packages; };
nixops4 =
let
sources = import ../../npins;
inherit (import sources.flake-inputs) import-flake;
inherit
(import-flake {
src = ../../.;
})
inputs
;
inherit (inputs) nixops4;
in
nixops4.packages.${prev.system}.default;
}

View file

@ -60,6 +60,8 @@ let
];
in
python3.pkgs.buildPythonPackage {
_class = "package";
pname = name;
inherit (pyproject.project) version;
pyproject = true;

View file

@ -8,6 +8,8 @@
}:
buildPythonPackage rec {
_class = "package";
pname = "django-pydantic-field";
version = "v0.3.12";
pyproject = true;

Some files were not shown because too many files have changed in this diff Show more