Introduce test for deploying all services with nixops4 apply #329

Niols · 2025-05-08T09:57:21+02:00

Niols commented

2025-05-08 09:57:21 +02:00

This PR adds a CLI deployment test. It builds on top of Fediversity/Fediversity#323. This test features a deployer node and four target nodes. The deployer node runs nixops4 apply on a deployment built with our actual code in deployment/default.nix, which pushes onto the four target machines combinations of Garage/Mastodon/Peertube/Pixelfed depending on a JSON payload. We check that the expected services are indeed deployed on the machines. Getting there involved reworking the existing basic test to extract common patterns, and adding support for ACME certificates negotiation inside the NixOS test.

What works:

deployer successfully runs nixops4 apply with various payloads
target machines indeed get the right services pushed onto them and removed
services on target machines successfully negotiate ACME certificates

What does not work: the services themselves depend a lot on DNS and that is not taken care of at all, so they are probably very broken. Still, this is a good milestone.

Future work:

Working on DNS in the test, making sure the services run correctly, actually testing the services (eg. with Selenium scripts etc.)
A deployment of some services (Fediversity? dummy?) triggered from the panel (← this is what I will work on next).
A deployment of some services triggered from Terraform (@kiara is on it, I believe).

Test it yourself by running nix build .#checks.x86_64-linux.deployment-basic -vL and nix build .#checks.x86_64-linux.deployment-cli -vL. On the very beefy machine that I am using, the basic test runs in ~4 minutes and the CLI test in ~17 minutes. We know from Fediversity/Fediversity#323 that the basic test runs in ~12 minutes on the CI runner, so maybe about an hour for the CLI test?

You can check the rendering of the documentation (in particular for Mermaid diagrams) here.

I believe the commits are good as they are and should be fast-forwarded only.

Closes https://git.fediversity.eu/Fediversity/Fediversity/issues/276 This PR adds a CLI deployment test. It builds on top of https://git.fediversity.eu/Fediversity/Fediversity/pulls/323. This test features a deployer node and four target nodes. The deployer node runs `nixops4 apply` on a deployment built with our actual code in `deployment/default.nix`, which pushes onto the four target machines combinations of Garage/Mastodon/Peertube/Pixelfed depending on a JSON payload. We check that the expected services are indeed deployed on the machines. Getting there involved reworking the existing basic test to extract common patterns, and adding support for ACME certificates negotiation inside the NixOS test. What works: - deployer successfully runs `nixops4 apply` with various payloads - target machines indeed get the right services pushed onto them and removed - services on target machines successfully negotiate ACME certificates What does not work: the services themselves depend a lot on DNS and that is not taken care of at all, so they are probably very broken. Still, this is a good milestone. Future work: - Working on DNS in the test, making sure the services run correctly, actually testing the services (eg. with Selenium scripts etc.) - A deployment of some services (Fediversity? dummy?) triggered from the panel (← this is what I will work on next). - A deployment of some services triggered from Terraform (@kiara is on it, I believe). --- Test it yourself by running `nix build .#checks.x86_64-linux.deployment-basic -vL` and `nix build .#checks.x86_64-linux.deployment-cli -vL`. On the very beefy machine that I am using, the basic test runs in ~4 minutes and the CLI test in ~17 minutes. We know from https://git.fediversity.eu/Fediversity/Fediversity/pulls/323 that the basic test runs in ~12 minutes on the CI runner, so maybe about an hour for the CLI test? You can check the rendering of the documentation (in particular for Mermaid diagrams) [here](https://git.fediversity.eu/Niols/Fediversity/src/branch/integration-test-multiple-rebased/deployment/README.md). I believe the commits are good as they are and should be fast-forwarded only.

❤️ 1

Niols added 3 commits 2025-05-08 09:57:22 +02:00

Move sample JSON file closer to deployment 22cedb8f25

Factorise common deployment test parts; improve and document basic test 7b7863fed2

Introduce CLI deployment test

/ check-pre-commit (pull_request) Successful in 11s

Details

/ check-peertube (pull_request) Failing after 10s

Details

/ check-panel (pull_request) Successful in 1m14s

Details

/ check-deployment-basic (pull_request) Successful in 44m58s

Details

/ check-deployment-cli (pull_request) Successful in 44m56s

Details

2bfca55a07

fricklerhandwerk reviewed 2025-05-08 10:08:58 +02:00

deployment/README.md Outdated

					
				@ -7,0 +15,4 @@

				$ nix build .#checks.<system>.deployment-<name> -vL

				```

				## Basic deployment check

fricklerhandwerk commented

2025-05-08 10:08:58 +02:00

### Basic deployment check

`### Basic deployment check`

Niols commented

2025-05-08 15:02:45 +02:00

Done in cc0c5d0519.

Done in cc0c5d05199d0288ffcc0b35ddd91b1430fe2fc3.

Niols marked this conversation as resolved

fricklerhandwerk reviewed 2025-05-08 10:11:47 +02:00

deployment/README.md Outdated

					
				@ -7,0 +14,4 @@

				``` console

				$ nix build .#checks.<system>.deployment-<name> -vL

				```

fricklerhandwerk commented

2025-05-08 10:11:47 +02:00

Maybe worth mentioning here that since nixops4 apply operates on a flake, the tests take this repository's flake as a template, and this also why there are some dummy files that will be overwritten inside the test. Although why are we even keeping the unused files? They could just as well all be created during the test, no?

Maybe worth mentioning here that since `nixops4 apply` operates on a flake, the tests take this repository's flake as a template, and this also why there are some dummy files that will be overwritten inside the test. Although why are we even keeping the unused files? They could just as well all be *created* during the test, no?

Niols commented

2025-05-08 15:03:38 +02:00

Done in cc0c5d0519. I did remove most of the files, but some are still necessary when building the VMs initially. I would like to get rid of them, though. We could also imagine getting rid of flakes but creating a whole flake in the test.

Done in cc0c5d05199d0288ffcc0b35ddd91b1430fe2fc3. I did remove most of the files, but some are still necessary when building the VMs initially. I would like to get rid of them, though. We could also imagine getting rid of flakes but creating a whole flake in the test.

Niols marked this conversation as resolved

fricklerhandwerk reviewed 2025-05-08 10:27:04 +02:00

deployment/check/common/makeNixosTestModule.nix Outdated

					
				@ -0,0 +236,4 @@

				          in

				          ''

				            deployer.copy_from_host("${targetNetworkJSON}", "/root/target-network.json")

				            deployer.succeed("mv /root/target-network.json work/${pathFromRoot}/${tm}-network.json")

fricklerhandwerk commented

2025-05-08 10:27:04 +02:00

Why not copy it to the final location directly?

Niols commented

2025-05-08 15:02:35 +02:00

That's how it was done in the nixops4-nixos; I assumed there was a reason and never checked. Done in cc0c5d0519.

That's how it was done in the nixops4-nixos; I assumed there was a reason and never checked. Done in cc0c5d05199d0288ffcc0b35ddd91b1430fe2fc3.

Niols marked this conversation as resolved

fricklerhandwerk reviewed 2025-05-08 10:34:20 +02:00

deployment/check/common/makeMakeTargetResourceModule.nix Outdated

					
				@ -0,0 +35,4 @@

				        ./minimalTarget.nix

				        (lib.modules.importJSON (pathToRoot + "/${pathFromRoot}/${tm}-network.json"))

				      ]

				      ++ optional enableAcme (makeAcmeClientModule {

fricklerhandwerk commented

2025-05-08 10:34:20 +02:00

It would be a lot simpler and much more idiomatic if you enabled acme right there in the module. Then we wouldn't need the extra file, which makes it harder to track down what's happening.

{
  # ...
  nixos.module = nixos: {
    # ...
    security.acme = {
      acceptTerms = enableAcme;
      # ...
    };
    networking.extraHosts = mkIf nixos.config.security.acme.acceptTerms "${acmeNodeIP acme.test}";
    # ...
  }; 
}

It would be a lot simpler and much more idiomatic if you enabled acme right there in the module. Then we wouldn't need the extra file, which makes it harder to track down what's happening. ```nix { # ... nixos.module = nixos: { # ... security.acme = { acceptTerms = enableAcme; # ... }; networking.extraHosts = mkIf nixos.config.security.acme.acceptTerms "${acmeNodeIP acme.test}"; # ... }; } ```

Niols commented

2025-05-09 10:42:04 +02:00

Done in b03973603e.

Done in b03973603ee41e506430e97a824a2036eccdf6e4.

Niols marked this conversation as resolved

kiara reviewed 2025-05-08 10:42:53 +02:00

deployment/check/cli/nixosTest.nix Outdated

					
				@ -0,0 +92,4 @@

				    extraTestScript = ''

				      with subtest("Run deployment with no services enabled"):

				        deployer.succeed("cd work && nixops4 apply check-deployment-cli-nothing --show-trace --no-interactive 1>&2")

kiara commented

2025-05-08 10:42:53 +02:00

where does this work directory come from?

where does this `work` directory come from?

fricklerhandwerk commented

2025-05-08 10:57:59 +02:00

It's created in the unpacking subtest defined in the test module generator.

It's created in the `unpacking` subtest defined in the [test module generator](https://git.fediversity.eu/Fediversity/Fediversity/pulls/329/files#diff-708976230d3a93d725a457b5a9407dc82ec3710b).

Niols commented

2025-05-08 14:05:52 +02:00

I think this shows a lack of documentation and that's a good point. Maybe we can even unpack the flake in the current working directory such that we don't need to cd work. I have to check whether there are things in the current working directory that would clash with that.

I think this shows a lack of documentation and that's a good point. Maybe we can even unpack the flake in the current working directory such that we don't need to `cd work`. I have to check whether there are things in the current working directory that would clash with that.

Niols commented

2025-05-08 15:02:04 +02:00

Done in cc0c5d0519.

Done in cc0c5d05199d0288ffcc0b35ddd91b1430fe2fc3.

❤️ 1

Niols marked this conversation as resolved

fricklerhandwerk commented

2025-05-08 11:01:19 +02:00

TODOs from verbal sync:

modularise all test parameters, it's the main barrier to readability
try to fold the fake node into a nested derivation (not a blocker)
try to address the minor FIXMEs such as the dangling peertube and gixy dependencies (nice to have)

TODOs from verbal sync: - modularise all test parameters, it's the main barrier to readability - try to fold the `fake` node into a nested derivation (not a blocker) - try to address the minor FIXMEs such as the dangling `peertube` and `gixy` dependencies (nice to have)

👍 1

kiara approved these changes 2025-05-08 11:03:15 +02:00

Niols added 1 commit 2025-05-08 14:33:46 +02:00

Apply suggestions from code review

/ check-pre-commit (pull_request) Successful in 11s

Details

/ check-peertube (pull_request) Failing after 12s

Details

/ check-panel (pull_request) Successful in 1m9s

Details

/ check-deployment-basic (pull_request) Successful in 6s

Details

/ check-deployment-cli (pull_request) Successful in 41m49s

Details

cc0c5d0519

Niols force-pushed integration-test-multiple-rebased from 0c90805339 to cc0c5d0519

2025-05-08 15:01:50 +02:00

Compare

Niols added 2 commits 2025-05-09 08:50:24 +02:00

Re-modularise common stuff b03973603e

Get rid of fake node

/ check-pre-commit (pull_request) Successful in 12s

Details

/ check-peertube (pull_request) Failing after 11s

Details

/ check-panel (pull_request) Successful in 1m13s

Details

/ check-deployment-basic (pull_request) Successful in 43m41s

Details

/ check-deployment-cli (pull_request) Successful in 43m39s

Details

dfa1f8cf9e

Niols commented

2025-05-09 08:52:08 +02:00

Added two commits. The first one re-modularises the common parts of the test. This is a whole reorganisation. The second one gets rid of the fake node concept and makes a temporary NixOS system to achieve the same goal. I am not sure it makes sense to review these commits separately as they really touch everything.

👍 1

Niols commented

2025-05-09 12:44:39 +02:00

TODOs for oral sync:

Type and document all options
Merge common/acme/* into common/*
Make testCerts an option with default import ...

Simplify node.acme

imports = [ "${inputs.nixpkgs}/nixos/tests/common/acme/server" ];
systemd.services.pebble.environment.PEBBLE_VA_ALWAYS_VALID = "1";

TODOs for oral sync: - Type and document all options - Merge `common/acme/*` into `common/*` - Make `testCerts` an option with default `import ...` - Simplify node.acme ``` imports = [ "${inputs.nixpkgs}/nixos/tests/common/acme/server" ]; systemd.services.pebble.environment.PEBBLE_VA_ALWAYS_VALID = "1"; ```

❤️ 1

Niols added 1 commit 2025-05-09 15:51:13 +02:00

Apply suggestions from code review

/ check-pre-commit (pull_request) Successful in 11s

Details

/ check-peertube (pull_request) Failing after 11s

Details

/ check-panel (pull_request) Successful in 1m11s

Details

/ check-deployment-basic (pull_request) Successful in 42m50s

Details

/ check-deployment-cli (pull_request) Successful in 42m48s

Details

1e930fd23e

Fediversity/Fediversity#329 (comment)

Niols commented

2025-05-09 15:51:26 +02:00

@Niols wrote in Fediversity/Fediversity#329 (comment):

TODOs for oral sync:
Type and document all options

Merge common/acme/* into common/*

Make testCerts an option with default import ...
Simplify node.acme
imports = [ "${inputs.nixpkgs}/nixos/tests/common/acme/server" ];
systemd.services.pebble.environment.PEBBLE_VA_ALWAYS_VALID = "1";

All done. What do we think of the current status of things?

@Niols wrote in https://git.fediversity.eu/Fediversity/Fediversity/pulls/329#issuecomment-6532: > TODOs for oral sync: > > * Type and document all options > * Merge `common/acme/*` into `common/*` > * Make `testCerts` an option with default `import ...` > * Simplify node.acme > ```text > imports = [ "${inputs.nixpkgs}/nixos/tests/common/acme/server" ]; > systemd.services.pebble.environment.PEBBLE_VA_ALWAYS_VALID = "1"; > ``` All done. What do we think of the current status of things?

👍 1

Niols commented

2025-05-15 10:35:36 +02:00

With this last commit, CI should be a happy green. Is there anything that remains before we can merge? I only think of things that can be left out and added in another PR, except maybe for the name of this “CLI” deployment test - now would be a good time to find a better name.

👍 1

kiara commented

2025-05-15 11:28:08 +02:00

@Niols i'm good, thanks for finishing this!
@fricklerhandwerk?

@Niols i'm good, thanks for finishing this! @fricklerhandwerk?

fricklerhandwerk changed title from ~~Introduce CLI deployment test~~ to Introduce test for deploying all services with nixops4 apply

2025-05-15 11:35:04 +02:00

fricklerhandwerk reviewed 2025-05-15 11:35:34 +02:00

deployment/README.md Outdated

					
				@ -7,0 +56,4 @@

				### CLI deployment check

				The CLI deployment check factors out the panel by running a direct invokation of

fricklerhandwerk commented

2025-05-15 11:35:34 +02:00

invocation

Niols commented

2025-05-15 12:04:06 +02:00

Done in a328818f1d7c8a5e055b90eb372be82e3fccd252.

Niols marked this conversation as resolved

fricklerhandwerk reviewed 2025-05-15 11:36:20 +02:00

deployment/README.md Outdated

					
				@ -7,0 +54,4 @@

				  deployer -->|deploys| target_machines

				```

				### CLI deployment check

fricklerhandwerk commented

2025-05-15 11:36:20 +02:00

### Service deployment check using `nixops4 apply`

How about this?

```` ### Service deployment check using `nixops4 apply` ```` How about this?

Niols commented

2025-05-15 11:47:46 +02:00

Sounds good. And for the name in the code, deployment-cli -> deployment-nixops4?

Sounds good. And for the name in the code, `deployment-cli` -> `deployment-nixops4`?

Niols commented

2025-05-15 12:04:20 +02:00

Done in a328818f1d7c8a5e055b90eb372be82e3fccd252.

fricklerhandwerk reviewed 2025-05-15 11:37:03 +02:00

deployment/README.md Outdated

					
				@ -7,0 +124,4 @@

				### [WIP] Panel deployment check

				This is a full deployment check running the panel on the deployer machine,

				deploying some services and checking that they are indeed on the target

fricklerhandwerk commented

2025-05-15 11:37:03 +02:00

deploying some services through the panel, and...

Niols commented

2025-05-15 12:04:14 +02:00

Done in a328818f1d7c8a5e055b90eb372be82e3fccd252.

Niols marked this conversation as resolved

fricklerhandwerk commented

2025-05-15 11:38:49 +02:00

This looks pretty good from afar, but I'd like to take another full pass and then merge.

👍 1

Niols force-pushed integration-test-multiple-rebased from a328818f1d to 0b3b252292

2025-05-16 12:21:23 +02:00

Compare

Niols commented

2025-05-16 12:21:47 +02:00

Rebased on recent main.

Rebased on recent `main`.

kiara commented

2025-05-16 13:39:47 +02:00

ci fail :(

Niols commented

2025-05-16 14:30:40 +02:00

Ah dayum :-(

Niols commented

2025-05-16 14:31:12 +02:00

I guess the culprit is the freshly merged Fediversity/Fediversity#330; we should have waited for this test, probably.

I guess the culprit is the freshly merged https://git.fediversity.eu/Fediversity/Fediversity/pulls/330; we should have waited for this test, probably.

Niols force-pushed integration-test-multiple-rebased from 27634f2294 to 9dcd0e360e

2025-05-17 09:12:01 +02:00

Compare

fricklerhandwerk added 1 commit 2025-05-18 19:35:05 +02:00

readme: one line per sentence, small wording fixes

/ check-pre-commit (pull_request) Successful in 11s

Details

/ check-peertube (pull_request) Successful in 18s

Details

/ check-panel (pull_request) Successful in 1m7s

Details

/ check-deployment-basic (pull_request) Successful in 39m40s

Details

/ check-deployment-cli (pull_request) Successful in 39m38s

Details

8d67e49ba1

fricklerhandwerk added 2 commits 2025-05-18 22:20:27 +02:00

reword comment ba8f071a6e

simplify path finding

/ check-pre-commit (pull_request) Successful in 11s

Details

/ check-peertube (pull_request) Successful in 18s

Details

/ check-panel (pull_request) Successful in 1m5s

Details

/ check-deployment-basic (pull_request) Successful in 41m54s

Details

/ check-deployment-cli (pull_request) Successful in 41m52s

Details

76802f96a7

fricklerhandwerk added 2 commits 2025-05-19 01:24:41 +02:00

clarify purpose of magic sharedOptions d225e22d4c

don't use dummy files

/ check-pre-commit (pull_request) Successful in 11s

Details

/ check-peertube (pull_request) Successful in 17s

Details

/ check-panel (pull_request) Successful in 1m8s

Details

/ check-deployment-basic (pull_request) Successful in 42m16s

Details

/ check-deployment-cli (pull_request) Successful in 42m14s

Details

9710c72f58

seems they're not needed after all

fricklerhandwerk approved these changes 2025-05-19 02:17:22 +02:00

fricklerhandwerk left a comment

Pretty good, awesome that it's all green now! Let's merge it like that, and keep cleaning it up. It's still a bit too hard to follow for my taste, because some unnecessary abstraction obscures the actual test cases. Ideally test cases would contain pretty much a (simplified) deployment as one would actually write it for production, and next to it a script that interacts with it to check invariants.

fricklerhandwerk merged commit ee5c2b90b7 into main

2025-05-19 02:18:56 +02:00

fricklerhandwerk referenced this pull request from a commit

2025-05-19 02:18:56 +02:00

Introduce test for deploying all services with `nixops4 apply` (#329)

fricklerhandwerk deleted branch integration-test-multiple-rebased

2025-05-19 02:18:56 +02:00

fricklerhandwerk referenced this pull request

2025-05-19 02:20:18 +02:00

NixOS test of a deployment #276

Niols referenced this pull request

2025-05-21 12:32:13 +02:00