kiara/Fediversity
1
0
Fork 0
forked from Fediversity/Fediversity
Code Pull requests Activity
800 commits 107 branches 0 tags 49 MiB
Commit graph

79 commits

Author SHA1 Message Date
Kiara Grouwstra
1d519dceab
move applications from config to options to actually type-check application configurations 2025-07-01 14:26:43 +02:00
Kiara Grouwstra
dbd3e90238
test: factor out runtime-environment 2025-06-30 21:21:35 +02:00
Kiara Grouwstra
7906e9fc92
add comment on migrations 2025-06-30 17:41:08 +02:00
Kiara Grouwstra
d135c94afa
rename single-ssh-host for brevity 2025-06-30 17:22:27 +02:00
Kiara Grouwstra
c764c0f7b6
better reflect naming from diagram configuration data flow 2025-06-30 14:20:21 +02:00
Kiara Grouwstra
34529a7de4
data model: migration 2025-06-23 19:22:47 +02:00
Kiara Grouwstra
6c2022d064
data model: deployment 2025-06-23 16:35:11 +02:00
Kiara Grouwstra
f51462afc9
data model: runtime environment 
allows declaring options so instantiations may configure required
settings
 2025-06-23 16:35:04 +02:00
Kiara Grouwstra
fefcd93bc1
grant run-time environments their own modules with their own description 2025-06-23 11:25:18 +02:00
Kiara Grouwstra
c1f3aa6aed
have run-time environments use their corresponding run-time configurations 2025-06-23 09:34:59 +02:00
Kiara Grouwstra
8b2ee21dbe
data model: add run-time configuration 2025-06-23 09:06:52 +02:00
Nicolas “Niols” Jeannerod
bd1cfd7a7c Introduce test for deploying all services via FediPanel (#361) 
Closes #277

Same as #329 but where we run the FediPanel and interact with it via a browser
instead of running NixOps4 directly.

Reviewed-on: Fediversity/Fediversity#361
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
 2025-06-18 12:37:47 +02:00
Kiara Grouwstra
939f9d961d add data model entity: application (#387) 
part of #103.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Reviewed-on: Fediversity/Fediversity#387
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
Co-committed-by: Kiara Grouwstra <kiara@procolix.eu>
 2025-06-17 17:11:52 +02:00
Nicolas “Niols” Jeannerod
4801433ae0 Get rid of the need for deployer.pub (#385) 
The tests still work because we manually write the deployer's public key in `/root/.ssh/authorized_keys` on the target machines. In itself, however, the configuration that we push does not allow the deployer to push anything on the target machines.

Context: Fediversity/Fediversity#361 (comment)
Reviewed-on: Fediversity/Fediversity#385
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
 2025-06-17 16:34:29 +02:00
Valentin Gagarin
1b832c1f5b bypass native flake input for Nixpkgs (#374) 
@Niols the sheer amount of hassle and noise indicates that it may be better to first split out a `flake.nix` just for the tests. And all this clutter doesn't even explain yet *why* we thought it needs to be there.

closes #279.

Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Reviewed-on: Fediversity/Fediversity#374
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-committed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
 2025-06-12 13:05:11 +02:00
Nicolas “Niols” Jeannerod
69b2e535fe Document nullable fields sanitation (#365) 
Reviewed-on: Fediversity/Fediversity#365
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
 2025-06-10 11:57:01 +02:00
Nicolas “Niols” Jeannerod
09119803e8 Deployment: handle nullable config fields 
This is quite frustrating. In the meantime, it does get the deployment
working again.
 2025-06-06 11:50:48 +02:00
Nicolas “Niols” Jeannerod
56b953526b Deployment tests: Check status of services before deploying 2025-06-06 10:54:06 +02:00
Nicolas “Niols” Jeannerod
2fae356d0a Deployment tests: also make acmeNodeIP available in NixOS test 2025-06-06 10:52:49 +02:00
Nicolas “Niols” Jeannerod
046f7c5998 Deployment tests: comment on Pebble's certificate 2025-06-06 10:52:18 +02:00
Nicolas “Niols” Jeannerod
ee5c2b90b7 Introduce test for deploying all services with nixops4 apply (#329) 
Closes Fediversity/Fediversity#276

This PR adds a CLI deployment test. It builds on top of Fediversity/Fediversity#323. This test features a deployer node and four target nodes. The deployer node runs `nixops4 apply` on a deployment built with our actual code in `deployment/default.nix`, which pushes onto the four target machines combinations of Garage/Mastodon/Peertube/Pixelfed depending on a JSON payload. We check that the expected services are indeed deployed on the machines. Getting there involved reworking the existing basic test to extract common patterns, and adding support for ACME certificates negotiation inside the NixOS test.

What works:
- deployer successfully runs `nixops4 apply` with various payloads
- target machines indeed get the right services pushed onto them and removed
- services on target machines successfully negotiate ACME certificates

What does not work: the services themselves depend a lot on DNS and that is not taken care of at all, so they are probably very broken. Still, this is a good milestone.

Test it yourself by running `nix build .#checks.x86_64-linux.deployment-basic -vL` and `nix build .#checks.x86_64-linux.deployment-cli -vL`. On the very beefy machine that I am using, the basic test runs in ~4 minutes and the CLI test in ~17 minutes. We know from Fediversity/Fediversity#323 that the basic test runs in ~12 minutes on the CI runner, so maybe about an hour for the CLI test?

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Reviewed-on: Fediversity/Fediversity#329
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
 2025-05-19 02:18:54 +02:00
Valentin Gagarin
5f66a034f3 actually use the typed configuration in deployment 
that change somehow slipped through the cracks previously
 2025-05-09 12:46:11 +02:00
Valentin Gagarin
6100b278b6 generate Python data models from module options (#285) 
this shows a proof of concept for generating Django forms from NixOS modules

note that the form behavior is still rather clumsy and doesn't exactly map to the module semantics:
- since forms can only be sent wholesale, empty form fields will show up as empty strings
  and break validation without additional cleanup (not done here)
- it's not possible to faithfully translate `type = submodule { /* ... */}; default = {};`, since the default
  is translated to an empty dict `{}`. this is because the JSON schema converter does not preserve type information.
  this can be added by making it use `$defs` [1], but that would likely amount to half a rewrite
- there's a glitch in enum default values that needs to be fixed in `datamodel-code-generator` [0]

[0]: dd44480359/src/datamodel_code_generator/parser/base.py (L1015)
[1]: https://json-schema.org/understanding-json-schema/structuring#defs

a generated file will be placed into the source (by the development shell and the package respectively)
that declares Pydantic types from which to render the form. it looks something like this:

```python
from __future__ import annotations

from enum import Enum
from typing import Optional

from pydantic import BaseModel, Extra, Field
from drf_pydantic import BaseModel

class Domain(Enum):
    fediversity_net = 'fediversity.net'

# ...

class Model(BaseModel):
    class Config:
        extra = Extra.forbid

    domain: Optional[Domain] = Field(
        'fediversity.net',
        description='Apex domain under which the services will be deployed.\n',
    )

  # ...
```
 2025-05-01 01:26:52 +02:00
Nicolas “Niols” Jeannerod
f5db62e053 Add a basic integration test (#323) 
This PR adds a basic deployment test to the repository. This test will, in a NixOS test, run a deployer VM and a target VM, and check that we manage to run `nixops4 apply` on the deployer VM to change things on the target VM. The ideas are all @roberth's and this test has been extremely heavily inspired by https://github.com/nixops4/nixops4-nixos/blob/main/test/default/nixosTest.nix.

Reviewed-on: Fediversity/Fediversity#323
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
 2025-04-30 15:03:36 +02:00
Valentin Gagarin
b645660118 deployment module: set configuration explicitly (#315) 
this change is a no-op (it merely indents the option definitions by one,
by setting `config` explicitly) and prepares an addition of option
declarations that would otherwise be lost in the huge diff.

Reviewed-on: Fediversity/Fediversity#315
Reviewed-by: kiara Grouwstra <kiara@procolix.eu>
 2025-04-22 11:30:33 +02:00
Nicolas “Niols” Jeannerod
af3b2a62fd
Create a configuration resource even if the service is disabled 2025-04-02 15:06:17 +02:00
Nicolas “Niols” Jeannerod
4db91bd0b7
Transmit initial user from JSON to services 2025-02-28 10:57:31 +01:00
Nicolas “Niols” Jeannerod
f34f2e45ab
Get panel config as a block - use eg. mastodon.enable 2025-02-27 11:43:44 +01:00
Nicolas “Niols” Jeannerod
07b3cd90d7
Make clearer what nixos.module is 2025-02-27 11:39:07 +01:00
Nicolas “Niols” Jeannerod
ea98ccebfc
Very vague and minimalistic description of deployment/ 2025-02-27 11:35:55 +01:00
Nicolas “Niols” Jeannerod
c66889f58f
Better naming of makeMakeDeployment arguments 2025-02-27 11:35:55 +01:00
Nicolas “Niols” Jeannerod
e27cc6e96a
Generalise test deployment; get config from JSON 2025-02-27 11:35:54 +01:00
Nicolas “Niols” Jeannerod
3f9c174d97
Get rid of the deployment/ directory 2025-02-25 11:57:20 +01:00
Nicolas “Niols” Jeannerod
fb5bed9042
Remove other host keys 2025-02-25 11:57:20 +01:00
Nicolas “Niols” Jeannerod
636e4636f7
Add test machines 2025-02-25 11:57:20 +01:00
Nicolas “Niols” Jeannerod
0bea2d0bb1
Mark deployment/ as deprecated 2025-02-21 20:24:41 +01:00
Nicolas “Niols” Jeannerod
398dc05ad5
Rename provisioning host keys to match nixosConfiguration entries 2025-02-21 20:24:40 +01:00
Nicolas “Niols” Jeannerod
57e6127a7a
Move provision.sh and remove.sh to infra/ 2025-02-21 20:24:40 +01:00
Nicolas “Niols” Jeannerod
f8ec8e7d93
Decouple id and name 2025-02-21 20:24:40 +01:00
Nicolas “Niols” Jeannerod
d77b04ec18
Apply makeInstallerIso to conf in provision.sh 2025-02-21 20:24:40 +01:00
Nicolas “Niols” Jeannerod
bf0a35de6c
makeInstaller -> makeInstallerIso 2025-02-21 20:24:40 +01:00
Nicolas “Niols” Jeannerod
9e95287715
Make API URL an argument of the provisioning script 2025-02-21 20:24:40 +01:00
Nicolas “Niols” Jeannerod
cd83536e2f
Allow Garage and services to run on different machines 2025-02-21 17:52:50 +01:00
Nicolas “Niols” Jeannerod
1eeaa04df6
Introduce fediversity.garage.enable 2025-02-21 17:52:50 +01:00
Nicolas “Niols” Jeannerod
a5d226ed22
Get rid of fediversity.enable 2025-02-21 17:52:50 +01:00
Nicolas “Niols” Jeannerod
78a85b27ff
Put the S3 secrets into files 
...but not everywhere, there remains some FIXMEs where ultimately the
secrets do get into the store.
 2025-02-21 17:52:50 +01:00
Nicolas “Niols” Jeannerod
b547912794
Make access and secret keys parameters 2025-02-21 17:52:50 +01:00
Nicolas “Niols” Jeannerod
a1cfcf1d71
Same treatment for Peertube 2025-02-21 17:52:50 +01:00
Nicolas “Niols” Jeannerod
c6fb52915a
More clean up of the deployment flake 
Co-authored-by: Kiara Grouwstra <kiara@procolix.eu>
 2025-02-21 10:01:06 +01:00
Nicolas “Niols” Jeannerod
df182d5a67
Follow changes in removal script 2025-02-21 10:01:06 +01:00