forked from Fediversity/Fediversity
Nicolas Jeannerod
8412a3f1fe
Co-authored-by: Taeer Bar-Yam <taeer.bar-yam@moduscreate.com> Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io> Reviewed-on: Fediversity/simple-nixos-fediverse#26 Co-authored-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com> Co-committed-by: Nicolas “Niols” Jeannerod <nicolas.jeannerod@moduscreate.com>
128 lines
6.6 KiB
Markdown
128 lines
6.6 KiB
Markdown
# Fediverse VMs
|
|
|
|
This repo is, for now, an attempt to familiarize myself with NixOS options for Fediverse applications, and build up a configuration layer that will set most of the relevant options for you (in a semi-opinionated way) given some high-level configuration. The goal is something in the same vein as [nixos-mailserver](https://gitlab.com/simple-nixos-mailserver/nixos-mailserver) but for fediversity.
|
|
|
|
Eventually, this will be tailored to high-throughput multi-machine setups. For now, it's just a small set of configurations to run in VMs.
|
|
|
|
## Running the VMs
|
|
|
|
you can build a VM using
|
|
|
|
```bash
|
|
nixos-rebuild build-vm --flake .#<vm_name>
|
|
```
|
|
|
|
where `<vm_name>` is one of `mastodon`, `peertube`, `pixelfed`, or `all`
|
|
|
|
and then run it with
|
|
```bash
|
|
./result/bin/run-nixos-vm
|
|
```
|
|
|
|
After the machine boots, you should be dropped into a root shell.
|
|
|
|
Note that state will be persisted in the `nixos.cqow2` file. Delete that and restart the VM to reset the state.
|
|
|
|
With the VM running, you can then access the apps on your local machine's web browser (using the magic of port forwarding) at the following addresses
|
|
|
|
NOTE: it sometimes takes a while for the services to start up, and in the meantime you will get 502 Bad Gateway.
|
|
|
|
- Mastodon: through the reverse proxy at <https://mastodon.localhost:8443> and directly at <http://mastodon.localhost:55001>
|
|
- You can create accounts on the machine itself by running `mastodon-tootctl accounts create test --email test@test.com --confirmed --approve`
|
|
- Account-related activities (logging in/out; preferences) can only be done on the insecure direct page <http://mastodon.localhost:55001>
|
|
- After you've logged in, you can go back to the secure page and you will remain logged in
|
|
- some operations may remove the port number from the URL. You'll have to add that back in manually
|
|
|
|
- PeerTube: <http://peertube.localhost:9000>
|
|
- The root account can be accessed with username "root". The password can be obtained by running the following command on the VM:
|
|
```bash
|
|
journalctl -u peertube | perl -ne '/password: (.*)/ && print $1'
|
|
```
|
|
- Creating other accounts has to be enabled via the admin interface. `Administration > Configuration > Basic > Enable Signup` or just add an account directly from `Administration > Create user`. But functionality can also be tested from the root account.
|
|
|
|
- Pixelfed: <http://pixelfed.localhost:8000>
|
|
- Account creation via the web interface won't work until we figure out email
|
|
- For now, they can be created on the VM command line
|
|
```bash
|
|
pixelfed-manage user:create --name=test --username=test --email=test@test.com --password=testtest --confirm_email=1
|
|
```
|
|
# Building an installer image
|
|
|
|
Build an installer image for the desired configuration, e.g. for `peertube`:
|
|
|
|
```bash
|
|
nix build .#installers.peertube
|
|
```
|
|
|
|
Upload the image in `./result` to Proxmox when creating a VM.
|
|
Booting the image will format the disk and install NixOS with the desired configuration.
|
|
|
|
# Deploying an updated machine configuration
|
|
|
|
> TODO: There is currently no way to specify an actual target machine by name.
|
|
|
|
Assuming you have SSH configuration with access to the remote `root` user stored for a machine called e.g. `peertube`, deploy the configuration by the same name:
|
|
|
|
```bash
|
|
nix run .#deploy.peertube
|
|
```
|
|
|
|
## debugging notes
|
|
|
|
- it is sometimes useful to `cat result/bin/run-nixos-vm` to see what's really going on (e.g. which ports are getting forwarded)
|
|
- relevant systemd services:
|
|
- mastodon-web.service
|
|
- peertube.service
|
|
- the `garage` CLI command gives information about garage storage, but cannot be used to actually inspect the contents. use `mc` (minio) for that
|
|
- run `mc alias set garage http://s3.garage.localhost:3900 --api s3v4 --path off $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY`
|
|
- in the chromium devtools, you can go to the networking tab and change things like response headers in a way that persists through reloads. this is much faster iteration time if that's what you need to epxeriment with.
|
|
|
|
## NixOS Tests
|
|
|
|
Tests live in the aptly named `tests/` directory, and can be accessed at the flake URI `.#checks.<system>.<test-name>` e.g. `nix build .#checks.x86_64-linux.mastodon-garage`.
|
|
They can also be run interactively with
|
|
```
|
|
nix build .#checks.<system>.<test>.driverInteractive
|
|
./result/bin/nixos-test-driver 2>output
|
|
````
|
|
you can `less output` and then `F` from a different terminal to follow along.
|
|
|
|
These tests are also equiped with the same port forwarding as the VMs, so when running interactively you should be able to access services through a browser running on your machine.
|
|
|
|
While running interactively, `rebuildableTests` allows you to modify the test nodes and then redeploy without restarting the test and waiting for the VMs to start up again. To do this you must start the jumphost by running `redeploy_jumphost.start()` inside the driver. Then from the command line
|
|
|
|
```
|
|
nix build .#checks.<system>.<test>.driverInteractive
|
|
./result/bin/rebuild
|
|
```
|
|
|
|
# questions
|
|
|
|
- what is meant to be shared between instances?
|
|
- this is relevant to the security model. If garage is being shared between instances, we have to be careful having configurations depend on each other.
|
|
- they are to be shared, BUT the user will have no direct control over configuration.
|
|
|
|
# resources
|
|
|
|
- Tutorial for setting up better logging: https://krisztianfekete.org/self-hosting-mastodon-on-nixos-a-proof-of-concept/
|
|
- Setting up mastodon development environment: https://docs.joinmastodon.org/dev/setup/
|
|
|
|
- Tutorial for PeerTube that doesn't use `createLocally`: https://nixos.wiki/wiki/PeerTube
|
|
|
|
- garage settings for specific apps: https://garagehq.deuxfleurs.fr/documentation/connect/apps/
|
|
|
|
- pixelfed has terrible / mostly non-existent documentation)
|
|
|
|
- for when we start worry about scaling up: https://docs.joinmastodon.org/admin/scaling/
|
|
|
|
# notes
|
|
|
|
When mastodon is running in production mode, we have a few problems:
|
|
- you have to click "accept the security risk"
|
|
- it takes a while for the webpage to come online. Until then you see "502 Bad Gateway"
|
|
- email sent from the mastodon instance (e.g. for account confirmation) should be accessible at <https://mastodon.localhost:55001/letter_opener>, but it's not working.
|
|
- mastodon is trying to fetch `missing.png` without ssl (`http://`). This isn't allowed, and i'm not sure why it's doing it.
|
|
- mastodon is trying to fetch `custom.css` from https://mastodon.localhost (no port), which is not the configured `LOCAL_DOMAIN`, so it's unclear why.
|
|
|
|
NixOS tests do not take the configuration from `virtualisation.vmVariant`. This seems like an oversight since people don't tend to mix normal NixOS configurations with the ones they're using for tests. This should be pretty easy to rectify upstream.
|