Fediverse VMs

This repo is, for now, an attempt to familiarize myself with NixOS options for Fediverse applications, and build up a configuration layer that will set most of the relevant options for you (in a semi-opinionated way) given some high-level configuration. The goal is something in the same vein as nixos-mailserver but for fediversity.

Eventually, this will be tailored to high-throughput multi-machine setups. For now, it's just a small set of configurations to run in VMs.

Running the VMs

you can build a VM using

nixos-rebuild build-vm --flake .#<vm_name>

where <vm_name> is one of mastodon, peertube, pixelfed, or all

and then run it with

./result/bin/run-nixos-vm

After the machine boots, you should be dropped into a root shell.

Note that state will be persisted in the nixos.cqow2 file. Delete that and restart the VM to reset the state.

With the VM running, you can then access the apps on your local machine's web browser (using the magic of port forwarding) at the following addresses

NOTE: it sometimes takes a while for the services to start up, and in the meantime you will get 502 Bad Gateway.

• Mastodon: through the reverse proxy at https://mastodon.localhost:8443 and directly at http://mastodon.localhost:55001

  • You can create accounts on the machine itself by running mastodon-tootctl accounts create test --email test@test.com --confirmed --approve
  • Account-related activities (logging in/out; preferences) can only be done on the insecure direct page http://mastodon.localhost:55001
  • After you've logged in, you can go back to the secure page and you will remain logged in
  • some operations may remove the port number from the URL. You'll have to add that back in manually

• PeerTube: http://peertube.localhost:9000

  • The root account can be accessed with username "root". The password can be obtained by running the following command on the VM:

    journalctl -u peertube | perl -ne '/password: (.*)/ && print $1'
    

  • Creating other accounts has to be enabled via the admin interface. Administration > Configuration > Basic > Enable Signup or just add an account directly from Administration > Create user. But functionality can also be tested from the root account.

• Pixelfed: http://pixelfed.localhost:8000

  • Account creation via the web interface won't work until we figure out email
  • For now, they can be created on the VM command line

    pixelfed-manage user:create --name=test --username=test --email=test@test.com --password=testtest --confirm_email=1
    

Building an installer image

Build an installer image for the desired configuration, e.g. for peertube:

nix build .#installers.peertube

Upload the image in ./result to Proxmox when creating a VM. Booting the image will format the disk and install NixOS with the desired configuration.

Deploying an updated machine configuration

TODO: There is currently no way to specify an actual target machine by name.

Assuming you have SSH configuration with access to the remote root user stored for a machine called e.g. peertube, deploy the configuration by the same name:

nix run .#deploy.peertube

debugging notes

• it is sometimes useful to cat result/bin/run-nixos-vm to see what's really going on (e.g. which ports are getting forwarded)
• relevant systemd services:
  • mastodon-web.service
  • peertube.service
• the garage CLI command gives information about garage storage, but cannot be used to actually inspect the contents. use mc (minio) for that
  • run mc alias set garage http://s3.garage.localhost:3900 --api s3v4 --path off $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
• in the chromium devtools, you can go to the networking tab and change things like response headers in a way that persists through reloads. this is much faster iteration time if that's what you need to epxeriment with.

NixOS Tests

Tests live in the aptly named tests/ directory, and can be accessed at the flake URI .#checks.<system>.<test-name> e.g. nix build .#checks.x86_64-linux.mastodon-garage. They can also be run interactively with

nix build .#checks.<system>.<test>.driverInteractive
./result/bin/nixos-test-driver 2>output

you can less output and then F from a different terminal to follow along.

These tests are also equiped with the same port forwarding as the VMs, so when running interactively you should be able to access services through a browser running on your machine.

While running interactively, rebuildableTests allows you to modify the test nodes and then redeploy without restarting the test and waiting for the VMs to start up again. To do this you must start the jumphost by running redeploy_jumphost.start() inside the driver. Then from the command line

nix build .#checks.<system>.<test>.driverInteractive
./result/bin/rebuild

questions

• what is meant to be shared between instances?
  • this is relevant to the security model. If garage is being shared between instances, we have to be careful having configurations depend on each other.
  • they are to be shared, BUT the user will have no direct control over configuration.

resources

• Tutorial for setting up better logging: https://krisztianfekete.org/self-hosting-mastodon-on-nixos-a-proof-of-concept/

• Setting up mastodon development environment: https://docs.joinmastodon.org/dev/setup/

• Tutorial for PeerTube that doesn't use createLocally: https://nixos.wiki/wiki/PeerTube

• garage settings for specific apps: https://garagehq.deuxfleurs.fr/documentation/connect/apps/

• pixelfed has terrible / mostly non-existent documentation)

• for when we start worry about scaling up: https://docs.joinmastodon.org/admin/scaling/

notes

When mastodon is running in production mode, we have a few problems:

• you have to click "accept the security risk"
• it takes a while for the webpage to come online. Until then you see "502 Bad Gateway"
• email sent from the mastodon instance (e.g. for account confirmation) should be accessible at https://mastodon.localhost:55001/letter_opener, but it's not working.
• mastodon is trying to fetch missing.png without ssl (http://). This isn't allowed, and i'm not sure why it's doing it.
• mastodon is trying to fetch custom.css from https://mastodon.localhost (no port), which is not the configured LOCAL_DOMAIN, so it's unclear why.

NixOS tests do not take the configuration from virtualisation.vmVariant. This seems like an oversight since people don't tend to mix normal NixOS configurations with the ones they're using for tests. This should be pretty easy to rectify upstream.