meta/meeting-notes/2024-11-20-architecture-meetup/2024-11-21-roberth-comments-post-meeting.md

# Architecture diagram

- Split out central database, netbox. They're separate databases
  "Operator" accounts and high-level configuration => NixPanel database
- Netbox will be accessed as a resource; perhaps move down
  - Netbox IP resource (IP allocation resource)
- Split up into nodes for each db so that it's clear which parts talk to which state.
- Nix configurator: NixOS is just a resource to NixOps, as Valentin says
- (PlantUML diagram) is this a package diagram? Package diagrams are meant to describe the application level structure (e.g. within NixPanel) rather than the system level structure (e.g. between NixPanel and NixOps)
  The arrows are supposed to be code dependencies, but many aren't. They seem to be some sort of interactions.

# Misc Notes

Valentin is on point about the NixOps decoupling

Ansible could be invoked by NixOps, over ssh remotely.
Only if there's a need. 100% on restricting to NixOS for now.

# Platform definitions

Running some "fixed" infrastructure on non-NixOS, like Proxmox, Garage, etc is fine. We do have an ambition to deploy everything from hardware using NixOps and NixOS, but this is not a blocker.
The way to think of this is what is the platform onto which a Fediversity is deployed:
- current platform: Proxmox + Garage + deSEC + netbox
- future alternative platform: bare metal
    - see [NixOps4-based-installation process] for an impression; out of scope for now iiuc
- other future alternative platform: Google Cloud, OVH, AWS, Scaleway, etc
    - no ambition to do this as part of Fediversity, but someone else may want to develop this
    - Nix language is capable of such abstractions

<details><summary>Example: Object Storage</summary>

Depending on the platform, the NixOps configuration will activate different resources. For example, object storage could be provided by
1. A provided Garage instance. The Fediversity NixOps configuration will need admin credentials to be entered.
   This is our first and for now only target.
2. Bare metal, in which case the Fediversity NixOps configuration may allocate VMs that run Garage, and configures Garage to give NixOps admin credentials.
3. A cloud native S3 compatible service

</details>

<details><summary>Abstractions in Nix</summary>

In NixOps4 we use the module system to perform abstractions.
Possible pattern:

- Option `objectStorage.backend.s3`: submodule with options to access an S3 compatible service
  - configured by hand when deploying onto an existing S3 service, such as Garage on Debian, or a cloud provider

- Option `objectStorage.backend.garage.enable`: whether to deploy a Garage cluster

- Option `objectStorage.buckets.<name>`: submodule with information about the bucket

- Config `resources."bucket-${name}"`: the various bucket resources, derived from the `objectStorage.buckets` option set

- Config `resources.garage.resources."${...}"`, conditional on `objectStorage.backend.garage.enable`: the resources to deploy Garage onto Proxmox or bare metal

The infra layer might be best modeled as a separate NixOps deployment. It does not need to run as part of the NixPanel user interface, but run by those who run the service provider.

</details>

# Secrets

- Sources
  - NixOps user level (workstation, not copied anywhere)
    - user ssh private key is reused by NixOS resource
    - credentials to access the NixOps state (e.g. in bucket object or over ssh)
    - (anything that bootstraps access to the NixOps state)
  - Outputs of resources will contain credentials (stored in NixOps state)
    - e.g. the admin password for anything created through NixOps resources
  - Sensitive infrastructure parameters inserted into the NixOps state initially
    - e.g. using a `pinentry` resource
    - or dummy resource that fails, but whose output attributes can be overridden by the NixOps CLI
- Destinations
  - Files on NixOS machines
    - copied to very private directory over ssh (`root`-only)
    - copied to their final place by NixOS systemd units
      <details>

      - work with NixOS community to standardize this
      - e.g. each service specifies with options where it expects its secrets, and waits for secret-specific systemd units using some convention
      - `nixops4-nixos`, `agenix`, `sops-nix`, etc. can all implement this interface

      </details>
    - handled in such a way that they don't end up in the Nix store (use protective wrapper when passed around in the Nix language)
  - NixOps resources
    - Passing one resource's output to another resource's input: the purpose of NixOps
    - NixOps state
      - Encrypted at rest
- Process
  - Simplest is to pass credentials around through NixOps, but indirections are possible. Benefits:
    - Implement key rotation without redeployment => vault-style secret management with short-lived secrets so that credential leaks are less damaging and less valuable
    - Fewer places with secrets => brings more structure to the organization's credentials management
      - Only if all secrets from the NixOps state can be moved over to the secret management system.
      - This will require more complexity in NixOps; would not recommend for now
      - Single point of failure, _or_ "use one hinge and oil it well" -- paraphrasing [age] authors or reference, out of context
  - `vaultwarden` can be written to by NixOps as needed, so that it can grant access declaratively

Security: compromise of a NixOps user compromises the whole deployment, and same for the NixOps state.
This probably includes equivalents of aforementioned "NixOps user level" secrets
  - State is likely compromised by compromising a NixOps user
  - NixPanel credentials are probably in the state, and they will need to have similar capabilities to those of the NixOps user credentials
Configuration management is expected to be able to do anything. (No need to freak out, but do use caution.)

# Terminology

- **operator**: I'm not super happy about "operator" instead of e.g. "customer". You're ignoring the Ops in NixOps. I can accept that, but it's not great communication.

  For NixOps it makes sense to use "operator" and "author" roles to describe its users. I don't see a better alternative for NixOps.

- **target platform**: Core services, infrastructure services, etc may be useful terms to distinguish the roles in terms of behavior and functionality.
  I think the missing term is "target platform", which captures the boundary of the scope of the Fediversity project. As described, this scope can be variable.

  For example, _infrastructure services_ like Garage or email can each individually be part of the _target platform_, as they are right now, or they can become part of Fediversity's infrastructure services.

  Being managed by Fediversity code is a less intrinsic property than it being "infrastructure", although arguably that's still a bit fuzzy; S3 not so much, but email is.

# Meta notes

- My contract is to develop NixOps, not NixPanel. I give advice.
- Usually "foo is done by NixOps" does not mean that it is implemented in the NixOps project, but rather that some action is initiated by calling a NixOps command, and the details of the operation are declared in a combination of
  - NixPanel output of some sort
  - A NixOps resource provider implementation
  - A set of Nix expressions developed by the Fediversity project
  - A Nix expression that's specific to a Fediversity deployment, which refers to the generic Fediversity expressions
    (kept to a minimum, things like does Fediversity deploy a garage or is it provided by the underlying platform)


[NixOps4-based-installation process]: https://git.fediversity.eu/Fediversity/meta/src/branch/main/architecture-docs/NixOps4-based-installation-process.md

[age]: https://github.com/FiloSottile/age