diff --git a/architecture.md b/architecture.md index 3413d5c..fdd1028 100644 --- a/architecture.md +++ b/architecture.md @@ -1,13 +1,11 @@ - - ## Actors - Maintainers - The group maintaining this repository. - We are creating the deployment workflows and service configurations, and curate changes proposed by contributing developers. + The group developing and maintaining this project. + We are creating the deployment workflows and service configurations, and curate changes proposed by contributors. -- Developers +- Contributors People with the technical background to engage with our work, and may contribute back, build on top of, remix, or feel inspired by our work to create something better. @@ -56,7 +54,23 @@ - Migrate - Move service configurations and deployment (including user data) from one hosting provider to another. + Move service configurations and deployments (including user data) from one hosting provider to another. + +- Runtime backend + + A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container runtime. + +- Runtime environment + + The thing a deployment runs on, an interface against which the deployment is working. See runtime backend. + +- Runtime configuration + + A specification for mapping components of a configuration to the runtime environment, e.g. which services to deploy to which virtual machines, or how to access object storage. + +- [NixOps4](https://nixops.dev) + + A tool to interact with mutable external resources based on declarations in the [Nix language](https://nix.dev/manual/nix/latest/language/). - Resource @@ -69,21 +83,12 @@ > Example: We need a resource provider for obtaining deployment secrets from a database. -- Runtime backend - - A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container runtime. - -- Runtime environment - - The thing a deployment runs on, an interface against which the deployment is working. See runtime backend. - -- Runtime config - - Configuration logic specific to a runtime backend, e.g. how to deploy, how to access object storage. - ## Technologies used -### [NixOS](https://nixos.org/) +This is an incomplete and evolving list of core components planned to be used in this project. +It will grow to support more advanced use cases as the framework matures. + +### Nix and [NixOS](https://nixos.org/) NixOS is a Linux distribution with a [vibrant](https://repology.org/repositories/graphs), [reproducible](https://reproducible.nixos.org/) and [security-conscious](https://tracker.security.nixos.org/) ecosystem. As such, we see NixOS as the only viable way to reliably create a reproducible outcome for all the work we create. @@ -92,35 +97,6 @@ Considered alternatives include: - containers: do not by themselves offer the needed reproducibility -#### [npins](https://github.com/andir/npins) - -Npins is a dependency pinning tool for Nix which leaves recursive dependencies explicit, keeping the consumer in control. - -Considered alternatives include: - -- Flakes: defaults to implicitly following recursive dependencies, leaving control with the publisher. - -### [SelfHostBlocks](https://nlnet.nl/project/SelfHostBlocks/) - -SelfHostBlocks offers Nix module contracts to decouple application configuration from implementation details, empowering user choice by providing sane defaults yet a [unified interface](https://nlnet.nl/project/SelfHostBlocks/). -Offered contracts include back-ups, reverse proxies, single sign-on and LDAP. -In addition, we have been in contact with its creator. - -Considered alternatives include: - -- nixpkgs-provided NixOS service modules: support far more applications, but tightly coupled with service providers, whereas we expect them to [sooner or later](https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257) follow suit. -- NixOS service modules curated from scratch: would support any setup imaginable, but does not seem to align as well with our research-oriented goals. - -### [OpenTofu](https://opentofu.org/) - -OpenTofu is the leading open-source framework for infrastructure-as-code. -This has led it to offer a vibrant ecosystem of 'provider' plugins integrating various programs and services. -As such, it can facilitate automated deployment pipelines, including with — relevant to our project — hypervisors and DNS programs. - -Considered alternatives include: - -- Terraform: not open-source - ### [Proxmox](https://proxmox.com/) Proxmox is a hypervisor, allowing us to create VMs for our applications while adhering to our goal of preventing lock-in. @@ -139,75 +115,15 @@ Considered alternatives include: - file storage: less centralized for backups -### [PostgreSQL](https://www.postgresql.org/) - -PostgreSQL is a relational database. -It is used by most of our applications. - -Considered alternatives include: - -- Sqlite: default option for development in many applications, but less optimized for performance, and less centralized for backups - -### [Valkey](https://valkey.io/) - -Valkey is a key-value store. -It is an open-source fork of Redis. - -Considered alternatives include: - -- Redis: not open-source - -### [OpenSearch](https://opensearch.org/) - -OpenSearch offers full-text search, and is used for this in many applications. -It is an open-source fork of ElasticSearch. - -Considered alternatives include: - -- ElasticSearch: not open-source - -### [OctoDNS](https://github.com/octodns/octodns) - -OctoDNS is a DNS server that may be configured using the Nix-native [NixOS-DNS](https://janik-haag.github.io/NixOS-DNS/). - -Considered alternatives include: - -- PowerDNS: offers a front-end option, but less geared toward the use-case of configuring by Nix - -### [Authelia](https://github.com/authelia/authelia) - -Authelia is a single sign-on provider that integrates with LDAP. - -Considered alternatives include: - -- KaniDM: does not do proper LDAP -- Authentik: larger package with focus on many things we do not need -- Keycloak: larger package with focus on many things we do not need - -### [lldap](https://github.com/lldap/lldap) - -Lldap is a light LDAP server, allowing to centralize user roles across applications. - -Considered alternatives include: - -- 389 DS: older larger package -- FreeIPA: wrapper around 389 DS - -### [Attic](https://github.com/zhaofengli/attic) - -Attic is a multi-tenant Nix cache featuring recency-based garbage collection written in Rust. - -Considered alternatives include: - -- cache-server: distributed cache written in Python that seems more of a research project than an actively maintained repository. - ## Architecture At the core of Fediversity lies a NixOS configuration module for a set of selected applications. -We will support using it with different run-time environments, such as a single NixOS machine or a ProxmoX hypervisor. -Depending on the targeted run-time environment, deployment will further involve OpenTofu as an orchestrator. -We further provide a [reference front-end](https://git.fediversity.eu/Fediversity/Fediversity/src/branch/main/panel) to configure applications. -To ensure reproducibility, we also offer Nix packaging for our software. + +- We will enable using it with **different run-time environments**, such as a single NixOS machine or a ProxmoX hypervisor. +- Depending on the targeted run-time environment, deployment may involve [NixOps4](https://nixops.dev) or [OpenTofu](https://opentofu.org/) as an **orchestrator**. +- We further provide demo front-end for **configuring applications** and configuring **run-time backends**. + +To ensure reproducibility, all software will be packaged with Nix. To reach our goals, we aim to implement the following interactions between [actors](#actors) (depicted with rounded corners) and system components (see the [glossary](#glossary), depicted with rectangles). @@ -217,41 +133,39 @@ To reach our goals, we aim to implement the following interactions between [acto The process of migrating one's applications to a different host encompasses: -1. domain registration: involves a (manual) update of DNS records at the registrar -1. deployed applications: using the reproducible configuration module -1. application data: - - back-up/restore scripts [using SelfHostBlocks](https://shb.skarabox.com/contracts.html) - - application-specific migration scripts, to e.g. reconfigure of connections/URLs +1. Domain registration: involves a (manual) update of DNS records at the registrar +1. Deploy applications: using the reproducible configuration module +1. Copy application data: + - Run back-up/restore scripts + - Run application-specific migration scripts, to e.g. reconfigure connections/URLs ### Data model -Whereas the bulk of our configuration logic is covered in the configuration schema, our reference front-end application does in fact store data. -The design for its data model to support the desired functionality is as follows, using the crow's foot notation to denote cardinality: +Whereas the bulk of our configuration logic is covered in the configuration schema, our reference front-end applications will store data. +The data model design for the configuration front-end needed support the desired functionality is as follows, using the crow's foot notation to denote cardinality: ### Host architecture -Whereas the core abstraction in Fediversity is a NixOS configuration module, a more full-fledged example architecture of the web host use-case we aim to support as part of our exploitation would be as follows, where VMs in question run Fediversity to offer our selected applications: +Whereas the core abstraction in Fediversity is a NixOS configuration module, a more full-fledged example architecture of the web host use-case we aim to support as part of our exploitation would be as follows, where virtual machines in question run Fediversity to offer our selected applications: ![](https://git.fediversity.eu/Fediversity/meta/raw/branch/main/architecture-docs/host-architecture.png) -## Break-down of project milestones +## Breakdown of project milestones and key results Whereas details of the implementation may need to be decided as the technical challenges involved become clear, we can already give a higher-level planning of relevant milestones and some of their salient features: -1. [implement a way to run online services emphasising user autonomy and data portability](https://git.fediversity.eu/Fediversity/Fediversity/issues/347) - 1. [Finalize architecture doc](https://git.fediversity.eu/Fediversity/Fediversity/issues/39) - 1. [code-based migration data model](https://git.fediversity.eu/Fediversity/Fediversity/issues/103) - 1. [migrating application data between hosting providers](https://git.fediversity.eu/Fediversity/Fediversity/issues/100) - 1. [application offering generalised](https://git.fediversity.eu/Fediversity/Fediversity/issues/369) - 1. [NixOS configuration as the core abstraction](https://git.fediversity.eu/Fediversity/Fediversity/issues/339) -1. [disseminate our results by engaging the open-source community to further expand on work in this direction](https://git.fediversity.eu/Fediversity/Fediversity/issues/348) - 1. [automated dev-ops workflows](https://git.fediversity.eu/Fediversity/Fediversity/issues/224) - 1. [external developers empowered to contribute](https://git.fediversity.eu/Fediversity/Fediversity/issues/288) -1. [exploit our work by enabling reproducible deployments of an initial set of portable applications](https://git.fediversity.eu/Fediversity/Fediversity/issues/349) - 1. [applications deployed on command](https://git.fediversity.eu/Fediversity/Fediversity/issues/99) - 1. [kick-started initial feedback cycle](https://git.fediversity.eu/Fediversity/Fediversity/issues/225) - 1. [brought into production](https://git.fediversity.eu/Fediversity/Fediversity/issues/228) - 1. [nix-less bootstrap](https://git.fediversity.eu/Fediversity/Fediversity/issues/332) - 1. [key features improving user experience supported](https://git.fediversity.eu/Fediversity/Fediversity/issues/289) +- Implement a way to run online services emphasizing user autonomy and data portability + - Integration tests pass for + - Setting up a fediversity hosting environment from a declarative configuration + - Configuring, deploying, and migrating a set of dummy applications + - Code passes data protection audit +- Disseminate our results by engaging the open-source community to further expand on work in this direction + - Present results on at least 3 conferences + - At least 5 applications compatible with Fediversity thanks to external contributions by 2027-03 +- Exploit our work by enabling reproducible deployments of an initial set of portable applications + - There are 3 fediverse applications available out of the box: + - Mastodon + - PeerTube + - Pixelfed