diff --git a/README.md b/README.md index 18ca913..66d0271 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ ```sh nix-shell -pandoc fediversity.md -o fediversity.html +pandoc architecture.md -o architecture.html +pandoc architecture.html -o architecture.pdf +pandoc --filter pandoc-include fediversity.md -o fediversity.html pandoc fediversity.html -o fediversity.pdf ``` diff --git a/architecture.md b/architecture.md new file mode 100644 index 0000000..2aeade3 --- /dev/null +++ b/architecture.md @@ -0,0 +1,236 @@ + + +## Actors + +- Developers + + The group working on this repository. + We are creating the deployment workflows and service configurations. + +- Hosting provider + + They provide and maintain the physical infrastructure, and run the software in this repository, through which operators interact with their deployments. + Hosting providers are technical administrators for these deployments, ensuring availability and appropriate performance. + + We target small- to medium-scale hosting providers with 20+ physical machines. + +- Operator + + They select the applications they want to run. + They don't need to own hardware or deal with operations. + Operators administer their applications in a non-technical fashion, e.g. as moderators. + They pay the hosting provider for registering a domain name, maintaining physical resources, and monitoring deployments. + +- User + + They are individuals using applications run by the operators, and e.g. post content. + +## Glossary + +- [Fediverse](https://en.wikipedia.org/wiki/Fediverse) + + A collection of social networking applications that can communicate with each other using a common protocol. + +- Application + + User-facing software (e.g. from Fediverse) run by the hosting provider for an operator. + +- Configuration + + A collection of settings for a machine running NixOS. + + > Example: Configurations are deployed to VMs. + +- Provision + + Make a resource, such as a virtual machine, available for use. + +- Deploy + + Put software, such as applications, onto computers. + The software includes technical configuration that links software components. + In our context, this a Configuration deployed to a runtime environment. + Most user-facing configuration remains untouched by the deployment process. + +- Migrate + + Move service configurations and deployment (including user data) from one hosting provider to another. + +- Resource + + A [resource for NixOps4](https://nixops.dev/manual/development/concept/resource.html) is any external entity that can be declared with NixOps4 expressions and manipulated with NixOps4, such as a virtual machine, an active NixOS configuration, a DNS entry, or customer database. + +- Resource provider + + A resource provider for NixOps4 is an executable that communicates between a resource and NixOps4 using a standardised protocol, allowing [CRUD operations](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) on the resources to be performed by NixOps4. + Refer to the [NixOps4 manual](https://nixops.dev/manual/development/resource-provider/index.html) for details. + + > Example: We need a resource provider for obtaining deployment secrets from a database. + +- Runtime backend + + A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container runtime. + +- Runtime environment + + The thing a deployment runs on, an interface against which the deployment is working. See runtime backend. + +- Runtime config + + Configuration logic specific to a runtime backend, e.g. how to deploy, how to access object storage. + +## Architecture + +At the core of Fediversity lies a NixOS configuration template containing selected applications. +We use this to deploy to selected run-time environments, so far targeting hypervisor ProxmoX. +We further provide a reference front-end to configure our template. +To ensure reproducibility, we also offer Nix packaging for our software. + +To reach our goals, we aim to implement the following interactions between [actors](#actors) (depicted with rounded corners) and system components (see the [glossary](#glossary), depicted with rectangles). + +![](https://git.fediversity.eu/Fediversity/meta/raw/branch/entity-relations/architecture-docs/interactions.svg) + +### Entity relationships + +Relationships among the entities used to model migrations are as follows, using the crow's foot notation to denote cardinality: + +![](https://git.fediversity.eu/Fediversity/meta/raw/branch/entity-relations/architecture-docs/entity-relations.svg) + +### Data model + +### Host architecture + +Whereas the core abstraction in Fediversity is a NixOS configuration template, a more full-fledged example architecture of the web host use-case we aim to support as part of our exploitation would be as follows, where VMs in question run Fediversity to offer our selected applications: + +![](https://git.fediversity.eu/Fediversity/meta/raw/branch/entity-relations/architecture-docs/host-architecture.png) + +## Break-down of project milestones + +Whereas details of the implementation may need to be decided as the technical challenges involved become clear, we can already give a higher-level planning of relevant milestones and some of their salient features: + +- Initial internal user to kick-start feedback process: + - automate provisioning of: + - [databases](https://git.fediversity.eu/Fediversity/Fediversity/issues/115) + - [object storage](https://git.fediversity.eu/Fediversity/Fediversity/issues/185) + - [virtual machines](https://git.fediversity.eu/Fediversity/Fediversity/issues/116) (and related resources) + - [DNS](https://git.fediversity.eu/Fediversity/Fediversity/issues/110) + - [SMTP service](https://git.fediversity.eu/Fediversity/Fediversity/issues/117) + - [email accounts](https://git.fediversity.eu/Fediversity/Fediversity/issues/118) + - [ephemeral state](https://git.fediversity.eu/Fediversity/Fediversity/issues/314) + - [publish specification](https://git.fediversity.eu/Fediversity/Fediversity/issues/334) using e.g. JSON Schema / OpenAPI + - [facilitate multi-tenancy](https://git.fediversity.eu/Fediversity/Fediversity/issues/241) + - [provision admin accounts](https://git.fediversity.eu/Fediversity/Fediversity/issues/178) + - [ensure users can update their configurations](https://git.fediversity.eu/Fediversity/Fediversity/issues/158) +- Software ready for web hosts to take into production: + - [garbage collection of unallocated resources](https://git.fediversity.eu/Fediversity/Fediversity/issues/188) + - [Nix-less bootstrap](https://git.fediversity.eu/Fediversity/Fediversity/issues/332) + - [ensure hosts may update users' deployments](https://git.fediversity.eu/Fediversity/Fediversity/issues/159) + - [allow control over version of module deployed](https://git.fediversity.eu/Fediversity/Fediversity/issues/304) + - [security audit](https://git.fediversity.eu/Fediversity/Fediversity/issues/291) + - [automate dependency updates](https://git.fediversity.eu/Fediversity/Fediversity/issues/65) +- Features to improve user experience and increase host adoption: + - [enqueue deployment syncs](https://git.fediversity.eu/Fediversity/Fediversity/issues/242) + - [application data back-ups](https://git.fediversity.eu/Fediversity/Fediversity/issues/123) + - [migrating application data between hosting providers](https://git.fediversity.eu/Fediversity/Fediversity/issues/100) + - [provide single sign-on](https://git.fediversity.eu/Fediversity/Fediversity/issues/212) + - [expand exposed configuration settings](https://git.fediversity.eu/Fediversity/Fediversity/issues/195) + - [allow disabling application while retaining data](https://git.fediversity.eu/Fediversity/Fediversity/issues/186) + - [aid needed user actions on schema update](https://git.fediversity.eu/Fediversity/Fediversity/issues/214) + - [scaling application resources](https://git.fediversity.eu/Fediversity/Fediversity/issues/119) + - [pooling application instances to shared VMs](https://git.fediversity.eu/Fediversity/Fediversity/issues/322) + - [allow use of external single sign-on](https://git.fediversity.eu/Fediversity/Fediversity/issues/161) + - [on migration, allow reconfiguring monolithic vs distributed](https://git.fediversity.eu/Fediversity/Fediversity/issues/341) +- Facilitate engagement from external developers: + - [create integration tests](https://git.fediversity.eu/Fediversity/Fediversity/issues/277) + - [add continuous integration builds to a public cache](https://git.fediversity.eu/Fediversity/Fediversity/issues/92) + - [reproduce required infrastructure](https://git.fediversity.eu/Fediversity/Fediversity/issues/336) + - [continuous deployment](https://git.fediversity.eu/Fediversity/Fediversity/issues/177) + - [separate staging/testing environments](https://git.fediversity.eu/Fediversity/Fediversity/issues/69) + - [facilitate user signup](https://git.fediversity.eu/Fediversity/Fediversity/issues/335) + - [facilitate code reviews](https://git.fediversity.eu/Fediversity/Fediversity/issues/302) + - [get documentation ready](https://git.fediversity.eu/Fediversity/Fediversity/issues/288) + - [upstream to NixOS](https://git.fediversity.eu/Fediversity/Fediversity/issues/333) + +## Technologies used + +### [NixOS](https://nixos.org/) + +NixOS is a Linux distribution with a [vibrant](https://repology.org/repositories/graphs), [reproducible](https://reproducible.nixos.org/) and [security-conscious](https://tracker.security.nixos.org/) ecosystem. +As such, we see NixOS as the only viable way to reliably create a reproducible outcome for all the work we create. + +Considered alternatives include: +- containers: do not by themselves offer the needed reproducibility + +### [OpenTofu](https://opentofu.org/) + +OpenTofu is the leading open-source framework for infrastructure-as-code. +This has led it to offer a vibrant ecosystem of 'provider' plugins integrating various programs and services. +As such, it can facilitate automated deployment pipelines, including with — relevant to our project — hypervisors and DNS programs. + +Considered alternatives include: +- Terraform: not open-source + +### [Proxmox](https://proxmox.com/) + +Proxmox is a hypervisor, allowing us to create VMs for our applications while adhering to our goal of preventing lock-in. +In addition, it has been [packaged for Nix](https://github.com/SaumonNet/proxmox-nixos) as well, simplifying our requirements to users setting up our software. + +Considered alternatives include: +- OpenNebula: seemed less mature + +### [Garage](https://garagehq.deuxfleurs.fr/) + +Garage is a distributed object storage service. +For compatibility with existing clients, it reuses the protocol of Amazon S3. + +Considered alternatives include: +- file storage: less centralized for backups + +### [PostgreSQL](https://www.postgresql.org/) + +PostgreSQL is a relational database. +It is used by most of our applications. + +Considered alternatives include: +- Sqlite: default option for development in many applications, but less optimized for performance, and less centralized for backups + +### [Valkey](https://valkey.io/) + +Valkey is a key-value store. +It is an open-source fork of Redis. + +Considered alternatives include: +- Redis: not open-source + +### [OpenSearch](https://opensearch.org/) + +OpenSearch offers full-text search, and is used for this in many applications. +It is an open-source fork of ElasticSearch. + +Considered alternatives include: +- ElasticSearch: not open-source + +### [PowerDNS](https://github.com/PowerDNS/pdns) + +PowerDNS is a mature DNS server. It further offers an admin front-end. + +Considered alternatives include: +- hickory-dns: no front-end +- core-dns: no front-end + +### [Authelia](https://github.com/authelia/authelia) + +Authelia is a single sign-on provider that integrates with LDAP. + +Considered alternatives include: +- KaniDM: does not do proper LDAP +- Authentik: larger package with focus on many things we do not need +- Keycloak: larger package with focus on many things we do not need + +### [lldap](https://github.com/lldap/lldap) + +Lldap is a light LDAP server, allowing to centralize user roles across applications. + +Considered alternatives include: +- 389 DS: older larger package +- FreeIPA: wrapper around 389 DS diff --git a/fediversity.md b/fediversity.md index 7fc4c0c..ad3e101 100644 --- a/fediversity.md +++ b/fediversity.md @@ -347,235 +347,9 @@ We will integrate that aspect into the high level process on a best effort basis # Implementation and planning - -## Actors - -- Developers - - The group working on this repository. - We are creating the deployment workflows and service configurations. - -- Hosting provider - - They provide and maintain the physical infrastructure, and run the software in this repository, through which operators interact with their deployments. - Hosting providers are technical administrators for these deployments, ensuring availability and appropriate performance. - - We target small- to medium-scale hosting providers with 20+ physical machines. - -- Operator - - They select the applications they want to run. - They don't need to own hardware or deal with operations. - Operators administer their applications in a non-technical fashion, e.g. as moderators. - They pay the hosting provider for registering a domain name, maintaining physical resources, and monitoring deployments. - -- User - - They are individuals using applications run by the operators, and e.g. post content. - -## Glossary - -- [Fediverse](https://en.wikipedia.org/wiki/Fediverse) - - A collection of social networking applications that can communicate with each other using a common protocol. - -- Application - - User-facing software (e.g. from Fediverse) run by the hosting provider for an operator. - -- Configuration - - A collection of settings for a machine running NixOS. - - > Example: Configurations are deployed to VMs. - -- Provision - - Make a resource, such as a virtual machine, available for use. - -- Deploy - - Put software, such as applications, onto computers. - The software includes technical configuration that links software components. - In our context, this a Configuration deployed to a runtime environment. - Most user-facing configuration remains untouched by the deployment process. - -- Migrate - - Move service configurations and deployment (including user data) from one hosting provider to another. - -- Resource - - A [resource for NixOps4](https://nixops.dev/manual/development/concept/resource.html) is any external entity that can be declared with NixOps4 expressions and manipulated with NixOps4, such as a virtual machine, an active NixOS configuration, a DNS entry, or customer database. - -- Resource provider - - A resource provider for NixOps4 is an executable that communicates between a resource and NixOps4 using a standardised protocol, allowing [CRUD operations](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) on the resources to be performed by NixOps4. - Refer to the [NixOps4 manual](https://nixops.dev/manual/development/resource-provider/index.html) for details. - - > Example: We need a resource provider for obtaining deployment secrets from a database. - -- Runtime backend - - A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container runtime. - -- Runtime environment - - The thing a deployment runs on, an interface against which the deployment is working. See runtime backend. - -- Runtime config - - Configuration logic specific to a runtime backend, e.g. how to deploy, how to access object storage. - -## Architecture - -At the core of Fediversity lies a NixOS configuration template containing selected applications. -We use this to deploy to selected run-time environments, so far targeting hypervisor ProxmoX. -We further provide a reference front-end to configure our template. -To ensure reproducibility, we also offer Nix packaging for our software. - -To reach our goals, we aim to implement the following interactions between [actors](#actors) (depicted with rounded corners) and system components (see the [glossary](#glossary), depicted with rectangles). - -![](https://git.fediversity.eu/Fediversity/meta/raw/branch/entity-relations/architecture-docs/interactions.svg) - -### Entity relationships - -Relationships among the entities used to model migrations are as follows, using the crow's foot notation to denote cardinality: - -![](https://git.fediversity.eu/Fediversity/meta/raw/branch/entity-relations/architecture-docs/entity-relations.svg) - -### Data model - - -## Break-down of project milestones - -Whereas details of the implementation may need to be decided as the technical challenges involved become clear, we can already give a higher-level planning of relevant milestones and some of their salient features: - -- Initial internal user to kick-start feedback process: - - automate provisioning of: - - [databases](https://git.fediversity.eu/Fediversity/Fediversity/issues/115) - - [object storage](https://git.fediversity.eu/Fediversity/Fediversity/issues/185) - - [virtual machines](https://git.fediversity.eu/Fediversity/Fediversity/issues/116) (and related resources) - - [DNS](https://git.fediversity.eu/Fediversity/Fediversity/issues/110) - - [SMTP service](https://git.fediversity.eu/Fediversity/Fediversity/issues/117) - - [email accounts](https://git.fediversity.eu/Fediversity/Fediversity/issues/118) - - [ephemeral state](https://git.fediversity.eu/Fediversity/Fediversity/issues/314) - - [publish specification](https://git.fediversity.eu/Fediversity/Fediversity/issues/334) using e.g. JSON Schema / OpenAPI - - [facilitate multi-tenancy](https://git.fediversity.eu/Fediversity/Fediversity/issues/241) - - [provision admin accounts](https://git.fediversity.eu/Fediversity/Fediversity/issues/178) - - [ensure users can update their configurations](https://git.fediversity.eu/Fediversity/Fediversity/issues/158) -- Software ready for web hosts to take into production: - - [garbage collection of unallocated resources](https://git.fediversity.eu/Fediversity/Fediversity/issues/188) - - [Nix-less bootstrap](https://git.fediversity.eu/Fediversity/Fediversity/issues/332) - - [ensure hosts may update users' deployments](https://git.fediversity.eu/Fediversity/Fediversity/issues/159) - - [allow control over version of module deployed](https://git.fediversity.eu/Fediversity/Fediversity/issues/304) - - [security audit](https://git.fediversity.eu/Fediversity/Fediversity/issues/291) - - [automate dependency updates](https://git.fediversity.eu/Fediversity/Fediversity/issues/65) -- Features to improve user experience and increase host adoption: - - [enqueue deployment syncs](https://git.fediversity.eu/Fediversity/Fediversity/issues/242) - - [application data back-ups](https://git.fediversity.eu/Fediversity/Fediversity/issues/123) - - [migrating application data between hosting providers](https://git.fediversity.eu/Fediversity/Fediversity/issues/100) - - [provide single sign-on](https://git.fediversity.eu/Fediversity/Fediversity/issues/212) - - [expand exposed configuration settings](https://git.fediversity.eu/Fediversity/Fediversity/issues/195) - - [allow disabling application while retaining data](https://git.fediversity.eu/Fediversity/Fediversity/issues/186) - - [aid needed user actions on schema update](https://git.fediversity.eu/Fediversity/Fediversity/issues/214) - - [scaling application resources](https://git.fediversity.eu/Fediversity/Fediversity/issues/119) - - [pooling application instances to shared VMs](https://git.fediversity.eu/Fediversity/Fediversity/issues/322) - - [allow use of external single sign-on](https://git.fediversity.eu/Fediversity/Fediversity/issues/161) -- Facilitate engagement from external developers: - - [create integration tests](https://git.fediversity.eu/Fediversity/Fediversity/issues/277) - - [add continuous integration builds to a public cache](https://git.fediversity.eu/Fediversity/Fediversity/issues/92) - - [reproduce required infrastructure](https://git.fediversity.eu/Fediversity/Fediversity/issues/336) - - [continuous deployment](https://git.fediversity.eu/Fediversity/Fediversity/issues/177) - - [separate staging/testing environments](https://git.fediversity.eu/Fediversity/Fediversity/issues/69) - - [facilitate user signup](https://git.fediversity.eu/Fediversity/Fediversity/issues/335) - - [facilitate code reviews](https://git.fediversity.eu/Fediversity/Fediversity/issues/302) - - [get documentation ready](https://git.fediversity.eu/Fediversity/Fediversity/issues/288) - - [upstream to NixOS](https://git.fediversity.eu/Fediversity/Fediversity/issues/333) - -## Technologies used - -### [NixOS](https://nixos.org/) - -NixOS is a Linux distribution with a [vibrant](https://repology.org/repositories/graphs), [reproducible](https://reproducible.nixos.org/) and [security-conscious](https://tracker.security.nixos.org/) ecosystem. -As such, we see NixOS as the only viable way to reliably create a reproducible outcome for all the work we create. - -Considered alternatives include: -- containers: do not by themselves offer the needed reproducibility - -### [OpenTofu](https://opentofu.org/) - -OpenTofu is the leading open-source framework for infrastructure-as-code. -This has led it to offer a vibrant ecosystem of 'provider' plugins integrating various programs and services. -As such, it can facilitate automated deployment pipelines, including with — relevant to our project — hypervisors and DNS programs. - -Considered alternatives include: -- Terraform: not open-source - -### [Proxmox](https://proxmox.com/) - -Proxmox is a hypervisor, allowing us to create VMs for our applications while adhering to our goal of preventing lock-in. -In addition, it has been [packaged for Nix](https://github.com/SaumonNet/proxmox-nixos) as well, simplifying our requirements to users setting up our software. - -Considered alternatives include: -- OpenNebula: seemed less mature - -### [Garage](https://garagehq.deuxfleurs.fr/) - -Garage is a distributed object storage service. -For compatibility with existing clients, it reuses the protocol of Amazon S3. - -Considered alternatives include: -- file storage: less centralized for backups - -### [PostgreSQL](https://www.postgresql.org/) - -PostgreSQL is a relational database. -It is used by most of our applications. - -Considered alternatives include: -- Sqlite: default option for development in many applications, but less optimized for performance, and less centralized for backups - -### [Valkey](https://valkey.io/) - -Valkey is a key-value store. -It is an open-source fork of Redis. - -Considered alternatives include: -- Redis: not open-source - -### [OpenSearch](https://opensearch.org/) - -OpenSearch offers full-text search, and is used for this in many applications. -It is an open-source fork of ElasticSearch. - -Considered alternatives include: -- ElasticSearch: not open-source - -### [PowerDNS](https://github.com/PowerDNS/pdns) - -PowerDNS is a mature DNS server. It further offers an admin front-end. - -Considered alternatives include: -- hickory-dns: no front-end -- core-dns: no front-end - -### [Authelia](https://github.com/authelia/authelia) - -Authelia is a single sign-on provider that integrates with LDAP. - -Considered alternatives include: -- KaniDM: does not do proper LDAP -- Authentik: larger package with focus on many things we do not need -- Keycloak: larger package with focus on many things we do not need - -### [lldap](https://github.com/lldap/lldap) - -Lldap is a light LDAP server, allowing to centralize user roles across applications. - -Considered alternatives include: -- 389 DS: older larger package -- FreeIPA: wrapper around 389 DS +```include +architecture.md +``` ## Work plan and resources diff --git a/shell.nix b/shell.nix index 031e504..bf1305a 100644 --- a/shell.nix +++ b/shell.nix @@ -5,6 +5,7 @@ shell = pkgs.mkShellNoCC { packages = with pkgs; [ pandoc + pandoc-include texliveMedium librsvg ];