kiara/fedi-goals
1
0
Fork 1
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

tweak project proposal #1

Open
kiara wants to merge 95 commits from rewrite into main
pull from: rewrite
merge into: kiara:main
Conversation 0 Commits 95 Files changed 6 +52 -138
1 changed files with 52 additions and 138 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files
Showing only changes of commit 53af240ccf - Show all commits

190
architecture.md
View file

@ -1,13 +1,11 @@
<style>* {font-family: sans-serif;}</style>
## Actors
- Maintainers
The group maintaining this repository.
We are creating the deployment workflows and service configurations, and curate changes proposed by contributing developers.
The group developing and maintaining this project.
We are creating the deployment workflows and service configurations, and curate changes proposed by contributors.
- Developers
- Contributors
People with the technical background to engage with our work, and may contribute back, build on top of, remix, or feel inspired by our work to create something better.
@ -56,7 +54,23 @@
- Migrate
Move service configurations and deployment (including user data) from one hosting provider to another.
Move service configurations and deployments (including user data) from one hosting provider to another.
- Runtime backend
A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container runtime.
- Runtime environment
The thing a deployment runs on, an interface against which the deployment is working. See runtime backend.
- Runtime configuration
A specification for mapping components of a configuration to the runtime environment, e.g. which services to deploy to which virtual machines, or how to access object storage.
- [NixOps4](https://nixops.dev)
A tool to interact with mutable external resources based on declarations in the [Nix language](https://nix.dev/manual/nix/latest/language/).
- Resource
@ -69,21 +83,12 @@
> Example: We need a resource provider for obtaining deployment secrets from a database.
- Runtime backend
A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container runtime.
- Runtime environment
The thing a deployment runs on, an interface against which the deployment is working. See runtime backend.
- Runtime config
Configuration logic specific to a runtime backend, e.g. how to deploy, how to access object storage.
## Technologies used
### [NixOS](https://nixos.org/)
This is an incomplete and evolving list of core components planned to be used in this project.
It will grow to support more advanced use cases as the framework matures.
### Nix and [NixOS](https://nixos.org/)
NixOS is a Linux distribution with a [vibrant](https://repology.org/repositories/graphs), [reproducible](https://reproducible.nixos.org/) and [security-conscious](https://tracker.security.nixos.org/) ecosystem.
As such, we see NixOS as the only viable way to reliably create a reproducible outcome for all the work we create.
@ -92,35 +97,6 @@ Considered alternatives include:
- containers: do not by themselves offer the needed reproducibility
#### [npins](https://github.com/andir/npins)
Npins is a dependency pinning tool for Nix which leaves recursive dependencies explicit, keeping the consumer in control.
Considered alternatives include:
- Flakes: defaults to implicitly following recursive dependencies, leaving control with the publisher.
### [SelfHostBlocks](https://nlnet.nl/project/SelfHostBlocks/)
SelfHostBlocks offers Nix module contracts to decouple application configuration from implementation details, empowering user choice by providing sane defaults yet a [unified interface](https://nlnet.nl/project/SelfHostBlocks/).
Offered contracts include back-ups, reverse proxies, single sign-on and LDAP.
In addition, we have been in contact with its creator.
Considered alternatives include:
- nixpkgs-provided NixOS service modules: support far more applications, but tightly coupled with service providers, whereas we expect them to [sooner or later](https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257) follow suit.
- NixOS service modules curated from scratch: would support any setup imaginable, but does not seem to align as well with our research-oriented goals.
### [OpenTofu](https://opentofu.org/)
OpenTofu is the leading open-source framework for infrastructure-as-code.
This has led it to offer a vibrant ecosystem of 'provider' plugins integrating various programs and services.
As such, it can facilitate automated deployment pipelines, including with — relevant to our project — hypervisors and DNS programs.
Considered alternatives include:
- Terraform: not open-source
### [Proxmox](https://proxmox.com/)
Proxmox is a hypervisor, allowing us to create VMs for our applications while adhering to our goal of preventing lock-in.
@ -139,75 +115,15 @@ Considered alternatives include:
- file storage: less centralized for backups
### [PostgreSQL](https://www.postgresql.org/)
PostgreSQL is a relational database.
It is used by most of our applications.
Considered alternatives include:
- Sqlite: default option for development in many applications, but less optimized for performance, and less centralized for backups
### [Valkey](https://valkey.io/)
Valkey is a key-value store.
It is an open-source fork of Redis.
Considered alternatives include:
- Redis: not open-source
### [OpenSearch](https://opensearch.org/)
OpenSearch offers full-text search, and is used for this in many applications.
It is an open-source fork of ElasticSearch.
Considered alternatives include:
- ElasticSearch: not open-source
### [OctoDNS](https://github.com/octodns/octodns)
OctoDNS is a DNS server that may be configured using the Nix-native [NixOS-DNS](https://janik-haag.github.io/NixOS-DNS/).
Considered alternatives include:
- PowerDNS: offers a front-end option, but less geared toward the use-case of configuring by Nix
### [Authelia](https://github.com/authelia/authelia)
Authelia is a single sign-on provider that integrates with LDAP.
Considered alternatives include:
- KaniDM: does not do proper LDAP
- Authentik: larger package with focus on many things we do not need
- Keycloak: larger package with focus on many things we do not need
### [lldap](https://github.com/lldap/lldap)
Lldap is a light LDAP server, allowing to centralize user roles across applications.
Considered alternatives include:
- 389 DS: older larger package
- FreeIPA: wrapper around 389 DS
### [Attic](https://github.com/zhaofengli/attic)
Attic is a multi-tenant Nix cache featuring recency-based garbage collection written in Rust.
Considered alternatives include:
- cache-server: distributed cache written in Python that seems more of a research project than an actively maintained repository.
## Architecture
At the core of Fediversity lies a NixOS configuration module for a set of selected applications.
We will support using it with different run-time environments, such as a single NixOS machine or a ProxmoX hypervisor.
Depending on the targeted run-time environment, deployment will further involve OpenTofu as an orchestrator.
We further provide a [reference front-end](https://git.fediversity.eu/Fediversity/Fediversity/src/branch/main/panel) to configure applications.
To ensure reproducibility, we also offer Nix packaging for our software.
- We will enable using it with **different run-time environments**, such as a single NixOS machine or a ProxmoX hypervisor.
- Depending on the targeted run-time environment, deployment may involve [NixOps4](https://nixops.dev) or [OpenTofu](https://opentofu.org/) as an **orchestrator**.
- We further provide demo front-end for **configuring applications** and configuring **run-time backends**.
To ensure reproducibility, all software will be packaged with Nix.
To reach our goals, we aim to implement the following interactions between [actors](#actors) (depicted with rounded corners) and system components (see the [glossary](#glossary), depicted with rectangles).
@ -217,41 +133,39 @@ To reach our goals, we aim to implement the following interactions between [acto
The process of migrating one's applications to a different host encompasses:
1. domain registration: involves a (manual) update of DNS records at the registrar
1. deployed applications: using the reproducible configuration module
1. application data:
- back-up/restore scripts [using SelfHostBlocks](https://shb.skarabox.com/contracts.html)
- application-specific migration scripts, to e.g. reconfigure of connections/URLs
1. Domain registration: involves a (manual) update of DNS records at the registrar
1. Deploy applications: using the reproducible configuration module
1. Copy application data:
- Run back-up/restore scripts
- Run application-specific migration scripts, to e.g. reconfigure connections/URLs
### Data model
Whereas the bulk of our configuration logic is covered in the configuration schema, our reference front-end application does in fact store data.
The design for its data model to support the desired functionality is as follows, using the crow's foot notation to denote cardinality:
Whereas the bulk of our configuration logic is covered in the configuration schema, our reference front-end applications will store data.
The data model design for the configuration front-end needed support the desired functionality is as follows, using the crow's foot notation to denote cardinality:
<img src="https://git.fediversity.eu/Fediversity/meta/raw/branch/main/architecture-docs/panel-data-model.svg" alt="" style="max-width:600px;"/>
### Host architecture
Whereas the core abstraction in Fediversity is a NixOS configuration module, a more full-fledged example architecture of the web host use-case we aim to support as part of our exploitation would be as follows, where VMs in question run Fediversity to offer our selected applications:
Whereas the core abstraction in Fediversity is a NixOS configuration module, a more full-fledged example architecture of the web host use-case we aim to support as part of our exploitation would be as follows, where virtual machines in question run Fediversity to offer our selected applications:
![](https://git.fediversity.eu/Fediversity/meta/raw/branch/main/architecture-docs/host-architecture.png)
## Break-down of project milestones
## Breakdown of project milestones and key results
Whereas details of the implementation may need to be decided as the technical challenges involved become clear, we can already give a higher-level planning of relevant milestones and some of their salient features:
1. [implement a way to run online services emphasising user autonomy and data portability](https://git.fediversity.eu/Fediversity/Fediversity/issues/347)
1. [Finalize architecture doc](https://git.fediversity.eu/Fediversity/Fediversity/issues/39)
1. [code-based migration data model](https://git.fediversity.eu/Fediversity/Fediversity/issues/103)
1. [migrating application data between hosting providers](https://git.fediversity.eu/Fediversity/Fediversity/issues/100)
1. [application offering generalised](https://git.fediversity.eu/Fediversity/Fediversity/issues/369)
1. [NixOS configuration as the core abstraction](https://git.fediversity.eu/Fediversity/Fediversity/issues/339)
1. [disseminate our results by engaging the open-source community to further expand on work in this direction](https://git.fediversity.eu/Fediversity/Fediversity/issues/348)
1. [automated dev-ops workflows](https://git.fediversity.eu/Fediversity/Fediversity/issues/224)
1. [external developers empowered to contribute](https://git.fediversity.eu/Fediversity/Fediversity/issues/288)
1. [exploit our work by enabling reproducible deployments of an initial set of portable applications](https://git.fediversity.eu/Fediversity/Fediversity/issues/349)
1. [applications deployed on command](https://git.fediversity.eu/Fediversity/Fediversity/issues/99)
1. [kick-started initial feedback cycle](https://git.fediversity.eu/Fediversity/Fediversity/issues/225)
1. [brought into production](https://git.fediversity.eu/Fediversity/Fediversity/issues/228)
1. [nix-less bootstrap](https://git.fediversity.eu/Fediversity/Fediversity/issues/332)
1. [key features improving user experience supported](https://git.fediversity.eu/Fediversity/Fediversity/issues/289)
- Implement a way to run online services emphasizing user autonomy and data portability
- Integration tests pass for
- Setting up a fediversity hosting environment from a declarative configuration
- Configuring, deploying, and migrating a set of dummy applications
- Code passes data protection audit
- Disseminate our results by engaging the open-source community to further expand on work in this direction
- Present results on at least 3 conferences
- At least 5 applications compatible with Fediversity thanks to external contributions by 2027-03
- Exploit our work by enabling reproducible deployments of an initial set of portable applications
- There are 3 fediverse applications available out of the box:
- Mastodon
- PeerTube
- Pixelfed