new path to point the report to

Reviewed-on: #34
Reviewed-by: Valentin Gagarin <valentin.gagarin@tweag.io>

architecture-docs/2025-05-architecture-replaced-by-mermaid.dot

@@ -1,4 +1,4 @@
-// usage: nix-shell -p graphviz --command dot -T png -o architecture.png architecture.dot
+// usage: nix-shell -p graphviz --command 'dot -T png -o architecture.png architecture.dot'
 
 digraph FediversityArchitecture {
   graph[fontname="Liberation Sans"];
@@ -7,11 +7,13 @@ digraph FediversityArchitecture {
   subgraph cluster_frontends {
     label="front-ends";
 
+    nix[label="NixOS module"];
     tofu[label="OpenTofu CLI"];
     panel[label="FediPanel"];
     protagio[label="NixPanel"];
   }
 
+  nix -> core;
   tofu -> core;
   panel -> core;
   protagio -> core;

architecture-docs/architecture.md

@@ -0,0 +1,176 @@
+# Fediversity Implementation and planning
+
+## Actors
+
+- Maintainers
+
+  The group developing and maintaining this project.
+  We are creating the deployment workflows and service configurations, and curate changes proposed by contributors.
+
+- Developers
+
+  People with the technical background to engage with our work, and may contribute back, build on top of, remix, or feel inspired by our work to create something better.
+
+- Hosting provider
+
+  They provide and maintain the physical infrastructure, and run the software in this repository, through which operators interact with their deployments.
+  Hosting providers are technical administrators for these deployments, ensuring availability and appropriate performance.
+
+  We target small- to medium-scale hosting providers with 20+ physical machines.
+
+- Operator
+
+  They select the applications they want to run.
+  They don't need to own hardware or deal with operations.
+  Operators administer their applications in a non-technical fashion, e.g. as moderators.
+  They pay the hosting provider for registering a domain name, maintaining physical resources, and monitoring deployments.
+
+- User
+
+  They are individuals using applications run by the operators, and e.g. post content.
+
+## Glossary
+
+- [Fediverse](https://en.wikipedia.org/wiki/Fediverse)
+
+  A collection of social networking applications that can communicate with each other using a common protocol.
+
+- Application
+
+  User-facing software (e.g. from Fediverse) configured by operators and used by users.
+
+- Configuration
+
+  A collection of settings for a piece of software.
+
+  > Example: Configurations are deployed to VMs.
+
+- Provision
+
+  Make a resource, such as a virtual machine, available for use.
+
+- Deploy
+
+  Put software onto computers.
+  The software includes technical configuration that links software components.
+
+- Migrate
+
+  Move service configurations and deployments (including user data) from one hosting provider to another.
+
+- Run-time backend
+
+  A type of digital environment one can run operating systems such as NixOS on, e.g. bare-metal, a hypervisor, or a container run-time.
+
+- Provider
+
+  An interface against which we deploy to a run-time backend.
+
+- Provider configuration
+
+  A configuration that specifies resources made available to deploy to and how to access these.
+
+- Resource
+
+  A resource is any external entity that we need for our set-up
+  This may include e.g. hypervisors, file systems, DNS entries, VMs or object storage instances.
+
+## Technologies used
+
+This is an incomplete and evolving list of core components planned to be used in this project.
+It will grow to support more advanced use cases as the framework matures.
+
+### Nix and [NixOS](https://nixos.org/)
+
+NixOS is a Linux distribution with a [vibrant](https://repology.org/repositories/graphs), [reproducible](https://reproducible.nixos.org/) and [security-conscious](https://tracker.security.nixos.org/) ecosystem.
+As such, we see NixOS as the only viable way to reliably create a reproducible outcome for all the work we create.
+
+Considered alternatives include:
+
+- containers: do not by themselves offer the needed reproducibility
+
+### [Proxmox](https://proxmox.com/)
+
+Proxmox is a hypervisor, allowing us to create VMs for our applications while adhering to our goal of preventing lock-in.
+In addition, it has been [packaged for Nix](https://github.com/SaumonNet/proxmox-nixos) as well, simplifying our requirements to users setting up our software.
+
+Considered alternatives include:
+
+- OpenNebula: seemed less mature
+
+### [Garage](https://garagehq.deuxfleurs.fr/)
+
+Garage is a distributed object storage service.
+For compatibility with existing clients, it reuses the protocol of Amazon S3.
+
+Considered alternatives include:
+
+- file storage: less centralized for backups
+
+## Architecture
+
+At the core of Fediversity lies a NixOS configuration module for a set of selected applications.
+
+- We will enable using it with **different run-time environments**, such as a single NixOS machine or a ProxmoX hypervisor.
+- Depending on the targeted run-time environment, deployment may involve [NixOps4](https://nixops.dev) or [OpenTofu](https://opentofu.org/) as an **orchestrator**.
+- We further provide demo front-end for **configuring applications** and configuring **run-time backends**.
+
+To ensure reproducibility, all software will be packaged with Nix.
+
+To reach our goals, we aim to implement the following interactions.
+
+The used legend is as follows:
+
+- Circle: [actor](#actors)
+- Angled box: type
+- Rectangle: value
+- Rounded box: function
+- Diamond: state
+- Arrow: points towards dependant
+
+For further info on components see the [glossary](#glossary).
+
+<!-- ![](./interactions-migration.svg) -->
+![](./interactions-migration.png)
+
+### Configuration data flow
+
+This data flow diagram refines how a deployment is obtained from an operator's application configuration and a hosting provider's runtime setup.
+
+An **application module** specifies operator-facing **application options**, and a **configuration mapping** which determines the application's underlying implementation. Application modules can be supplied by external developers, which would curate application modules against that interface.
+
+For its runtime setup, a hosting provider has to supply a **resource mapping** that would take their self-declared **provider configuration** (which determines the *available* resources) and the output of an application's resource mapping (which determine resource *requirements*) and produce a **configuration**. This configuration ships with a mechanism to be *deployed* to the infrastructure (which is described by the environment, and features the required resources), where it will accumulate **application state**.
+
+Applications and runtime environments thus interface through **resources**, the properties of which are curated by Fediversity maintainers.
+
+<!-- ![](./interactions-fediversity.svg) -->
+![](./interactions-fediversity.png)
+
+
+### Service portability
+
+The process of migrating one's applications to a different host encompasses:
+
+1. Domain registration: involves a (manual) update of DNS records at the registrar
+1. Deploy applications: using the reproducible configuration module
+1. Copy application data:
+    - Run back-up/restore scripts
+    - Run application-specific migration scripts, to e.g. reconfigure connections/URLs
+
+### Data model
+
+Whereas the bulk of our configuration logic is covered in the configuration schema, [implemented here](https://git.fediversity.eu/Fediversity/Fediversity/src/branch/main/deployment/data-model.nix) and [tested here](https://git.fediversity.eu/Fediversity/Fediversity/src/branch/main/deployment/data-model-test.nix), our reference front-end applications will store data.
+The data model design for the configuration front-end needed support the desired functionality is as follows, using the crow's foot notation to denote cardinality:
+
+<img src="./panel-data-model.svg" alt="" style="max-width:600px;"/>
+
+### Host architecture
+
+Whereas the core abstraction in Fediversity is a NixOS configuration module, a more full-fledged example architecture of the web host use-case we aim to support as part of our exploitation would be as follows, where virtual machines in question run Fediversity to offer our selected applications:
+
+![](https://git.fediversity.eu/Fediversity/meta/raw/branch/main/architecture-docs/host-architecture.png)
+
+### CI / CD
+
+In our simplest set-up, continuous integration and continuous deployment pipelines are handled using Forgejo's [built-in runner](https://code.forgejo.org/forgejo/runner), with relevant secrets handled using [Forgejo secrets](https://forgejo.org/docs/latest/developer/secrets/). Jobs we handle using CI include linting, formatting, testing, and a periodic life-cycle management job to keep our dependencies up-to-date.
+In a future iteration, we may make use of [Gerrit](https://gerrit.googlesource.com/) to better manage our review process for incoming merge requests.

architecture-docs/configuration-to-deployment-dataflow.md

@@ -0,0 +1,74 @@
+# 2025-06-24 Fediversity configuration data flow
+
+This data flow diagram refines the [global architecture diagram](./interactions-migration.mmd) with respect to how a deployment is obtained from an operator's application configuration and a hosting provider's runtime setup.
+
+## Legend
+
+- Circle: actor
+- Angled box: type
+- Rectangle: value
+- Rounded box: function
+- Diamond: state
+- Arrow: points towards dependant
+
+## Description
+
+An **application module** specifies operator-facing **application options**, and a **resource mapping** which determines the application's underlying implementation in terms of which resources it requires to be deployed. Application modules can be supplied by external **contributors**, which would package applications against that interface.
+
+For its **runtime environment**, a **hosting provider** has to supply a **policy mapping** that would take their self-declared **resource configuration** (which determines *available* resources, i.e. a description of the infrastructure on which to run the applications) and the output of an application's resource mapping (which determines resource *requirements*) and produce the description of a **deployment**.
+
+An **operator** can supply an **application configuration** and request the hosting provider to let the resulting deployment take effect, such that it will accumulate **application state**, which can later be migrated to other hosting providers.
+
+Applications and runtime environments thus interface through **resources**, the properties of which are curated by Fediversity **maintainers**.
+
+
+```mermaid
+flowchart
+
+subgraph configuration
+  application-config
+end
+subgraph resource[resource module]
+  resource-options
+  provider-options
+end
+subgraph application[application module]
+  application-options{{application-options}} --> application-config
+  application-config --> config-mapping
+  resource-options{{consumer-options}} --> config-mapping
+  config-mapping(resource-mapping)
+end
+subgraph fediversity[runtime-environment]
+  config-mapping -->|required resources| resource-mapping(policy-mapping)
+
+  provider-options{{resource-options}} --> resource-config --> |available resources| resource-mapping -->|configured resources| deployment[deployment]
+  deployment --> |deploy| state{application state}
+
+end
+
+maintainer((maintainer)) -->|curates| resource
+
+
+contributor((contributor)) -->|packages| application
+operator((operator)) -->|enters| configuration
+
+
+hosting-provider((hosting provider)) -->|maintains| fediversity
+
+```
+
+## Discussion
+
+Decoupling of operator-facing application configuration and resource use allows for portability at the infrastructure level: hosting providers determine how applications are deployed to available resources through policies which are agnostic to applications. Additionally it allows hosting providers to isolate applications from each other depending on their deployment model, leveraging the entire toolkit provided by the Nix ecosystem. This puts single-machine and cluster deployments on equal footing, since policies are simply Nix language functions.
+
+At first glance this is an increase in complexity as opposed to configuring NixOS directly, since it adds a layer of indirection. It also requries Fediversity maintainers to declare resource types for use by application packagers and hosting providers. But the layering drastically simplifies application composition over what NixOS can currently offer, frees the hosting provider to choose how to implement required services, and enables security-conscious deployments at very fine granularity.
+
+## Next steps
+
+- Versioning: Each of the components can change; model all supported interactions such that deployments will always be correct.
+
+  Primarily, both applications and runtime environments depend on the resource API, and need to be versioned against it. Application packages can be updated on top of that, and will need to provide facilities to migrate between both application versions and resource API versions, both in terms of configurations and application state.
+  
+- Provider migrations: Prototype an application-agnostic migration workflow.
+
+  Since state migration is application-specific, application packages will need to ship with instructions for how to move their data between hosting providers, also taking version migrations into account. This will likely require extending the resource API with ways to express where state is located, and give hosting providers the tools to expose interfaces though which migrations will take place.

architecture-docs/interactions-fediversity.mmd

@@ -0,0 +1,30 @@
+flowchart
+
+subgraph configuration
+  application-config
+end
+subgraph resource[resource module]
+  resource-options
+  provider-options
+end
+subgraph application[application module]
+  application-options{{application-options}} --> application-config
+  application-config --> config-mapping
+  resource-options{{resource-options}} --> config-mapping
+  config-mapping(config-mapping)
+end
+subgraph fediversity[fediversity setup]
+  config-mapping -->|required resources| resource-mapping(resource-mapping)
+
+  provider-options{{provider-options}} -->  provider-config --> |available resources| resource-mapping -->|configuration| deployment{deployment}
+
+end
+
+maintainer((maintainer)) -->|curates| resource
+
+
+contributor((developer)) -->|curates| application
+operator((operator)) -->|enters| configuration
+
+
+hosting-provider((hosting provider)) -->|maintains| fediversity

architecture-docs/interactions-migration.mmd

@@ -1,9 +1,9 @@
 flowchart
 
-  user(user) --> |use| deployment
+  user((user)) --> |use| deployment
 
   configuration1 -->|deploy| deployed1
-  devs(developers) --> |maintain| fediversity
+  maintainers --> |maintain| fediversity
 
   fediversity --> |update| provider1
   subgraph provider1["fediversity setup A"]
@@ -17,13 +17,13 @@ flowchart
     provider-config --> |implement runtime interfaces| configurations1
     subgraph host[runtime environment]
       deployment[applications]
-      state
+      state{state}
     end
   end
 
   deployment --> |store| state
 
-  operator(operator) --> |change| configuration1
+  operator((operator)) --> |change| configuration1
 
   subgraph provider2["fediversity setup B"]
      subgraph configurations2[configurations]
@@ -32,16 +32,16 @@ flowchart
     end
     subgraph host2[runtime environment]
       deployment2[applications]
-      state2[state]
+      state2{state}
     end
   end
 
-  operator --> |trigger| migration
+  operator --> |trigger| migration(migration)
   configurations1 & state --> migration
   migration --> configurations2 & state2
-  provider(hosting provider) --> |maintain| provider1
+  provider((hosting provider)) --> |maintain| provider1
   subgraph fediversity[fediversity source code]
     applications[application modules]
     backends[runtime backends]
-    config[runtime options]
+    config{{runtime options}}
   end

architecture-docs/panel-data-model.mmd

@@ -0,0 +1,24 @@
+---
+title: Data model of sample web application
+---
+erDiagram
+    operator {
+        string username
+        string password_hash
+    }
+    deployment {
+        json deployed_configuration
+        option[string] staged_configuration
+        option[string] version
+    }
+    backup["back-up"] {
+        string bucket
+        string endpoint
+    }
+    keypair {
+        string access_key
+        string secret_key
+    }
+    operator ||--o{ deployment : has
+    deployment ||--o{ backup : has
+    backup ||--|{ keypair : authorises

default.nix

@@ -0,0 +1,14 @@
+{
+  pkgs ? import <nixpkgs> { },
+}:
+{
+  shell = pkgs.mkShellNoCC {
+    packages = with pkgs; [
+      pandoc
+      texliveMedium
+      librsvg
+      mermaid-cli
+      plantuml
+    ];
+  };
+}

investment_analysis/readme.md

@@ -0,0 +1 @@
+This is where we put stuff.

meeting-notes/2025-06-12-developer-sync.md

@@ -0,0 +1,35 @@
+# 2025-06-11 Fediversity developer sync
+
+- @kiara has been updating the project proposal to reflect current ideas
+    - https://git.fediversity.eu/kiara/fedi-goals/src/branch/rewrite/
+    - need to make sure all stakeholders are on board with the updated vision
+    - new slated project manager would need to know what they're signing up for
+    - we reviewed and discussed the draft
+
+- @niols got the full integration test for deploying via the panel to pass
+    - https://git.fediversity.eu/Fediversity/Fediversity/pulls/361
+    - we tinkered on reducing the number of hacks involved
+    - @fricklerhandwerk had some ideas for that, but the sketches need cleanup
+        - https://git.fediversity.eu/Fediversity/Fediversity/pulls/364
+- @kiara and @fricklerhanderk continued editing the adjustments to the project proposal
+    - @kiara: whether/how to concentrate on external contributors (and if so, how to sensibly reflect that in a key result)
+    - @kiara: whether keeping the fediverse in even helps communicate our mission, considering fediversity could not fix the fediverse if we wanted to (given speed of adoption imo def isn't about self-hosting)
+- @fricklerhandwerk incorporated discussion results into the writeup
+    - https://git.fediversity.eu/kiara/fedi-goals/pulls/2
+    - https://git.fediversity.eu/kiara/fedi-goals/pulls/3
+
+## Draft success metrics for the adjusted objectives
+
+- Implement a way to run online services emphasizing user autonomy and data portability
+    - Integration tests pass for
+        - Setting up a fediversity hosting environment from a declarative configuration
+        - Configuring, deploying, and migrating a set of dummy applications
+    - Code passes data protection audit
+- Disseminate our results by engaging the open-source community to further expand on work in this direction
+    - Present results on at least 3 conferences
+    - At least 5 applications compatible with Fediversity thanks to external contributions by 2027-03
+- Exploit our work by enabling reproducible deployments of an initial set of portable applications
+    - There are 3 fediverse applications available out of the box:
+        - Mastodon
+        - PeerTube
+        - Pixelfed

meeting-notes/2025-07-01 developer sync.md

@@ -0,0 +1,12 @@
+Attendees: @niols @kiara @fricklerhandwerk
+
+- updated the project board
+- made some clarifications around splitting https://git.fediversity.eu/Fediversity/Fediversity/pulls/389 into manageable chunks
+- agreed on a general strategy to get a grip on the project layout:
+  - migrate everything to modules
+  - keep using flake-parts but don't use the flake parts except for CI checks
+    - we may simply swap it out with our custom modules eventually, since most of them will be custom anyway
+    - eventually we will be able to render most of contributor documentation via a Git hook
+  - a few small steps already pushed as PRs
+- @niols and @kiara will focus on speeding up CI and making the runner reproducible
+- @fricklerhandwerk will keep developing the data model with @kiara

meeting-notes/2025-07-03 data model review.md

@@ -0,0 +1,16 @@
+Attendees: @kiara @fricklerhandwerk
+
+- Discussed @fricklerhandwerk's WIP branch for implementing https://git.fediversity.eu/Fediversity/Fediversity/issues/103
+- The resource policy idea seems sound
+- The test sample doesn't look minimal at first glance, but in fact is
+  - An entire NixOS configuration is only minimal in terms of resource policy -- just put it in a VM
+  - An application that configures a whole NixOS needs to set many more things to make it run
+  - The login-shell sample also allows us to explicitly enumerate resource-specific configuration options to demonstrate how they're declared and used
+- We re-iterated on the stance not to touch actual applications before we have the whole workflow tested with samples
+  - Our packaged applications are already tested and in a state where we could translate them to the proper data model mechanically once it's in place and well-understood, so this will be a well-scoped problem for next year
+- It emerged again that we need to figure out a notion of compatibility (and therefore versioning) before being able to model migrations
+  - Once the data model draft type-checks and the tests pass, this will be the next research question
+  - We need to check if we want or have to make simplifying assumptions, such as not allowing version rollbacks
+  - Chances are, once we have compatibility clarified, migrations will be trivial
+- @kiara will spend some time today debugging the draft
+- Likely we'll need the next week to finish that part and play with it a bit, such as by adding a few more test cases.

meeting-notes/2025-07-08 weekly planning

@@ -0,0 +1,28 @@
+Attendees: @fricklerhandwerk @niols @kiara
+
+- @fricklerhandwerk
+    - almost completed the data model up to deployment, including a test
+        - @kiara fixed up some typos and evaluation errors
+        - remaining is to mold `nixops4Deployment` into a type that is accepted
+    - started compiling the mid-term technical report
+    - will pause work on the data model until that's done (next week)
+- @niols worked on extracting a minimal flake for the test
+    - (discussed how to handle lazy/strict source closures)
+- @kiara
+    - tried containerizing the CI but it didn't work out yet
+        - setting up a cache may improve turnaround time by a lot
+        - will pair with @niols the week after next
+    - tried patching the workflow to enable automatic dependency updates, made small progress
+        - we can do dependency bumps manually for now
+    - will pause work on CI until the technical report is done
+- todo for end-to-end beta: (REST) API for interacting with a runtime environment (general idea already settled in earlier discussions)
+    - GET json schema for configuring applications
+    - GET configuration
+    - POST configuration
+    - POST deploy
+    - GET deployment (version, config, status, ...)
+    - GET (stream) deployment progress
+    - Webhook: deployment state updates, e.g. completed, failed, ...
+    - TODO: workflow for migration
+        - ...
+        - 

meeting-notes/2025-07-22 developer sync.md

@@ -0,0 +1,20 @@
+Attendees: @kiara @fricklerhandwerk @niols
+
+- CI still bad: checkout action flaky, secrets exposed to all contributors
+  - We discussed switching to Gerrit (for review suggestions) and buildbot-nix (for CI)
+  - It will be a lot of effort, but it's already a lot of effort and we better spend it now than when more people join the project
+  - Chances are this will solve the secrets issue automatically
+- Regressions in Proxmox provisioning
+  - needs tests and docs, `proxmox-provision.sh` was written ad hoc beginning of the year
+  - we haven't been exercising our knowledge of provisioning since
+  - would be nice to rewrite in Python, also as a way to turn it into a NixOps4 resource provider
+- NixOps4 future:
+  - @roberth signaled the path to nested deployments may not be all that long
+  - @fricklerhandwerk: currently our use case is perfectly suited by the state of affairs (especially having everything one language), but eventually we'll need something more powerful
+  - @niols: dry run threatens to be a can of worms with nested deployments
+    - @fricklerhandwerk: we'd need to mock away everything, we don't have a programming pattern for that yet (sigh)
+    - @niols: for now we expose only the NixOS configurations to get evaluated separately
+- @kiara kept dabbling at the data model, fixing small issues
+  - @fricklerhandwerk will go through the WIP branch and cherry-pick what works
+- @niols will add tests to provisioning code to fix regressions from recent refactorings
+  - will start refactoring `infra/flake-parts.nix` to remove the wild function passing

shell.nix

@@ -0,0 +1 @@
+(import ./. { }).shell