Merge remote-tracking branch 'origin/main'

architecture-docs/data-model-requirements.md

@@ -0,0 +1,126 @@
+# migration data model requirements
+
+To transfer between two providers, the target provider must be able to import the sending provider's versions. (e.g.: a deployment may have latest fediversity, latest pixelfed, but previous mastadon) Thus, for each "realease" of the data model, it needs to be versioned, and applications/APIs also are versioned.
+ * (May need a way to show on the front-end which versions are in place, and which migrations are supported. However, for application versions which are completely controlled by the installation and setup, this is "solved".)
+
+for release version 0, focus on known current needs
+ * to be expanded later as each new application is added and can be transferred between providers
+ * review migration guides for the known apps with an eye to odd/unusual details that influence design choices (task for Niols? others?)
+
+Specifically, this suggests scoping to migrating:
+
+- managed infrastructure (rather than managed applications)
+- between servers owned by procolix
+- same proxmox version
+- NixOS VMs set up by us so we can guarantee identical application versions
+- hosting limited to a single application (to start)
+
+First, a bit of an inventory (list without much structure now, later will create structured form/schema with e.g. many-to-many links, useful for the migration code):
+ * clearly mark items that will not be in the first migration as eventually or speculative
+ * or reamove them if they would be too far in the future
+ * later we understand what is useful for migration code, we can extract and transform in to a format suitable as data model documentation
+
+Hosting Provider provides:
+* proxmox, git
+* hardware
+* filesystem storage
+* DNS automation hooks?
+* central/shared garage storage or only hardware+diskspace for the garage VMs to create storage?
+  * with central: more efficient but less isolated
+
+FooUniversity (Operator)
+* invoice info
+  * is all info expected to be transferred from provider A to provider B?
+    * May not want to transfer e.g. bank details, because they are already set up at B
+    * May also depend on regulation (which information are you allowed to hand out?)
+* Admins:
+  * credentials
+* persistent identifiers
+  * mappings between them (also need to travel across providers)
+    * e.g. if we can't change content URLs, we may need to create (and from then on carry around) a redirects mapping
+    * those mappings are likely application-specific, but they all belong to the same type class
+* domain(s)
+    * what is needed for DNS management?
+    * users
+      * display name
+      * email(s)
+      * login id
+        * oauth2 (eventually)
+        * 2fa
+        * password
+        * passkeys (eventually)
+        * LDAP? (eventually?)
+    * all applications
+      * sub domain ( social.example.org vs example.org/social )
+      * info for proxmox setup such as to provision VMs (to reproduce proxmox )
+        * mem
+        * cpus
+        * storage mounts
+        * IPs likely not the same in the target network
+      * storage
+        * filesystem
+           * very well specified per application
+        * blob storage config (garage, s3-like)
+           * index
+           * Can we make it a requirement that Garage is behind a predictable URL, eg. `<application>.garage.<customer domain>`? As opposed to something vendor-specific, eg. `pixelfed-university.garage.procolix.com/<customer domain>/<application>`
+           * may need to rewrite URLs to blobs automatically, depending on the underlying URL scheme, which may be per setup or application
+        * limits? per application? per user? where are these used/set/enforced?
+        * TODO: what does e.g. borgmatic need to back up storage?
+        * out of scope?: focus on actual state, disregarding reconstructable stuff
+      * SQL database
+        * dump/snapshot
+        * TODO: what does e.g. borgmatic need to back up databases?
+* application specifics
+  * postfix? (is email in version 0?)
+  * pixelfed
+    * where is blob storage
+      * in the specific case of Pixelfed, if blob storage changed URL, we might need to rewrite the pictures URLs in the database (try to avoid this)
+    * redis (in the case of pixelfed, it is not just a cache)
+      * misc config: theme, name of instance, email of sysadmin
+    * database
+    * on-disk files
+      * Daniel Supernault is currently making it so evertying can be stored remotely in a garage or sql database
+    * users (login id) (in database? in redis?)
+      * user preferences/settings
+  * peertube
+  * mastodon
+  * matrix? (is it in version 0?)
+* logos
+
+Other considerations:
+
+- Put a boundary for what is
+  - operator-configurable
+  - needs to get fixed, but at the implementation level
+  - what can be configured dynamically per environment
+- Most importantly we need to preserve persistent identifiers
+
+- When transforming the data-model code to a deliverable version of the data model as part of the technical architecture document, documenting user-data storage and with respects fot security and GDPR
+
+See also:
+
+- possible overlap/inspiration: Stalw.art [configuration docs](https://stalw.art/docs/server/general)
+
+## MVP scoping ideas
+
+User story 1: New customer
+When a new customer goes to the Fediversity website we want to show that user what Fediversity is all about and what it can give to the customer. This points the customer to a signup form where they can enter all the details that are needed to get it working. Here they can also decide what applications to use (at first no more than three). Details can be, the user/admin login, name, address, bank details, domain, other users, and applications. Than when the customer hits the install/provision/go button everything starts to install automagically. After which the customer is presented with (some) url's to login to.
+
+User story 2: Take out / move to other instance
+At any time a customer may wish to change service providers. They can easily go to an admin screen where they can get their configuration and data packaged for transfer. This packaged data can be provided to a new service provider where they will be up-and-running again easily, with minimal downtime.
+
+proposed MVP scope:
+- block storage
+- blob storage (garage)
+- physical servers
+- proxmox vm management
+- nixops service
+- nixops scripts
+- 1 to 3 applications packaged in Nix (Mastodon, Peertube, Pixelfed)
+- frontend / website
+- working dns, can be external, but automated
+- takeout area
+- import area
+- 2 Fediversity environments to transfer between
+- demonstration of User story 1
+- demonstration of User story 2

architecture-docs/on-flakes.md

@@ -0,0 +1,45 @@
+# Use more principled dependency management than flakes
+
+As per [nix.dev](https://nix.dev/concepts/flakes#should-i-use-flakes-in-my-project):
+
+> Flakes emphasize reproducible artifacts and convenience for their consumers, while classic Nix tools center around composable building blocks and customisation options for developers.
+
+We have currently used nix flakes to:
+
+- manage project dependencies
+- specify:
+
+Parts of our project that touch upon flakes:
+
+- our dependency management
+- how we interface with dependencies
+- how we (might) expose packages at upstream repositories, to facilitate e.g. `nix run`
+- the interface we expose (to users + nixops4)
+  - architectures
+  - formatter
+  - checks
+  - pre-commit hooks
+  - development shell
+  - configuration for nixos / nixops
+
+While its RFC was [drafted](https://github.com/NixOS/rfcs/pull/49) and implemented by Nix creator Eelco Dolstra thru Tweag (which is now divided on flakes) on behalf of Target (which we no longer have indication still uses nix), its aim to address various topics at once - as per their [introduction](https://www.tweag.io/blog/2020-05-25-flakes/#what-problems-do-flakes-solve) including composability, reproducibility, offering a consistent UI, and discoverability - appears to have yielded mixed results, flying in the face of the Unix philosophy, and drawing criticism from various prominent community members, including:
+
+- [samueldr](https://samuel.dionne-riel.com/blog/2023/09/06/flakes-is-an-experiment-that-did-too-much-at-once.html) (author of NixOS Mobile): trying too many things at once (Unix philosophy)
+- [jade](https://jade.fyi/blog/flakes-arent-real/) (Lix core member): locking dependencies of subprojects
+<!-- - [fricklerhandwerk](https://nix.dev/concepts/flakes#why-are-flakes-controversial) (Tweag, head of documentation): no parameterization, defying Unix philosophy -->
+- [raitobezarius](https://discourse.nixos.org/t/nix-flakes-is-an-experiment-that-did-too-much-at-once/32707/3) (board observer, Lix core member): technical issues
+- [infinisil](https://discourse.nixos.org/t/experimental-does-not-mean-unstable-detsyss-perspective-on-nix-flakes/32703/2) (constitutional assembly): dependency explosion, no support to pass arguments, eagerly copying flake directories to the store, bad UX
+- [sandro](https://discourse.nixos.org/t/introducing-flakehub/32044/31) (one of the biggest contributors to nixpkgs): making semantic versioning part of commercial project Flakehub rather than of Nix itself
+- [alyssais](https://github.com/NixOS/rfcs/pull/49#pullrequestreview-262606855) (authored Spectrum OS): inability to patch flakes
+- [tazjin](https://news.ycombinator.com/item?id=40060403) (thru TVL co-authored Tvix): complexity
+- [jtojnar](https://discourse.nixos.org/t/why-are-flakes-still-experimental/29317/12): not awaiting RFC process
+- [piegames](https://discourse.nixos.org/t/introducing-flakehub/32044/52) (formerly member of the moderation team, further [contributed 25 commits to dependency management alternative `npins`](https://github.com/andir/npins/commits?author=piegamesde)): integration into Nix core
+- [solene](https://discourse.nixos.org/t/introducing-flakehub/32044/24): commercial interests
+<!-- - [open letter](https://save-nix-together.org/): undermining community processes by declaring flakes stable -->
+
+Alternatives:
+
+- dependency management: potentially better done using `npins`, which makes this more explicit, whereas flakes offer performance improvements when interacting with a large Nix package repository like Nixpkgs
+- how we interface with dependencies: mostly can be done without flakes, which may in fact help prevent pulling in recursive dependencies we do not use
+- how we (might) expose packages at upstream repositories, to facilitate e.g. `nix run`: no good alternatives for `nix run` exist [currently](https://github.com/NixOS/nix/pull/4702#issuecomment-2233787312), aside from it seeming preferable to defer deviating from the norm here to community RFCs
+- the interface we expose: flakes ignore unstaged files, cache at the cost of eagerly copying flake directories to the store, don't support passing arguments, make it harder to evaluate just part of a project

meeting-notes/2025-02-10 standup notes.md

@@ -0,0 +1,43 @@
+​**Date and Time:** 2025-02-10 09:30   
+
+**Attendees**  
+**Present:** Björn, Hans, Kevin, Kiara, Koen, Richard, Ronny, Gheorghe  
+**Absent:** Valentin (known), Robert(known), Lois (known)  
+
+Roadmap
+https://wiki.fediversity.eu/wiki/Roadmap
+
+Question: Demo deliverable 1? 
+https://git.fediversity.eu/Fediversity/Fediversity/milestone/42
+
+Question: Deliverable 2
+Deadline: Friday 21 Feb
+Demo: Monday 24 Feb 
+
+* Kiara
+	* Had a meeting with Eric on the data model 
+	* Writing an article on flakes 
+	* Limesurvery testing 
+* Richard
+	* No blockers 
+	* Open for new tasks
+* Hans 
+	* No blockers
+	* Matrix playbook is almost ready 
+* Kevin
+	* Will mostly work on Procolix stuff
+	* Not available today 
+* Ronny 
+	* Would like to work on the MVP with Björn 
+* Gheorghe 
+	* No blockers 
+	* Will work on internal artefacts
+	* Will share a new approach in a day or 3 towards this project
+* Koen
+	* Worked on PeerTube server 
+	* No blockers 
+	* Will work on other project today
+	* Tomorrow is the presentation by students working on a project 'how to promote' loops. Will take Lois & Daphne to participate
+	* We're missing people this morning at the latest moment, which is a bit worrying giving the schedule. 
+* Bjorn: 
+	* Team, please pair with Valentin to align on how to realize our MVP in time. 

meeting-notes/2025-02-11 standup notes.md

@@ -0,0 +1,51 @@
+**​Present:** Eric, Hans, Kevin, Lois, Koen, Ronny, Kiara, Valentin, Nicolas, Gheorghe, Bjorn  
+
+* Eric
+	* reviewed data model requirements with Kiara and Gheorghe
+	* current emphasis on what is needed for MVP, with an eye to what is needed transfer between hosting providers
+	* living doc, must evolve with code releases
+	* idealy extracted from code
+	* not a checkbox exercise
+* Hans
+	* No updates 
+	* Hit some roadblocks with the Matrix setup
+	* Will deliver a full Matrix server today
+	* Need to work on the documentation 
+* Kevin
+	* no updates
+	* heared from Niols he broke the forgejo-ci again will see if can fix it today from home 
+* Lois
+	* No updates yet 
+	* Will work with Valentin; available till lunch than other obligations 
+* Koen
+	* no blockers
+	* meeting with Hogeschool Rotterdam this afternoon
+	* discussed stand-up attendance with Richard
+* Ronny
+	* Discussion with Georghe and Bjorn about the MVP, PM and reporting. Reporting will be lighter than Georghe expected. 
+* Kiara
+	* yesterday read up on priorities/decisions
+	* today
+		* will try to get a sense of user stories needed for demo in April
+		* may also meet with Bjorn/Valentin on this
+	* no blockers
+* Valentin
+	* Checked in with Kiara yesterday
+	* Reveiwed new documentation written by Nicolas
+	* Planned to continue pairing with Lois today
+	* Available for questions/discussion/reviews
+* Nicolas
+	* no blockers
+	* updated to latest NixOps4 (with the change from 1 repo 2 branches to 2 repos)
+	* broke the CI machine as promised, unclear what caused this, will investigate with Kevin
+	* wrote some documentation
+* Gheorghe
+	* Yesterday meet Bjorn & Ronny in a fantastic meeting. Start helping on WP5.
+	* Today I'll have an internal very important meeting. COntinue to help WP5.
+	* No Blockers.
+* Bjorn
+	* Worked on the Fediversity presentation 
+	* Had a chat with Gheorghe, Ronny & Koen on the planning / roadmap 
+	* Will work on the presentation a bit, but I also have other obligations today 
+	* Will not be available on Thursday
+	* No  blockers

meeting-notes/2025-02-12 standup notes.md

@@ -0,0 +1,50 @@
+**Standup 2025-02-12 @09:30**  ​
+
+**Present:** Björn, Koen, Eric, Valentin, Gheorghe, Kevin, Lois, Nicholas, Ronny, Hans, Kiara    
+
+* Valentin
+	* Paired with Kiara and Lois to break down issues from user stories -- very fruitful, increased clarity a lot
+		* Reviewed the issues created by Kiara
+	* Internal meetings
+	* Took a look at Robert's exit code PR
+	* Today: Continue pairing with Lois on the Django setup
+		* Need to discuss timining/resource planning, should meet with Bjorn and whoever else has to be involved
+* Eric
+	* brainstorm for MVP scope ideas
+* Kevin
+	* rebooted the forgejo-ci into the previous nix config so nicolas could access it again after he broke it
+* Koen
+	* attended presentation of HogeSchool Rotterdam team (recorded, needs montage)
+	* Remy sent ajusted processing document to EDPS.
+* Lois
+	* Worked with Kiara and Valentin on creating usersstories
+	* Visited de Hogeschool
+* Nicolas
+	* Would like to know which issue to pickup
+	* No blockers 
+	* CI/CD: every change makes it break. Need Kevin
+* Ronny
+	* Had a meeting with Eric in which we described a propasal for the MVP and at least two presentable user stories.
+	* Talked with Bjorn about the divergence in what the MVP is and that we need to point all the noses in the same direction.
+* Gheorghe
+	* No blocker
+	* Yesterday: Internal meetings and exchanged some documents with Eric
+	* Today: Continue with internal meetings and continue helping WP5 
+* Hans
+	* No updates, every thing solved reveals new things to be solved.
+	* Matrix server for fediversity.eu should be ready today.
+* Kiara
+	* Yesterday worked on fleshing out stories
+	* Today will work on:
+		* catch up w/ @robert for orchestrator stories
+		* get clarity on ownership boundaries with hosts together with @koen
+		* BLOCKER?: capacity planning 
+* Robert
+	* Added exit code handling
+	* Solved wiki access with Niols earlier
+	* Apologies for MIA yesterday, will enter availability
+* Bjorn
+	* Did not work on Fediversity presentation 
+	* Had a chat with Kiara
+	* Had a chat with Ronny  
+	* Today available till 14:00	

meeting-notes/2025-02-12-mvp-dev-meeting.md

@@ -0,0 +1,91 @@
+**MVP Dev Meeting 2025-02-12 @10:00**   ​
+**Present:** Björn, Eric, Valentin, Gheorghe, Kevin, Lois, Ronny, Kiara   
+
+## TODO
+* @Bjorn/@RonnyAsk Robert & Nicolas about current state of affairs of the components below on and below NixOPS4 since there might be more available than we think
+* DNS is unclear. **Update** Spoke with Koen about this. There is a redundant DNS available for development. More info soon. 
+* @Bjorn: Need to create a List of definitions / glossary 
+* @Bjorn/@Ronny: Adjust names (Nix  => Fediversity)
+
+## User stories discussion
+
+Operator: the person who is managing the system / users. Need to define more clearly. 
+
+* **Panel incl. API / Central database** 
+    * Team working on this: Lois, Kevin, Kiara (tech lead)       
+	* Python app (Django)
+	* Independent from NixOS for now (will package this later)
+		* Note from Valentin: This will be extremely brittle, and we will aquire notable tech debt that needs to be resolved in order to make it reusable by different providers
+	* First prove the idea, than make it better. We have to be aware of the risk, but we also need to accept some risks.  
+
+* **NixOps4 <--> Central Database**   
+    * Team working on this: Kiara (glue code) (Robert NixOps4) 
+	* Glue code talking JSON with NixOps and whatever we specify as an API with NixPanel
+	* Can be written in any language
+
+* **NixOps4** 
+	* Robert is responsible for this part 
+
+* **Identity Management** 
+	* Out of scope for the MVP 
+	* Use the Django included options for now 
+
+* **Central services (Help needed from Procolix)**
+	* **DNS (Hans?)**
+		* Blocker: DNS is a great unknown
+		* Need an API to create DNS entries
+			* Kevin: Currently using PowerDNS, which has an API
+			* Bjorn: Can we use that for the MVP or will that mess with production?
+				* Kevin: Hans would be the person to ask
+		* Ronny: Koen said it might be easy to spin up another instance for PowerDNS. Development DNS setup is a requirement. 
+	* **Email (Hans?)**
+		* Need to have a development setup 
+	* Storage
+		* is it working? (Niols?)
+		* can it talk to the orchestrator? (Robert)
+		* Garage is working (not supposed to be deployed automatically; is a given just like DNS/Email/Hardware/Virtualisation Provider (Proxmox)) 
+* **Virtualisation**   
+    * Creating VM's in Proxmox is out of scope for the MVP. 
+    * Assumption: VM's with NixOS (+ their root SSH keys) are existing already for the Hello World
+	* Need to have enough VM's available (help from ProcoliX) & these need to be provisioned manually 
+	* NixOps just sends the config and let's NixOS do its thing
+	* Package a Fediverse service (Mastodon/PeerTube/Pixelfed) this is what Nicolas has been working on. 
+
+## Next meeting 
+### Thursday 13-02-2025 @10:00 (directly after standup )   
+**Goal:** Refinement of the issues already in Forgejo based on this meeting we had today.     
+* Ronny will lead this (Bjorn is unavailable Thursday 13th )   
+* Participation from Robert & Nicolas is mandatory. Without them we need to reschedule.  
+
+## Important dates  
+* We will have a demo (~30min) & refinement session (~1.5hrs) of these issues every 2 wks to guide us towards the completion of the MVP after our regular standup so around 10:00.  
+
+* Deliverable 2
+    * Deadline: 2025-02-21
+    * Demo deliverable: 2025-02-24 
+    * Refinement for deliverable 3: 2024-02-24 
+
+* Deliverable 3
+    * Deadline: 2025-03-07
+    * Demo deliverable: 2025-03-10 
+    * Refinement for deliverable 4: 2024-03-10 
+
+* Deliverable 4
+    * Deadline: 2025-03-21
+    * Demo deliverable: 2025-03-24 
+    * Refinement for deliverable 4.5: 2024-03-24 
+
+* Deliverable 4.5 smaller focussed on _"Crossing the t's & dotting the i's"_ 
+    * Deadline: 2025-03-28
+    * Demo deliverable: 2025-03-31 
+
+* MPV Presentation & demo 1-2 April @FediForum. 
+    * Exact time TBD.
+    * Presenter TBD.       
+
+
+
+
+
+
+

meeting-notes/2025-02-13 standup notes.md

@@ -0,0 +1,36 @@
+**Standup 2025-02-13 @09:30**
+
+**Present:** Eric, Valentin, Gheorghe, Kevin, Lois, Nicholas, Ronny, Hans, Kiara    
+Absent: Koen (known), Bjorn (known)
+
+* Ronny
+    * Good meeting yesterday about the MVP  
+* Valentin
+    * Participated in MVP planning yesterday
+    * Added a glossary, Nicolas reviewed
+    * Pushed a Django on NixOS setup (800 LOC as promised) so we can start with reproducibility quickly
+* Georghe
+    * No blockers
+    * Yesterday: At Tweag, a Modus Create company, I performed specific internal project management activities.
+    * Today At Tweag, a Modus Create company, I performed specific internal project management activities.
+* Eric
+    * No blockers
+    * MVP meeting yesterday and todaygit
+    * no other fediversity work planned for today
+* Kiara
+    * Yesterday talked to @roberth about todo, worked on the data model
+    * Today will continue on data model, talk to @niols about todo
+* Kevin
+    * No blockers
+    * helped nicolas with troubleshooting the ci machine its working now may needs some tweaking to make it faster
+    * so probaly be working tuning the ci machine
+* Lois
+    * No blockers
+    * Had a meeting about the upcoming MVP
+* Nicolas
+    * No blockers
+    * Got a configuration working for the test CI machine - successfully ran a CI run, about twice faster as with the previous CI machine (a tweaked VM)
+    * Documentation of services - will open a PR today.
+* Robert
+    * Discussed NixOps related work with Kiara
+

meeting-notes/2025-02-13-mvp-dev-meeting.md

@@ -0,0 +1,7 @@
+**MVP dev meeting 2025-02-13 @10:00**
+
+**Present:** Eric, Valentin, Gheorghe, Kevin, Lois, Nicholas, Ronny, Hans, Kiara    
+
+
+![Image 1](Presentatie_Fediversity_2025-02-13T10_43_50.795Z.png)
+![Image 2](Presentatie_Fediversity_2025-02-13T10_43_58.845Z.png)

meeting-notes/2025-02-14 standup notes.md

@@ -0,0 +1,21 @@
+**​Standup** 2025-02-14 @09:30
+**Present:** Björn, Ronny, Robert, Gheorghe    
+
+Happy Valentin's day!   
+Thank your free software developers & maintainers https://fsfe.org/news/nl/nl-202502.en.html#ilovefs  
+
+* Ronny
+	* Meeting was excellent 
+	* Notes are now uploaded
+	* Spoke with Bjorn & Koen in the afternoon
+* Bjorn
+	* Yesterday had other obligations, but spoke with Ronny & Koen in the afternoon
+	* Today will look at the notes from yesterday
+	* Maybe finish up the presentation 
+* Gheorghe
+	* No blockers
+	* Yesterday: Excelent meeting with all colleagues 
+	* Today: Start to identify the epics and create the user stories with the collagues. Gherkin will be used for testing
+* Robert
+	* Made small PRs to nixops4-nixos for ssh.user, sudo.enable, SSH options
+	* No blockers