From f9779723ed87496c4e7648c32d4b5b6c5019ad6c Mon Sep 17 00:00:00 2001 From: Kiara Grouwstra Date: Tue, 4 Mar 2025 13:02:52 +0100 Subject: [PATCH 1/5] wip: data model requirements --- architecture-docs/data-model-requirements.md | 100 +++++-------------- json-schema.yaml | 12 +++ 2 files changed, 39 insertions(+), 73 deletions(-) create mode 100644 json-schema.yaml diff --git a/architecture-docs/data-model-requirements.md b/architecture-docs/data-model-requirements.md index d9e79fa..b528631 100644 --- a/architecture-docs/data-model-requirements.md +++ b/architecture-docs/data-model-requirements.md @@ -1,7 +1,13 @@ # migration data model requirements -To transfer between two providers, the target provider must be able to import the sending provider's versions. (e.g.: a deployment may have latest fediversity, latest pixelfed, but previous mastadon) Thus, for each "realease" of the data model, it needs to be versioned, and applications/APIs also are versioned. - * (May need a way to show on the front-end which versions are in place, and which migrations are supported. However, for application versions which are completely controlled by the installation and setup, this is "solved".) +(updated) deployment incl variables, backup creation/restore + + + + +Assumptions: + +* Our deployment fully controls all versions, bypassing concerns on version mismatches. for release version 0, focus on known current needs * to be expanded later as each new application is added and can be transferred between providers @@ -10,101 +16,49 @@ for release version 0, focus on known current needs Specifically, this suggests scoping to migrating: - managed infrastructure (rather than managed applications) -- between servers owned by procolix +- between servers initially owned by procolix - same proxmox version - NixOS VMs set up by us so we can guarantee identical application versions - hosting limited to a single application (to start) +- retaining the same domain name +- migrating the applications rather than also say control of domains First, a bit of an inventory (list without much structure now, later will create structured form/schema with e.g. many-to-many links, useful for the migration code): - * clearly mark items that will not be in the first migration as eventually or speculative - * or reamove them if they would be too far in the future - * later we understand what is useful for migration code, we can extract and transform in to a format suitable as data model documentation +* clearly mark items that will not be in the first migration as planned for later or speculative +* or remove them if they would be too far in the future +* later we understand what is useful for migration code, we can extract and transform in to a format suitable as data model documentation Hosting Provider provides: + * proxmox, git * hardware * filesystem storage -* DNS automation hooks? +* DNS automation hooks (RFC 2136, optionally authenticated by TSIG (RFC 2845) or GSS-TSIG (RFC 3645)) * central/shared garage storage or only hardware+diskspace for the garage VMs to create storage? * with central: more efficient but less isolated FooUniversity (Operator) -* invoice info - * is all info expected to be transferred from provider A to provider B? - * May not want to transfer e.g. bank details, because they are already set up at B - * May also depend on regulation (which information are you allowed to hand out?) -* Admins: - * credentials -* persistent identifiers - * mappings between them (also need to travel across providers) - * e.g. if we can't change content URLs, we may need to create (and from then on carry around) a redirects mapping - * those mappings are likely application-specific, but they all belong to the same type class * domain(s) - * what is needed for DNS management? - * users - * display name - * email(s) - * login id - * oauth2 (eventually) - * 2fa - * password - * passkeys (eventually) - * LDAP? (eventually?) - * all applications - * sub domain ( social.example.org vs example.org/social ) - * info for proxmox setup such as to provision VMs (to reproduce proxmox ) - * mem - * cpus - * storage mounts - * IPs likely not the same in the target network - * storage - * filesystem - * very well specified per application - * blob storage config (garage, s3-like) - * index - * Can we make it a requirement that Garage is behind a predictable URL, eg. `.garage.`? As opposed to something vendor-specific, eg. `pixelfed-university.garage.procolix.com//` + * may need to rewrite URLs to blobs automatically, depending on the underlying URL scheme, which may be per setup or application * limits? per application? per user? where are these used/set/enforced? - * TODO: what does e.g. borgmatic need to back up storage? + * TODO: what does e.g. borgmatic need to back up? * out of scope?: focus on actual state, disregarding reconstructable stuff - * SQL database - * dump/snapshot - * TODO: what does e.g. borgmatic need to back up databases? -* application specifics - * postfix? (is email in version 0?) + * pixelfed - * where is blob storage - * in the specific case of Pixelfed, if blob storage changed URL, we might need to rewrite the pictures URLs in the database (try to avoid this) - * redis (in the case of pixelfed, it is not just a cache) - * misc config: theme, name of instance, email of sysadmin - * database - * on-disk files - * Daniel Supernault is currently making it so evertying can be stored remotely in a garage or sql database - * users (login id) (in database? in redis?) - * user preferences/settings - * peertube - * mastodon - * matrix? (is it in version 0?) -* logos + + -Other considerations: + When transforming the data-model code to a deliverable version of the data model as part of the technical architecture document, documenting user-data storage and with respects to security and GDPR -- Put a boundary for what is - - operator-configurable - - needs to get fixed, but at the implementation level - - what can be configured dynamically per environment -- Most importantly we need to preserve persistent identifiers + -- When transforming the data-model code to a deliverable version of the data model as part of the technical architecture document, documenting user-data storage and with respects fot security and GDPR - -See also: - -- possible overlap/inspiration: Stalw.art [configuration docs](https://stalw.art/docs/server/general) + ## MVP scoping ideas User story 1: New customer -When a new customer goes to the Fediversity website we want to show that user what Fediversity is all about and what it can give to the customer. This points the customer to a signup form where they can enter all the details that are needed to get it working. Here they can also decide what applications to use (at first no more than three). Details can be, the user/admin login, name, address, bank details, domain, other users, and applications. Than when the customer hits the install/provision/go button everything starts to install automagically. After which the customer is presented with (some) url's to login to. +When a new customer goes to the Fediversity website we want to show that user what Fediversity is all about and what it can give to the customer. This points the customer to a signup form where they can enter all the details that are needed to get it working. Here they can also decide what applications to use (at first no more than three). Details can be, the admin login, domain, and applications. Then when the customer confirms everything starts to install automagically, after which the customer is presented with (some) url's to login to. User story 2: Take out / move to other instance At any time a customer may wish to change service providers. They can easily go to an admin screen where they can get their configuration and data packaged for transfer. This packaged data can be provided to a new service provider where they will be up-and-running again easily, with minimal downtime. @@ -114,8 +68,8 @@ proposed MVP scope: - blob storage (garage) - physical servers - proxmox vm management -- nixops service -- nixops scripts + + - 1 to 3 applications packaged in Nix (Mastodon, Peertube, Pixelfed) - frontend / website - working dns, can be external, but automated diff --git a/json-schema.yaml b/json-schema.yaml new file mode 100644 index 0000000..481a4c8 --- /dev/null +++ b/json-schema.yaml @@ -0,0 +1,12 @@ +--- +type: object +properties: + # version: + # type: number + applications: + type: object + additionalProperties: + type: object + properties: + # version: + # type: number From cd0a53200a0b7dd1f807b23e0218f40f0d39490a Mon Sep 17 00:00:00 2001 From: Kiara Grouwstra Date: Tue, 25 Mar 2025 13:52:08 +0100 Subject: [PATCH 2/5] migration: add concern raised by @jan --- architecture-docs/data-model-requirements.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/architecture-docs/data-model-requirements.md b/architecture-docs/data-model-requirements.md index b528631..6c50ffb 100644 --- a/architecture-docs/data-model-requirements.md +++ b/architecture-docs/data-model-requirements.md @@ -43,6 +43,8 @@ FooUniversity (Operator) * may need to rewrite URLs to blobs automatically, depending on the underlying URL scheme, which may be per setup or application * limits? per application? per user? where are these used/set/enforced? * TODO: what does e.g. borgmatic need to back up? + * complications: in case details such as connections change, those may need adjusting, implying application-specification reconfiguring + * potentially propagate thru by e.g. TF? * out of scope?: focus on actual state, disregarding reconstructable stuff * pixelfed From 1aeece12a0f5262cabb7b6c649979a29b6b7382f Mon Sep 17 00:00:00 2001 From: Kiara Grouwstra Date: Tue, 8 Apr 2025 09:12:40 +0200 Subject: [PATCH 3/5] data migration: add note on selfhostblocks --- architecture-docs/data-model-requirements.md | 1 + 1 file changed, 1 insertion(+) diff --git a/architecture-docs/data-model-requirements.md b/architecture-docs/data-model-requirements.md index 6c50ffb..df79b44 100644 --- a/architecture-docs/data-model-requirements.md +++ b/architecture-docs/data-model-requirements.md @@ -43,6 +43,7 @@ FooUniversity (Operator) * may need to rewrite URLs to blobs automatically, depending on the underlying URL scheme, which may be per setup or application * limits? per application? per user? where are these used/set/enforced? * TODO: what does e.g. borgmatic need to back up? + * note that selfhostblocks may already have some of the needed info here * complications: in case details such as connections change, those may need adjusting, implying application-specification reconfiguring * potentially propagate thru by e.g. TF? * out of scope?: focus on actual state, disregarding reconstructable stuff From 5b401313198d42437cb1986f3b23a1244efa26a6 Mon Sep 17 00:00:00 2001 From: cinereal Date: Tue, 27 May 2025 18:47:21 +0200 Subject: [PATCH 4/5] update data model requirements as per recent insights --- architecture-docs/data-model-requirements.md | 97 +++++--------------- 1 file changed, 21 insertions(+), 76 deletions(-) diff --git a/architecture-docs/data-model-requirements.md b/architecture-docs/data-model-requirements.md index df79b44..01bdb81 100644 --- a/architecture-docs/data-model-requirements.md +++ b/architecture-docs/data-model-requirements.md @@ -1,83 +1,28 @@ # migration data model requirements -(updated) deployment incl variables, backup creation/restore +Given: - - +- no change in control of domains; +- two Fediversity set-ups (to be provided by ProcoliX) with a run-time environment such as ProxmoX, for an initial test using the same version; +- an operator's configuration, including: + - DNS automation hooks for the desired domain (RFC 2136, optionally authenticated by TSIG (RFC 2845) or GSS-TSIG (RFC 3645)); + - a Fediversity configuration of at least a single application (to start). -Assumptions: +Our data model must describe a migration: -* Our deployment fully controls all versions, bypassing concerns on version mismatches. +- specifying [entity relations](https://mermaid.js.org/syntax/entityRelationshipDiagram.html#relationship-syntax) e.g. many-to-many; +- migrating both deployed and staged configurations; +- deploying of applications using the same versions; +- retaining relevant application state; +- handling of application-specific migration logic, such as to rewrite URLs as needed; -for release version 0, focus on known current needs - * to be expanded later as each new application is added and can be transferred between providers - * review migration guides for the known apps with an eye to odd/unusual details that influence design choices (task for Niols? others?) +Tests: -Specifically, this suggests scoping to migrating: - -- managed infrastructure (rather than managed applications) -- between servers initially owned by procolix -- same proxmox version -- NixOS VMs set up by us so we can guarantee identical application versions -- hosting limited to a single application (to start) -- retaining the same domain name -- migrating the applications rather than also say control of domains - -First, a bit of an inventory (list without much structure now, later will create structured form/schema with e.g. many-to-many links, useful for the migration code): -* clearly mark items that will not be in the first migration as planned for later or speculative -* or remove them if they would be too far in the future -* later we understand what is useful for migration code, we can extract and transform in to a format suitable as data model documentation - -Hosting Provider provides: - -* proxmox, git -* hardware -* filesystem storage -* DNS automation hooks (RFC 2136, optionally authenticated by TSIG (RFC 2845) or GSS-TSIG (RFC 3645)) -* central/shared garage storage or only hardware+diskspace for the garage VMs to create storage? - * with central: more efficient but less isolated - -FooUniversity (Operator) -* domain(s) - - * may need to rewrite URLs to blobs automatically, depending on the underlying URL scheme, which may be per setup or application - * limits? per application? per user? where are these used/set/enforced? - * TODO: what does e.g. borgmatic need to back up? - * note that selfhostblocks may already have some of the needed info here - * complications: in case details such as connections change, those may need adjusting, implying application-specification reconfiguring - * potentially propagate thru by e.g. TF? - * out of scope?: focus on actual state, disregarding reconstructable stuff - - * pixelfed - - - - When transforming the data-model code to a deliverable version of the data model as part of the technical architecture document, documenting user-data storage and with respects to security and GDPR - - - - - -## MVP scoping ideas - -User story 1: New customer -When a new customer goes to the Fediversity website we want to show that user what Fediversity is all about and what it can give to the customer. This points the customer to a signup form where they can enter all the details that are needed to get it working. Here they can also decide what applications to use (at first no more than three). Details can be, the admin login, domain, and applications. Then when the customer confirms everything starts to install automagically, after which the customer is presented with (some) url's to login to. - -User story 2: Take out / move to other instance -At any time a customer may wish to change service providers. They can easily go to an admin screen where they can get their configuration and data packaged for transfer. This packaged data can be provided to a new service provider where they will be up-and-running again easily, with minimal downtime. - -proposed MVP scope: -- block storage -- blob storage (garage) -- physical servers -- proxmox vm management - - -- 1 to 3 applications packaged in Nix (Mastodon, Peertube, Pixelfed) -- frontend / website -- working dns, can be external, but automated -- takeout area -- import area -- 2 Fediversity environments to transfer between -- demonstration of User story 1 -- demonstration of User story 2 +1. + A Fediversity user may wish to migrate their Fediversity set-up between monolithic and distributed configurations. + In an admin screen they can get their configuration and data for transfer. + Using this they may migrate to the desired configuration. +1. + At any time a Fediversity user may wish to migrate their Fediversity set-up. + They can go to an admin screen where they can get their configuration and data for transfer. + This data can be provided to a new service provider where they will be up-and-running again, with minimal downtime. From 4234ff4b335df9836a2b7e6586b75310c450a0dd Mon Sep 17 00:00:00 2001 From: cinereal Date: Tue, 27 May 2025 19:07:50 +0200 Subject: [PATCH 5/5] rm wip json schema again --- json-schema.yaml | 12 ------------ 1 file changed, 12 deletions(-) delete mode 100644 json-schema.yaml diff --git a/json-schema.yaml b/json-schema.yaml deleted file mode 100644 index 481a4c8..0000000 --- a/json-schema.yaml +++ /dev/null @@ -1,12 +0,0 @@ ---- -type: object -properties: - # version: - # type: number - applications: - type: object - additionalProperties: - type: object - properties: - # version: - # type: number