add draft requirements on the migration data model for NLNet
This commit is contained in:
Kiara Grouwstra
parent
346c3add5a
commit
9a24f91976
1 changed files with 126 additions and 0 deletions
126
architecture-docs/data-model-requirements.md
Normal file
126
architecture-docs/data-model-requirements.md
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
# migration data model requirements
|
||||
|
||||
To transfer between two providers, the target provider must be able to import the sending provider's versions. (e.g.: a deployment may have latest fediversity, latest pixelfed, but previous mastadon) Thus, for each "realease" of the data model, it needs to be versioned, and applications/APIs also are versioned.
|
||||
* (May need a way to show on the front-end which versions are in place, and which migrations are supported. However, for application versions which are completely controlled by the installation and setup, this is "solved".)
|
||||
|
||||
for release version 0, focus on known current needs
|
||||
* to be expanded later as each new application is added and can be transfered between providers
|
||||
* review migration guides for the known apps with an eye to odd/unusual details that influence design choices (task for Niols? others?)
|
||||
|
||||
Specifically, this suggests scoping to migrating:
|
||||
|
||||
- managed infrastructure (rather than managed applications)
|
||||
- between servers owned by procolix
|
||||
- same proxmox version
|
||||
- NixOS VMs set up by us so we can guarantee identical application versions
|
||||
- hosting limited to a single application (to start)
|
||||
|
||||
First, a bit of an inventory (list without much structure now, later will create structured form/schema with e.g. many-to-many links, useful for the migration code):
|
||||
* clearly mark items that will not be in the first migration as eventually or speculative
|
||||
* or reamove them if they would be too far in the future
|
||||
* later we understand what is useful for migration code, we can extract and transform in to a format suitable as data model documentation
|
||||
|
||||
Hosting Provider provides:
|
||||
* proxmox, git
|
||||
* hardware
|
||||
* filesystem storage
|
||||
* DNS automation hooks?
|
||||
* central/shared garage storage or only hardware+diskspace for the garage VMs to create storage?
|
||||
* with central: more efficient but less isolated
|
||||
|
||||
FooUniversity (Operator)
|
||||
* invoice info
|
||||
* is all info expected to be transfered from provider A to provider B?
|
||||
* May not want to transfer e.g. bank details, because they are already set up at B
|
||||
* May also depend on regulation (which information are you allowed to hand out?)
|
||||
* Admins:
|
||||
* credentials
|
||||
* persistent identifiers
|
||||
* mappings between them (also need to travel across providers)
|
||||
* e.g. if we can't change content URLs, we may need to create (and from then on carry around) a redirects mapping
|
||||
* those mappings are likely application-specific, but they all belong to the same type class
|
||||
* domain(s)
|
||||
* what is needed for DNS management?
|
||||
* users
|
||||
* display name
|
||||
* email(s)
|
||||
* login id
|
||||
* oauth2 (eventually)
|
||||
* 2fa
|
||||
* password
|
||||
* passkeys (eventually)
|
||||
* LDAP? (eventually?)
|
||||
* all applications
|
||||
* sub domain ( social.example.org vs example.org/social )
|
||||
* info for proxmox setup such as to provision VMs (to reproduce proxmox )
|
||||
* mem
|
||||
* cpus
|
||||
* storage mounts
|
||||
* IPs likely not the same in the target network
|
||||
* storage
|
||||
* filesystem
|
||||
* very well specified per application
|
||||
* blob storage config (garage, s3-like)
|
||||
* index
|
||||
* Can we make it a requirement that Garage is behind a predictable URL, eg. `<application>.garage.<customer domain>`? As opposed to something vendor-specific, eg. `pixelfed-university.garage.procolix.com/<customer domain>/<application>`
|
||||
* may need to rewrite URLs to blobs automatically, depending on the underlying URL scheme, which may be per setup or application
|
||||
* limits? per application? per user? where are these used/set/enforced?
|
||||
* TODO: what does e.g. borgmatic need to back up storage?
|
||||
* out of scope?: focus on actual state, disregarding reconstructable stuff
|
||||
* SQL database
|
||||
* dump/snapshot
|
||||
* TODO: what does e.g. borgmatic need to back up databases?
|
||||
* application specifics
|
||||
* postfix? (is email in version 0?)
|
||||
* pixelfed
|
||||
* where is blob storage
|
||||
* in the specific case of Pixelfed, if blob storage changed URL, we might need to rewrite the pictures URLs in the database (try to avoid this)
|
||||
* redis (in the case of pixelfed, it is not just a cache)
|
||||
* misc config: theme, name of instance, email of sysadmin
|
||||
* database
|
||||
* on-disk files
|
||||
* Daniel Supernault is currently making it so evertying can be stored remotely in a garage or sql database
|
||||
* users (login id) (in database? in redis?)
|
||||
* user preferences/settings
|
||||
* peertube
|
||||
* mastodon
|
||||
* matrix? (is it in version 0?)
|
||||
* logos
|
||||
|
||||
Other considerations:
|
||||
|
||||
- Put a boundary for what is
|
||||
- operator-configurable
|
||||
- needs to get fixed, but at the implementation level
|
||||
- what can be configured dynamically per environment
|
||||
- Most importantly we need to preserve persistent identifiers
|
||||
|
||||
- When transforming the data-model code to a deliverable version of the data model as part of the technical architecture document, documenting user-data storage and with respects fot security and GDPR
|
||||
|
||||
See also:
|
||||
|
||||
- possible overlap/inspiration: Stalw.art [configuration docs](https://stalw.art/docs/server/general)
|
||||
|
||||
## MVP scoping ideas
|
||||
|
||||
User story 1: New customer
|
||||
When a new customer goes to the Fediversity website we want to show that user what Fediversity is all about and what it can give to the customer. This points the customer to a signup form where they can enter all the details that are needed to get it working. Here they can also decide what applications to use (at first no more than three). Details can be, the user/admin login, name, address, bank details, domain, other users, and applications. Than when the customer hits the install/provision/go button everything starts to install automagically. After which the customer is presented with (some) url's to login to.
|
||||
|
||||
User story 2: Take out / move to other instance
|
||||
At any time a customer may wish to change service providers. They can easily go to an admin screen where they can get their configuration and data packaged for transfer. This packaged data can be provided to a new service provider where they will be up-and-running again easily, with minimal downtime.
|
||||
|
||||
proposed MVP scope:
|
||||
- block storage
|
||||
- blob storage (garage)
|
||||
- physical servers
|
||||
- proxmox vm management
|
||||
- nixops service
|
||||
- nixops scripts
|
||||
- 1 to 3 applications packaged in Nix (Mastodon, Peertube, Pixelfed)
|
||||
- frontend / website
|
||||
- working dns, can be external, but automated
|
||||
- takeout area
|
||||
- import area
|
||||
- 2 Fediversity environments to transfer between
|
||||
- demonstration of User story 1
|
||||
- demonstration of User story 2
|
||||
Loading…
Add table
Reference in a new issue