fediversity/fediversity
Watch
19
2
Fork
You've already forked fediversity
 10
Code Issues 92 Pull requests 24 Projects Activity Actions

schema updates automated #629

New issue
Open
opened 2025-12-03 15:28:19 +01:00 by kiara · 0 comments
kiara commented 2025-12-03 15:28:19 +01:00
Owner
Copy link

As a Fediversity developer,
I want to use available information on option changes,
so that I may use these to facilitate schema updates (#159).

notes

  • in case of options coming from Nix (rather than JSON-Schema), should cover various operations (ideally extracted from e.g. markers, or otherwise, see #628) - closed in #1073
    • alias/rename markers reconstructed for e.g. each stable release (for NixOS), to upgrade users who otherwise skipped major updates
  • could be deferred to put this burden on the operator (#214).
  • updating deployments across Fediversity releases may, aside from updates to deployed software, bring new interfaces of Fediversity deployment configuration, either general or stemming from updated application interfaces. these can be handled similar to package update migrations to migrate across.
  • migrating data across schema version migrations either forward or backward by default implies losing info (-> strict policy + full compatibility type = no ✔).

preserving data across schema version migrations requires:

  • use of identity-tracking info to handle key renames
  • refusing to remove information (forward delete, creation backward)

we need to keep a human (-> Fediversity maintainer) in the loop to tracking identity, or verify how to handle potential removal of parts of state, if maintaining everything turns out not viable.
since identity tracking cannot be inferred, version migration actions should likely be published through a repo on our end.
identify candidate matches on renames requires calculating string similarity across the addition/removal grid for option key paths and options' type definitions.

implementation notes

ideas to tackle this:

  • string similarity: e.g. by fuzz.token_sort_ratio
  • handle human confirmation using e.g. an editable PR containing a CSV with columns 'additions' and 'removals', with values containing key-path (of either individual options or namespaces aka submodules), with items on the same line indicating high string similarity, with rows sorted by such string similarity - such that a user would e.g. only need to realign matches and remove rows containing key-paths not part of legitimate matches.
    • while spreadsheets might make for a convenient UI for a human in the loop, this may not be desirable as a final storage format
    • how to present changes in namespace? match not just by name but also by value? and how to suggest/present/address namespace changes when a child value changed as well (but differently)?
  • JSON schema:
    • needs applications' JSON-Schemas (#627 - implemented for peertube in #1384)
    • given tracking identity is non-trivial (tho suggestions could likely be generated based from addition/removal diffs), mechanical libraries detect adds/removes but skip renames, e.g. json-schema-diff libraries:
  • UI schemas (#628) from back-end ported across schema migrations, whether statically (pre-calculate at the cost of storing duplicates) or dynamically (redundant calculations alleviated by caching)
  • custom deployment configuration version migration scripts based on diffs?
  • per update store the rename mapping, optionally removals/additions / type changes (could be inferred, but may be useful for migrating backward (?))
  • first-class support for namespace renames is relevant to reduce notification spam in changes UI, but for the purpose of migrating configuration values across updates, removals must be handled before such renames so that we can handle namespaces as namespaces for the notifications even if for the purpose of configuration update migration some value(s) got removed
  • child renames must similarly be handled before parent renames for migration, if distinct. handle by e.g. listing these alphabetically (so short first) while processing them in reverse?
  • migrate values of enum types not unlike correlating and migrating option keys - maybe having some markers like for aliasing nix options on rename?
NixOS option modules - ~~[ ] [NixOS option modules](github.com/NixOS/nixpkgs@511f22afbf/lib/modules.nix (L2203-L2226)): warns or errors about changes - errors of which should likely be caught in tests, presuming proper coverage, failing which we may need to keep track of service module changes.~~ - ~~we may be able to figure out what part of NixOS configurations we use, such as to get a sense of what changes to a schema (/ options modules) may be relevant for us.~~
**As** a Fediversity developer, **I want** to use available information on option changes, **so that** I may use these to facilitate schema updates (#159). ## notes - [x] in case of options coming from Nix (rather than JSON-Schema), should cover various [operations](https://docs.confluent.io/platform/7.9/schema-registry/fundamentals/schema-evolution.html#summary) (ideally extracted from e.g. [markers](https://github.com/NixOS/nixpkgs/blob/511f22afbfaccda862e13f8f2441c717bc962e89/lib/modules.nix#L2203-L2226), or otherwise, see #628) - closed in #1073 - [ ] alias/rename markers reconstructed for e.g. each stable release (for NixOS), to upgrade users who otherwise skipped major updates - could be deferred to put this burden on the operator (#214). - updating deployments across Fediversity releases may, aside from updates to deployed software, bring new interfaces of Fediversity deployment configuration, either general or stemming from updated application interfaces. these can be handled similar to package update migrations to migrate across. - migrating data across schema version migrations either [forward or backward](https://docs.confluent.io/platform/7.9/schema-registry/fundamentals/schema-evolution.html) by default implies losing info (-> strict policy + full compatibility type = no ✔). preserving data across schema version migrations requires: - use of identity-tracking info to handle key renames - refusing to remove information (forward delete, creation backward) we need to keep a human (-> Fediversity maintainer) in the loop to tracking identity, or verify how to handle potential removal of parts of state, if maintaining everything turns out not viable. since identity tracking cannot be inferred, version migration actions should likely be published through a repo on our end. identify candidate matches on renames requires calculating string similarity across the addition/removal grid for option key paths and options' type definitions. ### implementation notes ideas to tackle this: - [ ] string similarity: e.g. by [`fuzz.token_sort_ratio`](https://pypi.org/project/thefuzz/#token-sort-ratio) - [ ] handle human confirmation using e.g. an editable PR containing a CSV with columns 'additions' and 'removals', with values containing key-path (of either individual options or namespaces aka submodules), with items on the same line indicating high string similarity, with rows sorted by such string similarity - such that a user would e.g. only need to realign matches and remove rows containing key-paths not part of legitimate matches. - while spreadsheets might make for a convenient UI for a human in the loop, this may not be desirable as a final storage format - how to present changes in namespace? match not just by name but also by value? and how to suggest/present/address namespace changes when a child value changed as well (but differently)? - [ ] JSON schema: - needs applications' JSON-Schemas (#627 - implemented for peertube in #1384) - given tracking identity is non-trivial (tho suggestions could likely be generated based from addition/removal diffs), mechanical libraries detect adds/removes but skip renames, e.g. `json-schema-diff` libraries: - [getsentry's](https://github.com/getsentry/json-schema-diff) - [atlassian's](https://bitbucket.org/atlassian/json-schema-diff/src/master/): keyword support [partial](https://bitbucket.org/atlassian/json-schema-diff/src/c194e9836069546b8dab3243ce82ee8c87e235e9/KEYWORDS.md) - [ ] UI schemas (#628) from back-end ported across schema migrations, whether statically (pre-calculate at the cost of storing duplicates) or dynamically (redundant calculations alleviated by caching) - [ ] custom deployment configuration version migration scripts based on diffs? - [ ] per update store the rename mapping, optionally removals/additions / type changes (could be inferred, but may be useful for migrating backward (?)) - [ ] first-class support for namespace renames is relevant to reduce notification spam in changes UI, but for the purpose of migrating configuration values across updates, removals must be handled before such renames so that we can handle namespaces as namespaces for the notifications even if for the purpose of configuration update migration some value(s) got removed - [ ] child renames must similarly be handled before parent renames for migration, if distinct. handle by e.g. listing these alphabetically (so short first) while processing them in reverse? - [ ] migrate values of `enum` types not unlike correlating and migrating option keys - maybe having some markers like for aliasing nix options on rename? <details> <summary> NixOS option modules </summary> - ~~[ ] [NixOS option modules](https://github.com/NixOS/nixpkgs/blob/511f22afbfaccda862e13f8f2441c717bc962e89/lib/modules.nix#L2203-L2226): warns or errors about changes - errors of which should likely be caught in tests, presuming proper coverage, failing which we may need to keep track of service module changes.~~ - ~~we may be able to figure out what part of NixOS configurations we use, such as to get a sense of what changes to a schema (/ options modules) may be relevant for us.~~ </details>
kiara changed title from use schemas to facilitate schema updates to schema updates automated 2025-12-03 17:03:02 +01:00
toonn added this to the Dev project 2026-08-04 12:03:34 +02:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
panel-docs
repro-dev3
netbox-location
operator-group-comes-up
ci-var-image-fingerprint-null
deploy-report-degraded
automation/npins-update/systemd-openbao
automation/npins-update/nixpkgs
deprecate-deployments
jsonschema-frontend
update-nixpkgs-forgejo-16
fix-git-pages-native-cli
automation/npins-update/nixpkgs-unstable
automation/npins-update/forgejo-src
promote-updater
test-action-main
host-garage-central
demo-new
demo-old
schema-change-visuals
schema-change-viz
ci/panel-incus
automation/npins-update/terraform-provider-proxmox
automation/npins-update/nix-src
automation/npins-update/nix
automation/npins-update/flake-parts
automation/npins-update/clan-core
kiara/nspawn-rebase
fix/nixpkgs-contracts-26-05
feat/garage-to-hosting-provider
feat/authelia-ancilliary
kiara/nspawn-panel-fix
kiara/nspawn
issues/124-database-backups
fix/forgejo-ci-secrets-rekey
fix/forgejo-ci-network-gateway
fix/forgejo-ci-ipv6-gateway-interface
feat/tests-thru-flake
issue/126-filesystem-backup
automation/update-workflow
kiara/fix-pixelfed
issues/125-garage-backups
ci/pixelfed
pixelfed-ci
No results found.
Labels
Clear labels   
0 points
fibonacci complexity: 0 points

  
0.5 points
fibonacci complexity: 0.5 points

  
1 point
fibonacci complexity: 1 point

  
13 points
fibonacci complexity: 13 points

  
2 points
fibonacci complexity: 2 points

  
21 points
fibonacci complexity: 21 points

  
3 points
fibonacci complexity: 3 points

  
34 points
fibonacci complexity: 34 points

  
5 points
fibonacci complexity: 5 points

  
55 points
fibonacci complexity: 55 points

  
8 points
fibonacci complexity: 8 points

  
ambition
application-offering

  
ambition
configure-applications

  
ambition
front-end

  
ambition/install-applications
  
ambition
security

  
ambition
switch-host

  
ambition
update-applications

  
ambition
user-management

  
api service
features that may relate more to a (to be created) API service

  
blocked

   
component: fediversity panel

   
component: nixops4

   
documentation

   
estimation high: >3d

   
estimation low: <2h

   
estimation mid: <8h

   
infinite points
fibonacci complexity: infinite points

  
productisation
features out-scoped with the decoupling of the productisation effort

  
project-management
Project management related tasks

  
question

   
role: application developer

   
role: application operator

   
role: hosting provider

   
role: maintainer

   
security

   
technical debt

   
testing

   
type unclear

   
type: bug
Anything that doesn't work as advertised

  
type: deliverable

   
type: key result

   
type: objective

   
type: task

   
type: user story

   
user experience
No labels
0 points
0.5 points
1 point
13 points
2 points
21 points
3 points
34 points
5 points
55 points
8 points
ambition
application-offering
ambition
configure-applications
ambition
front-end
ambition/install-applications
ambition
security
ambition
switch-host
ambition
update-applications
ambition
user-management
api service
blocked
component: fediversity panel
component: nixops4
documentation
estimation high: >3d
estimation low: <2h
estimation mid: <8h
infinite points
productisation
project-management
question
role: application developer
role: application operator
role: hosting provider
role: maintainer
security
technical debt
testing
type unclear
type: bug
type: deliverable
type: key result
type: objective
type: task
type: user story
user experience
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Dev
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#289 key features improving user experience supported
fediversity/fediversity
#610 Configure applications
fediversity/fediversity
Reference
fediversity/fediversity#629
No description provided.