schema updates automated #629

New issue

Open

opened 2025-12-03 15:28:19 +01:00 by kiara · 0 comments

kiara commented

2025-12-03 15:28:19 +01:00

Owner

As a Fediversity developer,
I want to use available information on option changes,
so that I may use these to facilitate schema updates (#159).

notes

could be deferred to put this burden on the operator (#214).
updating deployments across Fediversity releases may, aside from updates to deployed software, bring new interfaces of Fediversity deployment configuration, either general or stemming from updated application interfaces. these can be handled similar to package update migrations to migrate across.
migrating data across schema version migrations both forward and backward by default implies losing info (-> strict policy + full compatibility type = no ✔). preserving data across forward schema version migrations works until fields (or used values) are removed. if we want to preserve data across backward migrations as well tho, this requires use of identity-tracking info, e.g. for key renames.
as breaking changes are non-trivial, given we may not always be able to track identity, and new (required) options may need us to reason about sane defaults, we may want to keep a human (-> Fediversity maintainer) in the loop for breaking changes or even any deletions of parts of state.
since identity tracking cannot be inferred, version migration actions should likely be published through a repo on our end.

implementation notes

ideas to tackle this:

JSON schema:
- may need application schemas (#627)
- given tracking identity is non-trivial (tho suggestions could likely be generated based from addition/removal diffs), mechanical libraries detect adds/removes but skip renames, e.g. json-schema-diff libraries:
  - getsentry's
  - atlassian's: keyword support partial
custom deployment configuration version migration scripts based on diffs?

NixOS option modules

- ~~[ ] [NixOS option modules](511f22afbf/lib/modules.nix (L2203-L2226)): warns or errors about changes - errors of which should likely be caught in tests, presuming proper coverage, failing which we may need to keep track of service module changes.~~ - ~~we may be able to figure out what part of NixOS configurations we use, such as to get a sense of what changes to a schema (/ options modules) may be relevant for us.~~

**As** a Fediversity developer, **I want** to use available information on option changes, **so that** I may use these to facilitate schema updates (#159). ## notes - could be deferred to put this burden on the operator (#214). - updating deployments across Fediversity releases may, aside from updates to deployed software, bring new interfaces of Fediversity deployment configuration, either general or stemming from updated application interfaces. these can be handled similar to package update migrations to migrate across. - migrating data across schema version migrations both [forward and backward](https://docs.confluent.io/platform/7.9/schema-registry/fundamentals/schema-evolution.html) by default implies losing info (-> strict policy + full compatibility type = no ✔). preserving data across forward schema version migrations works until fields (or used values) are removed. if we want to preserve data across backward migrations as well tho, this requires use of identity-tracking info, e.g. for key renames. - as breaking changes are non-trivial, given we may not always be able to track identity, and new (required) options may need us to reason about sane defaults, we may want to keep a human (-> Fediversity maintainer) in the loop for breaking changes or even any deletions of parts of state. - since identity tracking cannot be inferred, version migration actions should likely be published through a repo on our end. ### implementation notes ideas to tackle this: - [ ] JSON schema: - may need application schemas (#627) - given tracking identity is non-trivial (tho suggestions could likely be generated based from addition/removal diffs), mechanical libraries detect adds/removes but skip renames, e.g. `json-schema-diff` libraries: - [getsentry's](https://github.com/getsentry/json-schema-diff) - [atlassian's](https://bitbucket.org/atlassian/json-schema-diff/src/master/): keyword support [partial](https://bitbucket.org/atlassian/json-schema-diff/src/c194e9836069546b8dab3243ce82ee8c87e235e9/KEYWORDS.md) - [ ] custom deployment configuration version migration scripts based on diffs? <details> <summary> NixOS option modules </summary> - ~~[ ] [NixOS option modules](https://github.com/NixOS/nixpkgs/blob/511f22afbfaccda862e13f8f2441c717bc962e89/lib/modules.nix#L2203-L2226): warns or errors about changes - errors of which should likely be caught in tests, presuming proper coverage, failing which we may need to keep track of service module changes.~~ - ~~we may be able to figure out what part of NixOS configurations we use, such as to get a sense of what changes to a schema (/ options modules) may be relevant for us.~~ </details>