Fediversity/meta
13
2
Fork 4
Code Issues1 Pull requests Projects Releases Packages Wiki Activity

compile notes on design discussions concerning data model handling

Browse source
This commit is contained in:
Valentin Gagarin 2025-03-28 01:38:57 +01:00
parent 4279f6eba6
commit f77db3058b
1 changed files with 96 additions and 0 deletions

96
meeting-notes/2025-03-27_backend-model-conversion.md Normal file
View file

@ -0,0 +1,96 @@
# Architecture notes on backend data model conversion
These aspects have been discussed ad hoc between Kiara, Nicolas, and Valentin over the past two weeks, and this is to keep a record of our considerations.
We previously left some scattered notes in [2025-02-26 architecture discussion](../meeting-notes/2025-02-26-architecture-discussion.md) (Nicolas, Kiara, Valentin), but had some ongoing ad hoc back and forth in the background.
## Problem space
- We'll have to deal with version drift eventually:
- Once a configuration is deployed and we update the Fediversity software (which will likely include a Nixpkgs update), the configurable options may be different.
- We're maintaining our own simplified configuration layer on top of regular NixOS, because our services need to be wired with Garage.
This means we have quite a lot of control over the shape of the interface, but not over the semantics since the underlying service may still change, however subtly.
- It's likely we'll mostly *add* information, but we may also remove configuration options, or change their structure or meaning.
- Under the assumption that we won't force-upgrade operators' deployments, we somehow need to display that difference between what's currently deployed and what can be configured and then deployed
- This is not a problem if the data type for the configuration didn't change, because then it's simply a difference in value
- If the options for configuration change, it's a difference in type
- We could also use the opportunity to display something like release notes inline with the configuration changes, where we could explain why something changed, right where we display the change in structure.
- A prototype for storing versioned configurations was implemented in the following commits:
- [e41f9c572](https://git.fediversity.eu/Fediversity/Fediversity/commit/e41f9c572a9e76f6c009a4fc407ed58d183684f8)
- [9dd92b4cc](https://git.fediversity.eu/Fediversity/Fediversity/commit/9dd92b4cc180e88ffc14a96ec979a90c84e1676f)
- [981ba011a](https://git.fediversity.eu/Fediversity/Fediversity/commit/981ba011abb4bdf9230048bda4e9af2a96be40cf)
The idea is very similar to Django migrations, but at the level of JSON fields:
- Keep Python data structures around as modules for each version; those are supposed to be never changed
- Remember the version stored as JSON using a regular database field
- Take the right version's module when parsing the JSON value from the database
- The display code (HTML template, form behavior, etc.) is stored together with the data model
Not implemented was an equivalent of Django's `FormView` but for Pydantic schemas, which would generate forms automatically and could be adjusted in detail to have control over presentation.
Since configurations can have nested data structures for services, each of which may progress in version at different paces, we could also store versions per service to avoid code duplication; this wasn't sketched in the prototype, merely noted in comments.
As opposed to Django database migrations, the difference between schemas would be determined from the code (like snapshots) rather than from the database state.
- These diffs could be computed automatically and then stored for manual adjustments
- This would keep around both snapshots and diffs, but the diffs need to encode semantics and presentation
- With the diffs ready, one may as well discard the snapshots, but we can expect the diffing code to be rather hard to read
- Django avoids the pile-up of snapshots by only ever caring about exactly one database state.
- We can't simply throw away old schemas since there may be an arbitrary number of versions deployed or configured by all operators
- We would need to determine which versions are still in use before garbage-collecting old schemas
- Working with these configuration schema migrations can be expected to require some discipline, likely more than with regular database migrations, since they also affect presentation of at least two different schemas at once
- We also considered simply not caring about the whole versioning problem, but Valentin insists it will bite us before long and that we should exercise dealing with it far before going live.
- Another aspect is state migrations between deployed versions for each service, which would need to be handled at the service layer.
- This is a well-known, rather hard problem, and has been [covered by academic research](https://upsilon.cc/~zack/talks/2017/2017-11-27-hdr.pdf)
- We won't go into this in the forseeable future, but eventually it needs to be properly solved at sufficient scale to avoid mounting maintenance burden
These changes were reverted in [08d109cc8](https://git.fediversity.eu/Fediversity/Fediversity/commit/08d109cc826c2979af104af0919c75143bd79616), unfortunately without noting the reason.
- Kiara had proposed using some Django equivalent of [react-jsonschema-form](https://rjsf-team.github.io/react-jsonschema-form/) to render the front-end automatically from the underlying data model
- Valentin is opposed to the idea on multiple grounds:
- Under the assumption that we want a rather lightweight front-end in terms of amount of JavaScript loaded and executed in the browser, most of the work would need to happen on the server
- This would require the conversion to forms to happen in Django
- Most likely we'd want to use [DRF form rendering](https://www.django-rest-framework.org/topics/html-and-forms/#rendering-forms) on top of Pydantic models for that
- We could generate the Pydantic models at build time with https://github.com/koxudaxi/datamodel-code-generator
- The devshell could even place the generated files next to the code, where can be inspected as needed
- There's https://github.com/georgebv/drf-pydantic for automatically generating serialisers
- At that point it's unclear what we need the JSON schema for, because we may as well work directly with Pydantic models
- With JSON schemas, we have no fine-grained control over presentation
- There still would need to be a layer for adjusting spatial arrangement (such as sequence of fields)
- That would need to be done for each schema version, and versioning will be part of our application logic
- Again, it seems like it would make more sense to manage that centrally, in one language and with one representation of the data model, and simply display the result at the Django interface (HTML forms augmented with htmx, optionally JSON+REST if needed)
- Another question is whether we would start with JSON schemas or merely use them to pass around information about types.
- Valentin argues: if the browser is not supposed to handle application logic, we never need to pass around type information
- The only place where we'd be forced to use JSON schemas is if we generated the front-end data model from our NixOS modules
- Arguments against a standalone client-side application:
- It duplicates application logic we already have, and costs effort we could spend on developing the application itself
- We don't expect to ever deliver anything else than the web front-end, and Django gives us JSON+REST essentially for free anyway
- Keeping separate application state in the browser is a can of worms we don't have the capacity or team experience to deal with on the current timeline, and dealing with it wouldn't help address our business problem
- We can get dynamism with htmx at much less mental overhead than a client-side application, a fraction of compute requirements in the browser, and none of the supply chain hassle
- Another can of worms is bridging the gap between the Nix backend and the (in this case Django) frontend
- Clan developers already spent time with the problem and wrote a [module options to JSON schema converter](https://clan.lol/blog/json-schema-converter/)
- The general issues is that the module system is substantially more expressive than JSON schemas
- The problem is not that we *need* that expressive power, but that avoiding it requires care
- Generating schemas requires taking particular care when writing front-facing modules
- Nicolas proposed using a safe subset of `lib.types`
- Valentin: We could go even further and write a type library explicitly compatible with [JSON schema validators](https://json-schema.org/draft/2020-12/json-schema-validation)
- Avoiding laziness (such as `mkIf`) would still be on implementers
- This can work but may be brittle due to leaving opportunities for human error
- It's likely to be the most pragmatic approach for the forseeable future, since JSON schemas provide a reasonable collection of common types
- Putting in a JSON-schema-safe layer would be ideal, and will likely benefit the Nix ecosystem a lot by unlocking independent experimentation with UIs, but may require a non-trivial time investment.
- We'd then store all versions of the option declarations so we can read them back in the front-end
- Alternatively we could only keep the generated Python classes and the presentation code, because that's the only thing we'll need later
- In trat case we should have Git hooks, CI checks, and possibly a check at startup to ensure the generated code corresponds to the module options
- There's some recent work in the direction of [diffing module options and values](https://oceansprint.org/reports/2025/#nixos-module-system-enhancements) which we may be able to leverage for at least partially auto-generating version diffs
- Alternatively we could take the opposite direction and parse JSON schemas into module options
- This is how the current prototype works, except we don't even have a module or schema but simply expect the data structure to have a certain shape
- That's just an interim hack, and placing a module type at the interface promises to ease error analysis in tests
- It has the advantage that invalid expressions cannot be constructed, avoiding future friction entirely
- We can live with rather simple type checking at the module layer since the front end would do good-enough input validation already
- The main disadvantage is that the dependency is flipped: we'd have to model our back-end business logic (service configuration interfaces) in what would then exist as part of the front-end code
- The coupling between our service layer and NixOS is rightfully tight, and having a media gap due to a different language will incur mental overhead and slow down iteration and testing.
- The build setup will also get rather confusing because we'd have to first invoke the front-end code to wire up the back-end with the front end...
- Conlusion for next steps:
- Generate JSON schemas from our front-facing NixOS modules
- Work towards a JSON schema type library for module options, gain practice with safe usage patterns
- Generate a Python data models from JSON schema, and work with forms from there