generate Python data models from module options #285
No reviewers
Labels
No labels
api service
blocked
bug
component: fediversity panel
component: nixops4
documentation
estimation high: >3d
estimation low: <2h
estimation mid: <8h
productisation
project-management
question
role: sysadmin
security
technical debt
testing
type unclear
type: key result
type: objective
type: task
type: user story
user experience
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Fediversity/Fediversity#285
Loading…
Add table
Reference in a new issue
No description provided.
Delete branch "fricklerhandwerk/Fediversity:generate-module-options"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
edit by kiara: see #275.
this shows a proof of concept for generating Django forms from NixOS modules
it still needs lots of cleanup (and probably unbreaking the deployment action), and a sane contributor workflow when updating the underlying NixOS modules.
@ -18,3 +32,4 @@
inputsFrom = [ (pkgs.callPackage ./nix/package.nix { }) ];
packages = [
pkgs.npins
pkgs.jq
where is this used btw? maybe clarify in a comment there
@ -15,0 +22,4 @@
{
}
''
${codegen} --input ${schema} | sed '/from pydantic/a\
what's the replacement for? maybe clarify in a comment
@ -1,5 +1,16 @@
{
"pins": {
"clan-core": {
iiuc this is still unused (as per your comment below)?
@ -0,0 +1,430 @@
# TODO(@fricklerhandwerk):
# temporarily copied from
# https://git.clan.lol/clan/clan-core/src/branch/main/lib/jsonschema/default.nix
# to work around a bug in JSON Schema generation; this needs a PR to upstream
looking forward to the PR, both to clarify/explain the diff, as well as to separate concerns - as far as i'm concerned (until merged) feel free to pull this from such a fork rather than inlining here
thanks! i added a few comments, hoping to better understand things still
@ -0,0 +1,64 @@
/**
Deployment options as to be presented in the front end.
These are converted to JSON schema in order to generate front-end forms etc.
with this form generation, we may need to still further consider visual choices down the road, such as (in the current form)
input type="password"
@ -0,0 +10,4 @@
}:
let
inherit (lib) types mkOption;
in
down the road we may want to look into guaranteeing our different components use and expect the same structure, but having any check here definitely improves on what we had already :)
@ -1,6 +1,7 @@
# Nix
.direnv
result*
src/panel/configuration/schema.py
for review purposes, it'd be nice to demonstrate sample contents of such a generated file in the PR description
@ -0,0 +20,4 @@
# remove _module attribute from options
clean = opts: builtins.removeAttrs opts [ "_module" ];
# throw error if option type is not supported
if we were to at some point plug existing nix modules into this, we might consider a schema export ignoring (rather than crashing on) fields too complex in their defaults/type. i don't know if clan cares for a PR in that direction, otherwise we could fork.
@ -0,0 +41,4 @@
{
string = ''"${toString value}"'';
integer = toString value;
boolean = if value then "true" else "false";
not just
toString
as well?@ -0,0 +75,4 @@
in
{
options.services.${name} = {
${options}
to what extent would this differ from
deployment/options.nix
?@ -0,0 +2,4 @@
import textwrap
# TODO(@fricklerhandwerk): mix this in as a method on the Pydantic Schema itself
def generate_module(schema, service_name):
could you add a docstring (on module or function) on what this file does?
@ -47,3 +45,3 @@
self.assertTrue(config.parsed_value.mastodon.enable)
# this should not have changed
self.assertFalse(config.parsed_value.peertube.enable)
self.assertIsNone(config.parsed_value.peertube)
to better understand how this works, would you know how it decided not to send this sub-object?
here it's simply not setting those fields because they're optional
@ -0,0 +40,4 @@
}
""")
actual = to_module.generate_module(Example.schema(), "myservice")
were we not converting from nix (thru json schema) to django, rather than from django to nix? i'm confused on our moving parts here so could maybe do with a few extra comments and/or pr description
Yes, this is an artefact and I'll remove it.
@ -95,1 +70,4 @@
# TODO(@fricklerhandwerk):
# this is broken after changing the form view.
# but if there's no test for it, how do we know it ever worked in the first place?
@kevin @lois
@ -114,3 +97,1 @@
"pixelfed": deployment_params['pixelfed']['enable'],
"mastodon": deployment_params['mastodon']['enable']
}
"services": deployment_params.json(),
🎉
Oh yeah the first two commits also contain sketches to create modules from Python, but that was code from before I settled on the other direction (and also the second one is not particularly smart, because we could just build an expression directly rather than a string). As said, this needs cleanup and testing.
fwiw i tried a rebase.
the one annoying conflict i found pertained to reconciling passing arguments to the view, which in this case i resolved by having the view reuse the available info for now.
@ -0,0 +182,4 @@
description = lib.optionalAttrs (option ? description) {
description = option.description.text or option.description;
};
exposedModuleInfo = lib.optionalAttrs true (makeModuleInfo {
i see you moved this - why was that, and while we're at it why would anyone ever us a no-op like
optionalAttrs true
in the first place? symmetric considerations?@ -0,0 +127,4 @@
# parse options to jsonschema properties
properties = lib.mapAttrs (_name: option: (parseOption' (path ++ [ _name ]) option)) options';
# TODO: figure out how to handle if prop.anyOf is used
isRequired = prop: !(prop ? default || prop."$exportedModuleInfo".required or false);
what was this change about?
this was required to somehow propagate that fixed attrs are required fields
@ -0,0 +381,4 @@
# return jsonschema property definition for attrs
then
default
// (lib.reqursiveUpdate exposedModuleInfo {
what was the change here about?
a6e5cbb67e
to9c73741559
WIP: generate Python data models from module optionsto generate Python data models from module options@kiara is this small enough to be reviewable for you? I'd maybe split adding the new Python package into a separate commit and squash the rest. Making it nicer (specifically: simplifying the baroque module declaration to be like one would expect a module to be) is blocked on the JSON schema converter being clumsy with types, and it doesn't seem worth the effort to fix that at the moment.
@ -0,0 +4,4 @@
These are converted to JSON schema in order to generate front-end forms etc.
For this to work, options must not have types `functionTo` or `package`, and must not access `config` for their default values.
The options are written in a cumbersome way because the JSON schema converter can't evaluate a submoule option's default value, which thus all must be set to `null`.
typo
@ -0,0 +5,4 @@
For this to work, options must not have types `functionTo` or `package`, and must not access `config` for their default values.
The options are written in a cumbersome way because the JSON schema converter can't evaluate a submoule option's default value, which thus all must be set to `null`.
*/
interesting challenge, i wonder if we may need some better solution to this down the line
Yes, essentially the JSON schema converter needs to be aware of
$defs
. But that feels like half a rewrite (which I concluded after trying to weasel around that effort, unsuccessfully).thanks, added at #195 as a note on where we stand on this
@ -95,1 +67,4 @@
# TODO(@fricklerhandwerk):
# this is broken after changing the form view.
# but if there's no test for it, how do we know it ever worked in the first place?
maybe file in the issue tracker if this is a regression induced prior to this PR, as it would then be unrelated to it
This is tracked as #276
@ -121,2 +93,2 @@
},
},
"deployment_status": deployment_status,
"services": deployment_params.json(),
did this change not also require fricklerhandwerk/Fediversity#1/files (as alluded to at #285 (comment))?
Hm no, I think this is alright
how does that even work? the template still wants to show
.url
there which you removed while we don't otherwise set anything like it?i worry you may have overlooked this as earlier you mentioned relying wholly on CI to verify intended behavior.
Maybe. As noted, this interaction needs a test.
i've filed #326 to track this.
@ -33,0 +43,4 @@
codegen = "${python3Packages.datamodel-code-generator}/bin/datamodel-codegen";
pydantic = runCommand "schema.py" { } ''
${codegen} --input ${schema} | sed '/from pydantic/a\
from drf_pydantic import BaseModel' > $out
this sed expression looks weird, not ending in a slash, having the
a\<BREAKLINE>
, how does that work?sed magic... it replaces the matched line with the following line. 🤷
TODO: commit message on squash