API route for delivering JSON Schema from NixOS module options #1209
Labels
No labels
0 points
0.5 points
1 point
13 points
2 points
21 points
3 points
34 points
5 points
55 points
8 points
ambition
application-offering
ambition
configure-applications
ambition
front-end
ambition/install-applications
ambition
security
ambition
switch-host
ambition
update-applications
ambition
user-management
api service
blocked
component: fediversity panel
component: nixops4
documentation
estimation high: >3d
estimation low: <2h
estimation mid: <8h
infinite points
productisation
project-management
question
role: application developer
role: application operator
role: hosting provider
role: maintainer
security
technical debt
testing
type unclear
type: bug
type: deliverable
type: key result
type: objective
type: task
type: user story
user experience
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Blocks
#195 NixOS modules' options exposed
fediversity/fediversity
Reference
fediversity/fediversity#1209
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
As a user of the Fediversity API,
for a given Fediversity deployed service like e.g. Mastodon and a given Nixpkgs revision used to deploy the service,
I want to be able to load a JSON Schema from the backend in order to render configuration interfaces to the operator and the hosting provider.
The schema generation is done evaluating a nix expression and fetching the required Nixpkgs tarball, which needs to be done inside a Windmill job.
Generating the schema will take some time and compute (Download nixpkgs tarball and evaluate), therefore the schema should be cached by the API server. The cache keys would be the
services.mastodon)optionsToSchemaNix functionWe don't persist the cache beyond API server reboots as that might require migrations at some point and cache warming is reasonably fast.
The API route could be
/nixos-options-schema/<nixpkgs-revision>/<options-prefix>.this should currently be exposed thru
core/'s flake atttribute ofapi-deploymentsdefined incore/render.nix. the api now invokes that fromapi/src/fediversity_api/deployments_runtime.py.currently, the schema is generated based on our data model (
core/lib/data-model.nix'sfrontendConfiguration), which may expose all of or one of (PIN_EFFECT) its effects (#1071), that is, it does not currently filter to a single application - tho any configuration options already set by the hosting provider would already be filtered out from the schema exposed to the operator (a feature also added in #1071, done inapi/default.nix'spackage.hostingLeaves). to be fair, our effects do expose application-specific deployment (core/->nix flake show) - so far ourcore/setups/have just had their*-allvariants exposed tofrontendConfiguration, tho it should be viable to tweak that for this (and in fact,tf-incus-imageis today the sole setup exposing targeting a single application - in this case because it was an effect targeting a single host in the first place).we intentionally don't do this today - the revision we target is the fediversity one (in the event of a mono-repo split intended to mean the revision of
core/), because arbitrary nixpkgs versions can and will change stuff in such a way as to break shit on our end if we'd yet to account for them, as we interact with the nixos service options so are sensitive to changes there if we are to build nixos configurations in a robust fashion. as such, any updates must have been explicitly committed in ourcore/'s repo.module-schemawe have yet to switch over to from clan's schema generator (core/lib/schema-for-source.nix), awaiting @toonn's confirmation our internal convertor is ready.so currently it's not, because we presume for foreign versions to be rare (only expected during migration between hosting providers, see #100, where the bottleneck is expected to be the 1-day DNS cut-over), and even then, for flakes' eval cache to cache the schema generation's evaluation result (to short-cut subsequent invocations).
moreover, i think with windmill jobs we have so far mostly gotten out job stats (see
api/src/fediversity_api/windmill.py's_WM_STATE_TO_BUILD_STATUS), tho to be fair i did recently see something about making windmill like use stdout (i do seestdoutprints from that same file), tho not sure we've been using that to extract results just yet.we used to hand-roll caching on this, then i dropped that to avoid redundancy with flakes' cache (#1110).
to be fair, given that environment variables would render our evaluation impure, i haven't actually confirmed caching behavior there. (ideally, i would prefer not to require impurities for our evaluation, and hope that we would find ways to address that in a way that works well with caching - whether to evaluate without going the flakes, or by say doing @edolstra's configurable flakes, by the time that were to become available).
so, we should have an api route wrapping this api flake attribute, and you should be able to find it documented in our swagger, as generated by api's
openapiflake attribute.i wouldn't technically dare close #195 just yet in the sense we never actually tried exposing nixos modules in our applications just yet. this one may have had overlap with that tho - i think closing here might be ok.
Who wants to close #195?
This issue was intended as a necessary part of #195, like to make one subtask more specific and with a clear definition of done.
But the way I see it there is not yet an API route like the proposed
/nixos-options-schema/<nixpkgs-revision>/<options-prefix>, so this wouldn't be done yet.so the endpoint we got is https://swagger.pages.git.fediversity.eu/#/default/schema_get_for, which uh could use better documentation, tho i think the panel form visually explains it a bit more clearly.
this being based on our version (rather than nixpkgs) is intentional (our code has to interoperate with it to make valid configurations),
<options-prefix>we don't currently distinguish, tho the application's bit one might just grab off of such a schema shared between applications. splitting it could make more sense at some point, but so far i think having to do separate loading per applications at our current scale i'm not as sure about yet, particularly while our configuration is not application-specific, as it is currently like a GUI over ourcore/{effects,setups}/.y'all are free to try it
lemme try and update that
reading thru #1226, i'd maybe like to take a step back on the use-case here.
so, rather than exposing nixos option modules on-demand, i think the more likely use-case for our use-case would be to instead (in practice selectively) embed them into our individual applications' operator-facing option spaces.
now, to be fair, our panel form currently exposes (effects and) our
*-alldeployment methods, admittedly leaving the application-specific deployments as a gap, that we did previously describe (more generally) at https://git.fediversity.eu/fediversity/fediversity/src/branch/main/api/specification.md#imperative-rest-endpoints.even then however, in this case, given our baked-in general schema would already expose our JSON-Schema including of our applications, for in as far as such (NixOS) modules would have been included in there already, then we could just grab application-specific bits from there i guess.
This sounds like what I proposed at first in the meeting last week but you said that it should be the other way around, since we need to have jsonschema for different Nixpkgs revisions.
I think for misunderstandings like this it's important we write down plans as preciseley as possible so that we can achieve a collective understanding of architecture.
The new DoD for #195 that you edited in doesn't fullfill that requirement for me:
It leaves open with through which technology the NixOS modules are exposed, which nixpkgs revision(s) are used for that, how we handle changing implementations of optionsToSchema. Therefore I think this issue (or a different issue describing a different architectural decisison) is still needed. It is here to set a collective understanding of the details and also can be closed to signal that an important step of #195 was completed.
i'll concede i have basically not been doing definitions of done (caring less about accounting more about direction), and probably others will be better at identifying where user stories in themselves are unclear.
thanks for filing to push on this, yeah
i guess our docs are wip now, but i hope those will end up helping long-term.
on this one in particular, the mechanism isn't expected to change (still
module-schemaas per the existing delivery mechanism fromdata-model.nixfront-end configurations, JSON-Schema thru our existing REST end-point).the ticket in question is orthogonal to this (different versions have relevance only to #100, which has its existing mechanism on this described in https://git.fediversity.eu/fediversity/fediversity/src/branch/main/api/specification.md#endpoints-for-switching-between-hosting-providers-100), so here would assume the 99% case (-> our nixpkgs pin in
npins/sources.json).https://git.fediversity.eu/fediversity/nix-module-form has been getting updates recently to account for the changes either new or with outstanding PRs - not sure that answers the question? (i don't necessarily see changing schema descriptions as breaking stored data, while broken code we can just update accordingly.)
if this ticket is to tackle point 1 from fediversity/module-schema#12, it probably needs some further design esp. w.r.t. questions from #100 - otherwise i'd be inclined for now to suggest closing this for now as not currently actionable.
I agree to close this, I didn't understand the architecture enough to plan this (and frankly I still don't).
I'm a bit confused on the closure. The OP in this issue seems reasonable to me, the panel needs to get the schema for each Application it has to show a form for, an API cached on Nixpkgs revision (understood to include any overlays and such)/service module/module-schema revision ticks that box without adding any a priori redundant constraints IMO.
the part i found currently unproductive here was that it suggested dynamically exposing NixOS modules specifically.
now, if fediversity/module-schema#12's suggestion 2 needs a ticket, that's fine, but that'd be about fediversity applications, rather than NixOS modules.
if its suggestion 1 needs a ticket (probably to land after the easier proposal 2) with a more fleshed-out proposal, that's also fine - but this ticket i think somewhat glossed over such relevant design concerns, which lead to a PR that imo glossed over some open questions still.
like overall i'm not saying 'no', i'm saying "if we're making something, we should design and test it in context to ensure we can make it serve our use-case".
Are we sure about that? The difference between a NixOS module and an Application module would be that the latter exposes a subset of the former's options, right? And options outside that subset would be set by the Hosting Provider.
The Operator then fills in the options still required by the Application module and clicks deploy. This submits the form for the configuration and we want to validate the configuration. Two choices, either we validate only the options coming from the Operator, or we validate the entire Application configuration. The former can use the schema for just the Application module, the latter requires the schema for the NixOS module.
There's other factors that come into this, do we expect Hosting Providers to do all their configuration in Nix files or do we give them a more powerful panel?
Either way we need an API for getting the available options, the Fediversity deployment can simply return static schemas in response to the API requests but we can't statically add the schema to the panel because Hosting Providers' custom panels need the data too.
okay so. in short no: an option is not strictly tied to a module type (=
_class), in the sense that withoutdefaults referencing (external)configthey would, in fact, be portable.that means our applications' options don't have to be/use nixos options, and in fact, they so far never did, hence #195. instead, they have just been defining their own abstractions so far, instead of simply wrapping underlying nixos options - which up until recently (#1102) would have raised concerns on how to handle migrating values across schema changes on updates.
technically, not all options have to be (or are, for that matter) set.
now, our option set (as per
core#frontend-configuration-schema) will in fact have to be set by one or the other.so
examples/already contains/allows.toml, tho we then found out that that particular format has the drawback of lacking support for thenullvalue.the 'more powerful panel panel' would seem something we already have some of the technical foundations for, in the sense
panelfacilitates visualizing/setting 'their' options - it hides whatever has been set thruHOSTING_ENV_CONFIG(+ shows justPIN_EFFECT, if set).so, the hosting provider side currently for the most part isn't setting raw NixOS options.
instead, the bits exposed explicitly to just them is an abstraction of our own (resources), whereas the option set technically shared between hosting providers and operators (to allow this fuzzy boundary, i.e. the
core#frontend-configuration-schema) has an API endpoint, tho that isn't about NixOS options specifically (unless we were to expose them, as per #195).hosting provider panel could probably use a ticket of its own, but i see what you're saying: static adding would imply like having to reload the operator panel, which does indeed raise concerns on invalidating sessions.
to be fair,
panel/is intended as a demo front-end, that isn't necessarily an exact implementation guide, tho if this is a common feature it would seem fair to facilitate tackling it.@kiara wrote in #1209 (comment):
Since #195 is the way forward, I think we should design the API with that in
mind.
Regardless of #1102, wrapping NixOS options shields the user from noticing
changes only insofar as Fediversity can translate back and forth, it can only
ever be a band-aid. An alternative would be for Fediversity to add automatic
conversions of old configs to the newer structure expected by the NixOS
modules. This would have the same drawback of having Fediversity be an
obstruction to updates/adding Applications, but would expose the NixOS module
options and potentially could expose the newer options quicker for those that
value them more than the guidance.
It also assumes there's a single holistic schema. As discussed in
fediversity/module-schema#12 (point 0), that isn't practical. We want an API to
get the schema for one Application at a time.
There was talk of it being a product suitable for white-labeling before, that
does require a bit more than a mere demo that could have everything hardcoded.
Either way having to redeploy the panel for a simple data change seems quite
onerous to require of every Hosting Provider that implements a panel or even
simply runs ours. The simplest way to decouple the panel from the data is for
it to get that data at run-time, via an API call.
For all these reasons I think this issue is still relevant.
sure. now, that ticket is about having our applications' option submodules wrap nixos submodules, in which case our current method of exposing our option sets would suffice, which would seem to defeat the need for an endpoint dynamically exposing nixos options.
ideas/tickets/PRs welcome
sure, please do file this
panel/(intentionally) lacks accounting if not user entities facilitating that, which is business logic, not something on our end. in practice, one would likely want it to include non-fediversity hosting services as well.procolix's productization effort is intended to close this gap, and would be a distinct code-base.
as such, our role there (at these existing repos) is more about facilitating such integration than about making any of the opinionated calls involved (pricing models, billing methods) from our end.
i mean, that doesn't sound like what the current ticket proposed, which focused on NixOS options.
and even then, to get something deployed in the first place one would (currently) need to go thru the full schema already, in which case the full schema should be cached to cherry-pick sub-parts from, for the foreseeable future (while scaling this beyond that would imply first having to make it much easier to add new applications, which would probably involve normalizing more of our logic at nixpkgs - potentially quite long-term).
@kiara wrote in #1209 (comment):
These seem like details to be discussed on this issue to arrive at a
formulation of the goal. Not a reason to close this and wait until a perfect
issue is opened.
I'm still of the opinion that the Fediversity layer should be minimal, we take
the NixOS module options, subtract whatever the Hosting Provider already set or
explicitly marked as (not) for the Operator and show what's left over to the
Operator.
In what form are you thinking? Just an issue "As a Hosting Provider I want a
white-labelable panel?"
Or they run our panel as one of the applications (for lack of a better term)
they offer. As a second layer our panel could remain oblivious to billing.
In the self-hosting use case there's no billing to worry about either.
Adding this API seems like anticipating their needs to me, not a very (at all?)
opinionated call. If their feedback proves us wrong it's not like the API is
set in stone.
Maybe this is my personal bias but isn't this what we're working towards? What
else is module-schema supposed to do?
I don't see why this would require adding Applications to be easy first, rather
the opposite this is crucial in enabling that.
I'm not sure I follow you on the need for the full schema, in particular how it
would imply that doing a full schema and extracting sub-parts is necessary
versus fetching the individual Application schemas and composing them when
needed.
This API is supposed to enable displaying a form to configure a specific
Application. Fediversity-level options, mostly whether to enable deployment of
a particular Application, is a separate concern IMO. The front-end needs to be
informed by what Fediversity can offer, yes, but this is not a one-way street,
we also want the design of the front-end to inform what Fediversity should (try
to) offer.
rephrasing then, what does this add over #195 while we expect not to need a new endpoint?
technically, this direction could be facilitated by e.g. #161.
(deploying it along with their other stuff is something we do today - hence https://panel.abundos.eu/.)
that said, procolix currently got funding allocated to do a separate productization-oriented front-end, so changing course on this might involve not just a technical proposal, but some politics as well.
this is missing the point to me, as i noted earlier:
if we can reason about how things might fit in, then at least we'd have some more confidence on approach already.
convert options, which may transparently wrap NixOS options, but this does not mean the exposed options need to themselves be of
_class = "nixos";- hence i challenged that repo's description of 'Generate JSON schema for NixOS modules' toward phrasing like Nix (module system) modules, as there is nothing NixOS-specific about its functionality.i would think "operators' download of a static asset takes too long" is not really a problem we have today.
in fact, that static asset is now 30kb, nix-cached, currently baked into the panel on build, and browser-cached - on top of any individual exposed application's info being available right in there.
under our data model's resource abstraction, the hosting provider's environment resources resolve given the operator's fediversity configuration.
wanting to know hosting providers' configuration upfront then, presumes that in any such remaining nixos options, those resources would no longer hold relevance.
i won't say determining that is impossible, tho i would consider it non-trivial, at least, without say scrapping the resource entity.
our deployments now want a data model
[frontend-]configurationrather than an application's options.this relates to applications being presumed to depend on external (ancilliary) nodes to handle concerns like databases.
could we reconsider that? potentially yes - tho a hosting provider may want to plug in their own s3/database, whereas we may also well benefit from trying to make are nodes more ephemeral/cattle-like, rather than the nixos status quo of them being pet-like (which is probably nicer when a nixer will ssh in and take care of it than when... we're responsible for the functioning of nodes that no-one else might have the expertise to maintain).
a step closer in reach seems 'panel schema loaded dynamically', but that could work too as a higher-level ticket
@kiara wrote in #1209 (comment):
Why do you not expect to need a new endpoint? IIUC the
get_schemaAPIcurrently returns a schema generated from the wrapper in Fediversity. That's a
pain for front-end design, hence this issue.
Has Procolix stated that they want exactly what
get_schemacurrently does andnothing else? Either way, additional API that returns different information
wouldn't impede their use of
get_schema.The context here being Kerstin's thoughts about how to implement the panel UI.
This thread goes too off-topic IMO, maybe an issue on module-schema?
I feel like you read more into "NixOS" than is meant. The problem is that
modules are not a Nix language construct, as far as Nix is concerned, they're
just values like any other values. "Nixpkgs' lib module system expressions," is
as precise as I can come up with but doesn't roll off the tongue. Another
problem is that what we can actually consume is the result of
lib.evalModules, which has amodulesinput argument, this makes referringto the output as "modules" confusing.
This doesn't address what I stated. The size in bytes of the full Fediversity
wrapper schema is not a concern to me, nor to Kerstin I think?
The values that Hosting Providers set aren't of interest but rather which
options they have/will set. The front-end needs to know which options to show
to Operators.
Not clear to me why this needs to be reconsidered at this stage. The API
discussed here doesn't imply anything about ancilliary nodes.
(I opened #1288 re dynamic schema loading.)
now, 'a pain' sounds like an abstract concern, if not omitted as an explicit goal in the phrasing of the current ticket, but if we have a ticket elaborating on that such that we could reason how the suggested solution relates to the raised issue, then great.
sure, but what i'm seeing as the open questions here that'd have use for answers up-front include:
until we find answers, might one not conclude we're entering territory we're not actually comfortable with, given our use-case?
like, you can add it, sure, but then sooner or later you'd need to figure those out still - if not upfront, ideally.
#1226 (as per the OP here) was about NixOS modules - and that i think is what raised additional questions on how to reconcile toward our use-case.
right - our data model makes this non-trivial
you're probably right.
just saying to set expectations, right now using the info to actually deploy may involve the broader configuration still - fixing that may need a new deployment end-point for such application-specific submissions as well then, if we'd prefer the/a panel didn't need to track that for such an operation.
@kiara wrote in #1209 (comment):
OK, so you want more motivation for why an API to get Application schemas is
desirable.
I still think that anything that depends on the Nixpkgs revision should
probably be responsibility of the Hosting Provider. I assume you're thinking of
things like "ancilliary" services? Let's say the maddy module in the new
Nixpkgs revision is different, that'd be up to the Hosting Provider to
reconcile.
If the Hosting Provider is in control of the Nixpkgs revision, then they're
also responsible for updating/evolving it.
The wording needs to change to "Operator facing Application schema" or
something.
We'll have to make the information accessible somehow. Note that this can be
entirely separate from the Hosting Provider's Application configuration, it
could be an explicit option set elsewhere as an explicit allowlist of options
to show Operators for each Application.
Let me attempt to summarize the discussion:
Conclusion
Caches are invalidated on a redeploy, which fixes the Nixpkgs and
module-schema revisions, so this should not be a problem?
fetching a schema of the Operator facing options for a particular
Application.
schemas for individual Applications rather than an all-encompassing schema.
so what is the hosting provider here, in the event of a migration from one to another?
if someone jumps ship using a bunch of applications that didn't match the new provider's stuff, does that entitle them to a bunch of custom programming work from this new hosting provider, that may well not even know nix?
like, even given some concept of consent, i think we need clear lines of responsibility, one way or another, and they'd need to kind of line up with expertise.
okay, sure.
@kerstin?
if we are to do the explicit allow-list, should that not already suffice?
as in, i feel like "options already set" is somewhat of a proxy measure, in a sense.
like, if a service has mutually exclusive options, and we set one of them, then in practice, that should not mean the rest of those should (given they're still unset) suddenly become an operator concern.
in that sense, isn't the allow-list the more sensible path anyway?
i'm maybe getting more bullish on your notion here, see #1288 (comment).
i think the point of contention seems closer to i think having things in VCS in one way or another tackles concerns about updating path, while i'm not sure i've understood your position on such VCS, given you've been describing logic as config, rather than code.
@kiara wrote in #1209 (comment):
I don't think targeting Hosting Providers without Nix expertise is viable, at
least in the initial phases, maybe after some years of experience in the field.
I mentioned partial migrations in #1282. Hosting Providers should be in control
of what Applications they run. The Matrix spam flood from last year (I think it
was last year) might make most Hosting Providers reluctant from offering Matrix
Applications. Neither Operators nor the Fediversity team should be able to
force them to accept migrations of Applications they don't want to run.
Also note that the API suggested here is not meant to enable migrations really.
Just to give a panel a way to get the information it needs to display to
Operators.
Since Kerstin has expressed frustration with this issue I'll offer to open a
new one. Though it'd still involve communicating so we can clarify the
motivation more concretely.
My motivation for not requiring an explicit list is UX for Hosting Providers
pretty much. It's a bit annoying to repeat all the options you've just set.
(I've also thought it might be possible to do something like the priority system
in Nixpkgs but it'd be something like
mkOperatorfacingWith{Merge,Override}and
mkHostingProviderWithDefault(the default coming from Fediversity). Thisis very hypothetical and straying off-topic though, probably better discussed
elsewhere.)
Thank you, @toonn
Yeah I'd rather not work on this right now, but I'll keep following your discussion.
well, it looks like that seems to be the (sole?) pilot we are to enter here, so... whether a good idea or not, we may be about to find out?
would i consider this terrifying? yes - as i'd expressed last year, and for production i think it'd take more (#598).
if anything, SLAs there might be an expectation to manage with initial users.
overall, yeah, procolix will probably need to better get into nix as well.
agree, sure.
to be fair, supporting that shouldn't take that much.
so. options set explicitly (the data model's option set shared between operator and hosting provider) - yes.
the bits handled thru the resources makes this murkier, but to be fair, that wouldn't exactly be described by "all the options you've just set" (setting them is abstracted out).
that said, we could support exclude-lists as well as allow-lists, and in fact exclude from our end the bits we expect should not need to be set, given our resource model, even if we could statically infer which specific options end up set in practice (once an operator deploys given their configuration).