API specification published #334

New issue

Closed

opened 2025-05-15 20:14:11 +02:00 by kiara · 0 comments

kiara commented

2025-05-15 20:14:11 +02:00

Owner

As a front-end developer,
I want an (API) spec describing how to interact with Fediversity,
so that I can start work on front-ends.

implementation notes

given fediversity-core comprises a library exposing a set of expensive (in themselves) stateless programs, the core function of an API would be to wrap (queued) deployments invocations: any state would then live in such (hosting-provider-provided) queue (#242) and TF back-end (#515), which we would interface with to query and update.
the (asynchronous) web sockets may be more suited than the synchronous nature of HTTP requests (as used in our REST-based sample panel), to accommodate the duration of deployments, the intended progress logs (#184), on top of the expected queue-like architecture. as such, while we may define structures for outgoing and incoming messages, there would be no (formal) verbs/codes/headers like for REST, whereas URLs would apply only at the level of the websocket end-point, rather than having request-level endpoints.
- the accompanying protocol WAMP maps the synchronous request-response structure to such an asynchronous context:
  - requests: handled using RPC, where a caller (client) would go thru our dealer (API) to file operations such as enqueuing deployment builds to a builder (callee). note that neither side would need to know of anything but the broker (= API), making it possible for responses (see the following step) to be disconnected by intermediate steps such as queues.
  - responses: for ephemeral messages like updates may be handled using PubSub, where a publisher (e.g. nix builder node) would publish messages to a topic thru broker (our API) to subscribers (clients). ~~if supported, topics here might include a hierarchy of: build < deployment < namespace (e.g. user < org).~~ as hierarchies appear not supported, potential topics in our context would include builds and deployments.
deployments are identified by fragments in the TF back-end carrying their state. such identifiers are thus passed as part of the API's deployment requests, tho it is on the API's user to assign semantics (such as multi-tenancy) and corresponding authorization to these. as fediversity-core lacks notion of users, the API instead offers dynamic authorization. downstream web applications may thus provide a remote procedure handling these according to business logic, which the API will defer to by RPC.
a deployment endpoint could be configured to wrap a certain deployment implementation like our nix-shell -A devShells.x86_64-linux.operator --run deploy-ssh-hosts.sh default.nix, e.g. proxmox. deployments in this schema should be specified by our operator-facing JSON Schema ./src/panel/configuration/schema.json (older example) used in our panel.
- given the git-ignore, this schema:
  - could be published using a CI pipeline
  - should be exposed as an API end-point that front-ends could access
eventually deployment end-points need to further pass info on decoupled versions (#304)
while our underlying logic is not very imperative, front-ends may bring along more (PATCH-like) imperative expectations such as buttons over forms, which may eventually have use for according imperative-style REST end-points. queues (and superseding prior deployments for the given identifier) may help facilitate this.
core similarly has no notion of say unstaged deployment configurations, which could otherwise have been exposed on a panel-like GET endpoint on https://HOST/api/v1/deployments/1/old. as such, this is a concern for web applications.
API operations should mirror TF's CRUD-like ones (so like run/update, state / metrics, clear). in REST terms for example: should support a DELETE-like operation. a detail-like 'GET' could show current state from a TF back-end or indicate progress, if such progress updates wouldn't already come async on deployment trigger (POST / PUT). potentially some endpoint to cancel (implied in delete operations?), or leave this implied in deployment trigger websocket subscriptions.
potential effects of parallel handling progress (#242) of API design:
- no parallel deployment (within the same identifier): nothing to list on list-like GET endpoints.
- parallel deployment even within the same identifier: support managing separate deployment triggers, e.g. (REST context:) https://HOST/api/v1/deployments/proxmox/kiara-123/builds/456 vs. https://HOST/api/v1/deployments/proxmox/kiara-123.

open questions

what operations should be supported both ways, and what should their messages look like?
does PubSub's ephemeral nature make it a good match for e.g. log streaming (if needed), if a client arriving after the initial part could then miss that? how do CI systems go about this?
- this seems a matter of appending to a log file and serving or streaming it as requested.
are there good ways to combine synchronous and asynchronous elements in an API?

**As** a front-end developer, **I want** an (API) spec describing how to interact with Fediversity, **so that** I can start work on front-ends. ## implementation notes - given fediversity-core comprises a library exposing a set of expensive (in themselves) stateless programs, the core function of an API would be to wrap (**queued**) deployments invocations: any state would then live in such (hosting-provider-provided) queue (#242) and TF back-end (#515), which we would interface with to query and update. - the (asynchronous) **web sockets** may be more suited than the synchronous nature of HTTP requests (as used in our REST-based sample panel), to accommodate the duration of deployments, the intended progress logs (#184), on top of the expected queue-like architecture. as such, while we may define structures for outgoing and incoming messages, there would be no (formal) verbs/codes/headers like for REST, whereas URLs would apply only at the level of the websocket end-point, rather than having request-level endpoints. - the accompanying protocol [WAMP](https://wamp-proto.org/index.html) maps the synchronous request-response structure to such an asynchronous context: - requests: handled using RPC, where a caller (client) would go thru our dealer (API) to file operations such as enqueuing deployment builds to a builder (callee). note that neither side would need to know of anything but the broker (= API), making it possible for responses (see the following step) to be disconnected by intermediate steps such as queues. - responses: for ephemeral messages like updates may be handled using PubSub, where a publisher (e.g. nix builder node) would publish messages to a topic thru broker (our API) to subscribers (clients). ~~if supported, topics here might include a hierarchy of: build < deployment < namespace (e.g. user < org).~~ as hierarchies appear not supported, potential topics in our context would include builds and deployments. - deployments are identified by fragments in the TF back-end carrying their state. such identifiers are thus passed as part of the API's deployment requests, tho it is on the API's user to assign semantics (such as multi-tenancy) and corresponding authorization to these. as fediversity-core lacks notion of **users**, the API instead offers [dynamic](https://asphalt.readthedocs.io/projects/wamp/en/latest/dynamic_auth.html) [authorization](https://github.com/crossbario/crossbar/blob/master/docs/Authorization.rst#dynamic-authorization). downstream web applications may thus provide a remote procedure handling these according to business logic, which the API will defer to by RPC. - a deployment endpoint could be configured to wrap a certain deployment implementation like our `nix-shell -A devShells.x86_64-linux.operator --run deploy-ssh-hosts.sh default.nix`, e.g. `proxmox`. deployments in this schema should be specified by our operator-facing **JSON Schema** `./src/panel/configuration/schema.json` ([older example](https://git.fediversity.eu/kiara/fedi-goals/commit/6e385a527db6088bb6365d8682156600563edec7)) used in our panel. - given the git-ignore, this schema: - [ ] could be published using a **CI** pipeline - [ ] should be exposed as an API end-point that front-ends could access - [ ] eventually deployment end-points need to further pass info on **decoupled versions** (#304) - [ ] while our underlying logic is not very **imperative**, front-ends may bring along more (`PATCH`-like) imperative expectations such as buttons over forms, which may eventually have use for according imperative-style REST end-points. queues (and superseding prior deployments for the given identifier) may help facilitate this. - core similarly has no notion of say **unstaged** deployment configurations, which could otherwise have been exposed on a `panel`-like `GET` endpoint on `https://HOST/api/v1/deployments/1/old`. as such, this is a concern for web applications. - API operations should mirror **TF**'s CRUD-like ones (so like run/update, state / metrics, clear). in REST terms for example: should support a `DELETE`-like operation. a detail-like '`GET`' could show current [state](https://git.fediversity.eu/fediversity/fediversity/actions/runs/11954#jobstep-2-9273) from a TF back-end or indicate progress, if such progress updates wouldn't already come async on deployment trigger (`POST` / `PUT`). potentially some endpoint to cancel (implied in delete operations?), or leave this implied in deployment trigger websocket subscriptions. - potential effects of **parallel** handling progress (#242) of API design: - no parallel deployment (within the same identifier): nothing to list on list-like `GET` endpoints. - parallel deployment even within the same identifier: support managing separate deployment triggers, e.g. (REST context:) `https://HOST/api/v1/deployments/proxmox/kiara-123/builds/456` vs. `https://HOST/api/v1/deployments/proxmox/kiara-123`. ### open questions - what operations should be supported both ways, and what should their messages look like? - does PubSub's ephemeral nature make it a good match for e.g. log streaming (if needed), if a client arriving after the initial part could then miss that? how do CI systems go about this? - this seems a matter of appending to a log file and serving or streaming it as requested. - are there good ways to combine synchronous and asynchronous elements in an API?

Rows
Columns