Add OpenObserve telemetry storage node to the hosting-provider group #1116

Merged

kiara merged 3 commits from kiara/fediversity:prometheus-otel into main 2026-06-21 18:34:48 +02:00

Conversation 0 Commits 3 Files changed 14 +516 -46

kiara commented 2026-06-21 16:32:04 +02:00

Owner

Copy link

What

Adds a monitoring node to the hosting-provider group running OpenObserve as an all-signal telemetry store, and teaches the otel-collector (#1002) to additionally export traces, metrics, and logs to it.

The collector previously fanned every signal only to the debug exporter, so nothing was stored -- it landed in the journal and was lost. #1002 listed "storage back-end" as the open follow-up, and parent issue #184 requires tests covering collection and storage in a back-end.

OpenObserve ingests metrics, logs, and traces over OTLP from a single binary with no ClickHouse/Postgres/ZooKeeper (embedded sqlite meta + local disk in single-node mode), so it fits a small ~20-machine hosting provider. The original ask named "a prometheus node", but Prometheus stores metrics only and would drop the traces/logs that #184 cares most about; the decision (made with the maintainer) was to store all three signals in one backend.

Changes

• monitoring.nix (new) -- OpenObserve systemd unit under services.fediversity-monitoring. No services.openobserve NixOS module exists, so the unit is hand-written. The root password is delivered via LoadCredential + an ExecStart wrapper that exports ZO_ROOT_USER_PASSWORD from the credential file, keeping the secret out of the unit/store (same spirit as the S3/openbao secret handling). Outbound usage telemetry and the GeoIP database download are disabled.
• monitoring/default.nix (new) -- thin application declaration, modeled on otel-collector/default.nix; enables spiffe for consistency.
• otel-collector.nix -- new nullable openobserve option that, when set, adds an otlphttp/openobserve exporter to all three pipelines while keeping debug. Default null keeps existing tests and standalone use unchanged.
• hosts-common.nix -- wires the collector's openobserve.endpoint from the group's appHosts.monitoring, mirroring the spire serverAddress pattern. The collector reads its Authorization header from a file the deployment overlay provisions; that credential bootstrap is left deployment-side like the openbao/incus bootstraps.
• Registration: groups/hosting-provider.nix (placeholder node), applications/hosting-provider/default.nix (imports), openbao/openbao.nix (consumerHosts), checks/default.nix (test-monitoring-storage).
• monitoring-test.nix (new, test-monitoring-storage) -- co-locates the collector and OpenObserve on one node, posts a synthetic OTLP trace and metric to the collector's 4318, and asserts both are queryable back out of OpenObserve's search API. This proves the full receive -> process -> export -> store -> query path.

Testing

Run on fediversity-ci:

• test-monitoring-storage -- green (primary E2E)
• test-otel-collector-service, test-otel-wiring -- green (regression: the new collector option defaults to null)
• treefmt -- green

nix-unit-resources-otel fails only on the pre-existing offline-FOD limitation (the systemd-openbao source cannot be fetched from git.lix.systems in the sandbox); src/resources/otel is untouched by this change.

Follow-ups (out of scope)

• Grafana / dashboards on top of OpenObserve. (#1135)
• OpenObserve persistent object-store backend + retention tuning, left deployment-side like OpenBao's storage.
• Production openbao cross-node credential delivery (the test uses a static dev credential).

Disclaimer: I used a coding agent in the creation of this patch.

## What Adds a `monitoring` node to the hosting-provider group running **OpenObserve** as an all-signal telemetry store, and teaches the otel-collector (#1002) to additionally export traces, metrics, and logs to it. The collector previously fanned every signal only to the `debug` exporter, so nothing was stored -- it landed in the journal and was lost. #1002 listed "storage back-end" as the open follow-up, and parent issue #184 requires tests covering collection **and storage in a back-end**. OpenObserve ingests metrics, logs, and traces over OTLP from a single binary with no ClickHouse/Postgres/ZooKeeper (embedded sqlite meta + local disk in single-node mode), so it fits a small ~20-machine hosting provider. The original ask named "a prometheus node", but Prometheus stores metrics only and would drop the traces/logs that #184 cares most about; the decision (made with the maintainer) was to store all three signals in one backend. ## Changes - **`monitoring.nix`** (new) -- OpenObserve systemd unit under `services.fediversity-monitoring`. No `services.openobserve` NixOS module exists, so the unit is hand-written. The root password is delivered via `LoadCredential` + an `ExecStart` wrapper that exports `ZO_ROOT_USER_PASSWORD` from the credential file, keeping the secret out of the unit/store (same spirit as the S3/openbao secret handling). Outbound usage telemetry and the GeoIP database download are disabled. - **`monitoring/default.nix`** (new) -- thin application declaration, modeled on `otel-collector/default.nix`; enables spiffe for consistency. - **`otel-collector.nix`** -- new nullable `openobserve` option that, when set, adds an `otlphttp/openobserve` exporter to all three pipelines while keeping `debug`. Default null keeps existing tests and standalone use unchanged. - **`hosts-common.nix`** -- wires the collector's `openobserve.endpoint` from the group's `appHosts.monitoring`, mirroring the spire `serverAddress` pattern. The collector reads its `Authorization` header from a file the deployment overlay provisions; that credential bootstrap is left deployment-side like the openbao/incus bootstraps. - Registration: `groups/hosting-provider.nix` (placeholder node), `applications/hosting-provider/default.nix` (imports), `openbao/openbao.nix` (`consumerHosts`), `checks/default.nix` (`test-monitoring-storage`). - **`monitoring-test.nix`** (new, `test-monitoring-storage`) -- co-locates the collector and OpenObserve on one node, posts a synthetic OTLP trace and metric to the collector's `4318`, and asserts both are queryable back out of OpenObserve's search API. This proves the full receive -> process -> export -> store -> query path. ## Testing Run on `fediversity-ci`: - `test-monitoring-storage` -- green (primary E2E) - `test-otel-collector-service`, `test-otel-wiring` -- green (regression: the new collector option defaults to null) - `treefmt` -- green `nix-unit-resources-otel` fails only on the pre-existing offline-FOD limitation (the `systemd-openbao` source cannot be fetched from `git.lix.systems` in the sandbox); `src/resources/otel` is untouched by this change. ## Follow-ups (out of scope) - Grafana / dashboards on top of OpenObserve. (#1135) - OpenObserve persistent object-store backend + retention tuning, left deployment-side like OpenBao's storage. - Production openbao cross-node credential delivery (the test uses a static dev credential). Disclaimer: I used a coding agent in the creation of this patch.

kiara added 1 commit 2026-06-21 16:32:04 +02:00

feat(hosting-provider): add OpenObserve telemetry storage node ... d92d2abe0a

The otel-collector (#1002) only fanned signals to the `debug` exporter, so
nothing was stored. #184 requires tests covering collection + storage in a
back-end. Add a `monitoring` node running OpenObserve -- a single-binary,
all-signal store (traces/metrics/logs over OTLP, local disk + embedded sqlite,
no ClickHouse/Postgres) that suits a small hosting provider -- and teach the
collector to additionally export to it.

- `monitoring.nix`: OpenObserve systemd unit (`services.fediversity-monitoring`),
  root password delivered via `LoadCredential` + an `ExecStart` wrapper so the
  secret stays out of the store; outbound telemetry/GeoIP download disabled.
- `monitoring/default.nix` + registration in the group, app imports, openbao
  `consumerHosts`, and `checks/default.nix`.
- `otel-collector.nix`: nullable `openobserve` option adding an `otlphttp`
  exporter to all three pipelines; default null keeps existing behaviour.
- `hosts-common.nix`: wire the collector's endpoint from `appHosts.monitoring`,
  mirroring the spire `serverAddress` pattern.
- `monitoring-test.nix` (`test-monitoring-storage`): posts an OTLP trace and
  metric to the collector and asserts both are queryable back out of
  OpenObserve.

Production cross-node credential delivery, Grafana dashboards, and OpenObserve
persistent-storage/retention tuning are left as follow-ups.

Assisted-by: Claude:claude-opus-4-8

kiara added 1 commit 2026-06-21 16:41:54 +02:00

add ci yamls ... f8e3288f6f

Signed-off-by: Kiara Grouwstra <kiara@procolix.eu>

kiara changed title from WIP: Add OpenObserve telemetry storage node to the hosting-provider group to Add OpenObserve telemetry storage node to the hosting-provider group 2026-06-21 18:10:11 +02:00

kiara requested review from developers 2026-06-21 18:10:11 +02:00

kiara added 1 commit 2026-06-21 18:26:58 +02:00

test(monitoring): move storage test script into a .py file ... 7b9b61ec77

Keep the `test-monitoring-storage` driver script in
`monitoring-test.py` and `readFile` it from `monitoring-test.nix`, the
same way the operator app tests and `netbox-bootstrap-rest.py` keep their
Python in linted `.py` files. Scope-relax the `unresolved-reference` rule
for the new path in `ty.toml`, since the NixOS test driver injects
`monitoring`/`subtest`/`retry` as globals.

Assisted-by: Claude:claude-opus-4-8

kiara merged commit c8035caa28 into main 2026-06-21 18:34:48 +02:00

kiara deleted branch prometheus-otel 2026-06-21 18:34:48 +02:00

kiara referenced this pull request from a commit 2026-06-21 18:34:49 +02:00

Add OpenObserve telemetry storage node to the hosting-provider group (#1116)

kiara referenced this pull request 2026-06-28 13:25:04 +02:00

dashboards offer insight into collected telemetry #1135

kiara referenced this pull request 2026-06-28 14:18:19 +02:00

monitoring: provision OpenObserve dashboards (#1135) #1136

kiara referenced this pull request from a commit 2026-06-28 14:20:06 +02:00

monitoring: provision OpenObserve dashboards (#1135) (#1136)

kiara referenced this pull request 2026-06-28 15:58:28 +02:00

monitoring: openobserve credential delivery + garage object store (#1116) #1139

kiara referenced this pull request from a commit 2026-06-28 16:08:09 +02:00

monitoring: openobserve credential delivery + garage object store (#1116) (#1139)

Sign in to join this conversation.

Reviewers

No reviewers

fediversity/developers

Labels

Clear labels   

0 points

fibonacci complexity: 0 points

  

0.5 points

fibonacci complexity: 0.5 points

  

1 point

fibonacci complexity: 1 point

  

13 points

fibonacci complexity: 13 points

  

2 points

fibonacci complexity: 2 points

  

21 points

fibonacci complexity: 21 points

  

3 points

fibonacci complexity: 3 points

  

34 points

fibonacci complexity: 34 points

  

5 points

fibonacci complexity: 5 points

  

55 points

fibonacci complexity: 55 points

  

8 points

fibonacci complexity: 8 points

  

ambition

application-offering

  

ambition

configure-applications

  

ambition

front-end

  

ambition/install-applications

  

ambition

security

  

ambition

switch-host

  

ambition

update-applications

  

ambition

user-management

  

api service

features that may relate more to a (to be created) API service

  

blocked

  

component: fediversity panel

  

component: nixops4

  

documentation

  

estimation high: >3d

  

estimation low: <2h

  

estimation mid: <8h

  

infinite points

fibonacci complexity: infinite points

  

productisation

features out-scoped with the decoupling of the productisation effort

  

project-management

Project management related tasks

  

question

  

role: application developer

  

role: application operator

  

role: hosting provider

  

role: maintainer

  

security

  

technical debt

  

testing

  

type unclear

  

type: bug

Anything that doesn't work as advertised

  

type: deliverable

  

type: key result

  

type: objective

  

type: task

  

type: user story

  

user experience

No labels

0 points

0.5 points

1 point

13 points

2 points

21 points

3 points

34 points

5 points

55 points

8 points

ambition

application-offering

ambition

configure-applications

ambition

front-end

ambition/install-applications

ambition

security

ambition

switch-host

ambition

update-applications

ambition

user-management

api service

blocked

component: fediversity panel

component: nixops4

documentation

estimation high: >3d

estimation low: <2h

estimation mid: <8h

infinite points

productisation

project-management

question

role: application developer

role: application operator

role: hosting provider

role: maintainer

security

technical debt

testing

type unclear

type: bug

type: deliverable

type: key result

type: objective

type: task

type: user story

user experience

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

fediversity/fediversity!1116

No description provided.