NixOS tests break when CI runs them in parallel

kiara commented

2025-06-04 19:38:54 +02:00

Owner

as per @Niols, our NixOS tests often break because they're all running CI at the same time. a common symptom seems Input/output error. options to address this seem to include:

As a work-around to get these to work again for now, we could limit the runner to capacity 1 so that it only ever takes one job. While slow, this would be better than CI failing altogether.
isolate the CI runs. Note that this poses a challenge of providing enough isolation to resolve the problem, without unnecessarily affecting performance. Options include:
- have many runners (e.g. nix containers - see #405) running on shared resources while trying not to pay cost of isolation too much. with our hardware machine Forgejo-CI this might be our best option.
- ~~many Proxmox VMs that each only accept one job, tho~~ this might be excruciatingly slow.
if we manage the above and builds no longer come from the same store, we should use a Nix cache (#92) shared between our runners and ourselves, as some of this stuff is expensive to build, yet sharing the Nix daemon doesn't work well

c.f. CI resource issues (#13)

CI comparison

CI	license	nix integration	forgejo integration	secret scoping	concurrency groups (to cancel previous / overridden commits)	parallel runs (#362)
argunix	✅ GPL-3.0+	✅	✅	?	✅	?
botanix (talk)	✅ EUPL	✅	✅	❌	?	✅ thru `nix-eval-jobs`
`gradient` (supports OIDC, not yet LDAP)	✅ AGPL-3.0	✅	✅?	thru deploy `apiKeyFiles`
`buildbot`	✅ GPL2.0	✅ `buildbot-nix` (MIT)	✅ thru `buildbot-nix` or (probably) gitea plugin	secrets incl. effects	✅	~~test~~ ✅ thru `nix-eval-jobs`
`jenkins`	✅ MIT	✅ wiki, test, `services.jenkins` / `services.jenkinsSlave`	✅	✅	✅	nodes added manually or on demand by cloud providers incl. nomad/k8s/docker/mesos/vsphere/jclouds/libvirt/cloudify/buildstash/sharing/proxmox
`hydra` `services.hydra`	✅ GPL-3.0	✅ made for nixpkgs	hooks / ✅ tokens	❌	?	✅ thru `nix-eval-jobs`
concourse	✅ Apache-2.0	concourse.nix	?	?	?	?
`gitea` / `forgejo` runner	✅ GPLv3+	no special support but allow native runners	✅	✅	✅ functionality present, ~~tho i did not get it to cancel on push-force~~	✅
`spindle`	✅ MIT	✅ thru nixery	instead integrates with forge tangled, which seems maybe like a better forge	❌ no mention in docs?	❌ no mention in docs?	?
`woodpecker` / crow	✅ Apache-2.0	e.g. `flake-pipeliner`	✅	✅	❌	✅
`dagger`	✅ Apache-2.0	?	?	?	❌
`agola`	✅ Apache-2.0	?	✅	?	❌
`eka-ci`	✅ AGPL-3.0	? (made for eka over vanilla nix)		❌ (also just seems unfinished)
`garnix`	✅ BSD-3	✅	❌ github-only	❌	?	✅?
`typhon`	✅ AGPL3.0	✅		❌ (plus abandoned-ish)
`buildkite`	❌ runner MIT, web UI proprietary - ~~may be acceptable using gerrit (#302)?~~ tokens depend on their proprietary platform	✅ thru nix generators)	✅ by `buildkite-forgejo-webhook`	✅ thru policies	✅ (with env vars)
`hercules-ci`	❌ proprietary	✅
`nix-ci`	❌ proprietary	✅

as per @Niols, our NixOS tests often break because they're all running CI at the same time. a common symptom seems `Input/output error`. options to address this seem to include: - [x] As a work-around to get these to work again for now, we could limit the runner to `capacity` 1 so that it only ever takes one job. While slow, this would be better than CI failing altogether. - [ ] isolate the CI runs. Note that this poses a challenge of providing enough isolation to resolve the problem, without unnecessarily affecting performance. Options include: - have many runners (e.g. nix containers - see #405) running on shared resources while trying not to pay cost of isolation too much. with our hardware machine Forgejo-CI this might be our best option. - ~~many Proxmox VMs that each only accept one job, tho~~ this might be excruciatingly slow. - [ ] if we manage the above and builds no longer come from the same store, we should use a Nix cache (#92) shared between our runners and ourselves, as some of this stuff is expensive to build, yet sharing the Nix daemon doesn't work well c.f. CI resource issues (#13) ### CI comparison | CI | license | nix integration | [forgejo integration](https://git.fediversity.eu/fediversity/fediversity/settings/hooks) | secret scoping | concurrency groups (to cancel previous / overridden commits) | parallel runs (#362) | |-|-|-|-|-|-|-| | [argunix](https://codeberg.org/tfc/argunix) | ✅ GPL-3.0+ | ✅ | ✅ | ? | ✅ | ? | | [botanix](https://git.afnix.fr/botanix/botanix) ([talk](https://media.ccc.de/v/lixcon-2026-8-botanix-a-nix-native-ci-in-a-forge-agnostic-world)) | ✅ EUPL | ✅ | ✅ | ❌ | ? | ✅ thru `nix-eval-jobs` | | [`gradient`](https://github.com/Wavelens/Gradient) (supports OIDC, not yet LDAP) | ✅ AGPL-3.0 | ✅ | [✅](https://github.com/wavelens/gradient/issues/25)? | [thru deploy `apiKeyFiles`](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/wavelens/gradient/master/docs/gradient-api.yaml) | | [`buildbot`](https://github.com/buildbot/buildbot) | ✅ GPL2.0 | ✅ [`buildbot-nix`](https://github.com/nix-community/buildbot-nix/) (MIT) | ✅ [thru `buildbot-nix`](https://github.com/nix-community/buildbot-nix/?tab=readme-ov-file#step-3-configure-buildbot-nix) or (probably) [gitea plugin](https://github.com/lab132/buildbot-gitea) | [secrets](https://docs.buildbot.net/current/manual/secretsmanagement.html) incl. [effects](github.com/nix-community/buildbot-nix/#experimental-hercules-ci-effects) | [✅](https://buildbot.readthedocs.io/en/stable/manual/configuration/services/old_build_canceller.html) | ~~[test](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/buildbot.nix)~~ ✅ thru `nix-eval-jobs` | | [`jenkins`](https://www.jenkins.io/) | ✅ MIT | ✅ [wiki](https://wiki.nixos.org/wiki/Jenkins), [test](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/jenkins.nix), [`services.jenkins`](https://search.nixos.org/options?channel=unstable&show=services.jenkins.jobBuilder.jsonJobs&query=services.jenkins.) / [`services.jenkinsSlave`](https://search.nixos.org/options?channel=unstable&show=services.jenkins.jobBuilder.jsonJobs&query=services.jenkinsSlave.) | [✅](https://github.com/go-gitea/gitea/issues/18299#issuecomment-1044210813) | [✅](https://www.jenkins.io/doc/developer/security/secrets/) | [✅](https://community.jenkins.io/t/how-to-cancel-redundant-builds-on-branches-pulls-but-not-master/5381/2) | nodes added [manually](http://localhost:8080/manage/computer/) or on demand by [cloud providers](http://localhost:8080/manage/pluginManager/available?filter=Cloud+Providers) incl. nomad/k8s/docker/mesos/vsphere/jclouds/libvirt/cloudify/buildstash/sharing/proxmox | | [`hydra`](https://github.com/NixOS/hydra/) [`services.hydra`](https://search.nixos.org/options?channel=unstable&query=services.hydra) | ✅ GPL-3.0 | ✅ made for nixpkgs | [hooks](https://github.com/NixOS/hydra/blob/master/doc/manual/src/webhooks.md) / ✅ [tokens](https://discourse.nixos.org/t/hydra-integration-without-exposing-authentication-tokens-to-the-nix-store/13117) | ❌ | ? | ✅ thru `nix-eval-jobs` | | [concourse](https://concourse-ci.org/) | ✅ Apache-2.0 | [concourse.nix](https://discourse.nixos.org/t/concourse-nix-declaratively-manage-concourse-ci/75853) | ? | ? | ? | ?| ? | | [`gitea`](https://docs.gitea.com/next/usage/actions/act-runner/) / [`forgejo` runner](https://forgejo.org/docs/latest/admin/actions/) | ✅ GPLv3+ | no special support but allow native runners | ✅ | [✅](https://forgejo.org/docs/next/user/actions/reference/#onpull_request) | ✅ [functionality present](https://forgejo.org/docs/next/user/actions/reference/#concurrencygroup), ~~tho i did not get it to cancel on push-force~~ | ✅ | | [`spindle`](https://docs.tangled.org/spindles.html#pipelines) | ✅ MIT | ✅ [thru nixery](https://docs.tangled.org/spindles.html#pipelines) | instead integrates with forge [tangled](https://tangled.org/), which seems maybe like a better forge | ❌ no mention in docs? | ❌ no mention in docs? | ? | | [`woodpecker`](https://woodpecker-ci.org/) / [crow](https://www.reddit.com/r/opensource/comments/1ittoe9/crow_ci_dronewoodpecker_fork/) | ✅ Apache-2.0 | e.g. [`flake-pipeliner`](https://github.com/pinpox/woodpecker-flake-pipeliner) | ✅ | ✅ | [❌](https://github.com/woodpecker-ci/woodpecker/issues/1461) | ✅ | | [`dagger`](https://dagger.io/) | ✅ Apache-2.0 | ? | ? | ? | ❌ | | [`agola`](https://agola.io/) | ✅ Apache-2.0 | ? | ✅ | ? | ❌ | | [`eka-ci`](https://github.com/ekala-project/eka-ci) | ✅ AGPL-3.0 | ? (made for eka over vanilla nix) | | ❌ (also just seems unfinished) | | [`garnix`](https://github.com/garnix-io/garnix-ci) | ✅ BSD-3 | ✅ | ❌ github-only | [❌](https://garnix.io/docs/actions/#secrets-with-actions) | ? | ✅? | | [`typhon`](https://typhon-ci.org/) | ✅ AGPL3.0 | ✅ | | ❌ (plus abandoned-ish) | | [`buildkite`](https://github.com/buildkite/agent) | ❌ runner MIT, web UI proprietary - ~~may be acceptable using gerrit (#302)?~~ tokens [depend on their proprietary platform](https://buildkite.com/docs/agent/v3/tokens#create-a-token) | ✅ [thru](https://git.snix.dev/snix/snix/src/branch/canon/nix/buildkite/default.nix) [nix](https://discourse.nixos.org/t/announcing-nixkite-buildkite-pipelines-using-the-nixos-module-system/7266) [generators](https://github.com/hackworthltd/nix-buildkite-plugin/)) | ✅ by [`buildkite-forgejo-webhook`](https://github.com/rscorer/buildkite-forgejo-webhook) | ✅ [thru policies](https://buildkite.com/docs/pipelines/security/secrets) | [✅](https://buildkite.com/docs/pipelines/configure/workflows/controlling-concurrency) (with [env vars](https://buildkite.com/docs/pipelines/configure/environment-variables#variable-interpolation)) | | [`hercules-ci`](https://hercules-ci.com/) | ❌ proprietary | ✅ | | [`nix-ci`](https://nix-ci.com/) | ❌ proprietary | ✅ |

kiara added the

type: bug

label

2025-06-04 19:38:54 +02:00

Niols referenced this issue

2025-06-05 10:01:35 +02:00

Introduce test for deploying all services via FediPanel #361

kiara added this to the Fediversity project

2025-06-18 14:33:07 +02:00

kiara referenced this issue

2025-06-20 12:15:31 +02:00

WIP: add attic cache #397

kiara referenced this issue

2025-06-24 15:06:35 +02:00

WIP: Compartimentalise CI using Docker #393

kiara referenced this issue

2025-06-24 15:06:47 +02:00

try out existing nix container made for gitea actions #405

~~kiara referenced this issue 2025-07-08 18:57:27 +02:00~~

WIP: add attic cache #397

~~kiara referenced this issue 2025-07-27 08:55:56 +02:00~~

WIP: add woodpecker CI #479

~~kiara referenced this issue 2025-07-30 12:51:40 +02:00~~

WIP: add woodpecker CI #479

~~kiara referenced this issue 2025-08-04 22:02:23 +02:00~~

use dedicated Nix builder #366

kiara referenced this issue

2025-08-05 11:33:00 +02:00

WIP: add woodpecker CI #479

kiara self-assigned this

2025-08-07 15:13:32 +02:00

kiara referenced this issue

2025-09-13 12:57:59 +02:00

use dedicated Nix builder #366

kiara added a new dependency

2025-12-02 00:45:49 +01:00

#224 automated dev-ops workflows

~~kiara referenced this issue 2025-12-02 00:46:35 +01:00~~

automated dev-ops workflows #224

kiara

2025-12-02 19:11:52 +01:00

closed this issue
added the
2 points
label

~~kiara referenced this issue 2025-12-14 19:11:35 +01:00~~

secrets may be handled securely w.r.t. use in PRs #463

kiara referenced this issue

2025-12-15 15:44:47 +01:00

secrets may be handled securely w.r.t. use in PRs #463

kiara

2025-12-15 15:45:05 +01:00

reopened this issue
added
5 points
and removed
2 points
labels

kiara added a new dependency

2026-01-29 12:13:55 +01:00

#704 tests' memory metadata

kiara referenced this issue

2026-01-29 12:14:09 +01:00

tests' memory metadata #704

kiara commented

2026-02-02 17:00:20 +01:00

Author

Owner

could maybe be fixed by using containers over VMs?

could maybe be fixed by using [containers over VMs](https://github.com/NixOS/nixpkgs/pull/479968)?

kiara referenced this issue

2026-03-03 11:59:38 +01:00

code reviewers can suggest changes #302

kiara commented

2026-04-05 13:55:58 +02:00

Author

Owner

isolation approaches to try (given node access) short of switching CI:

ci-multi-runner - so far had builds not go thru over OOMs, retry after #721
ci-microvm-runners - haven't had the hardware to test, 16 GB RAM seems not enough for our big tests while our CI node has 32 GB RAM.

isolation approaches to try (given node access) short of switching CI: - [`ci-multi-runner`](https://git.fediversity.eu/fediversity/fediversity/compare/main...kiara:ci-multi-runner) - so far had builds not go thru over OOMs, retry after #721 - [`ci-microvm-runners`](https://git.fediversity.eu/fediversity/fediversity/compare/main...kiara:ci-microvm-runners) - haven't had the hardware to test, 16 GB RAM seems not enough for our big tests while our CI node has 32 GB RAM.

kiara referenced this issue

2026-04-09 23:50:54 +02:00

raise CI runner MemoryMax from 16G to 24G #815

kiara referenced this issue

2026-04-21 10:57:22 +02:00

scale up CI runners #811

kiara added a new dependency

2026-04-21 10:58:21 +02:00

#811 scale up CI runners

kiara referenced this issue

2026-04-25 14:30:12 +02:00

Beefier Forgejo actions machines #13

kiara commented

2026-05-06 21:15:57 +02:00

Author

Owner

manually spinned up runner user services at king seem to no longer reproduce this issue - which perhaps the mentioned ci-multi-runner set-up could help reproduce

manually spinned up runner user services at `king` seem to no longer reproduce this issue - which perhaps the mentioned `ci-multi-runner` set-up could help reproduce

kiara referenced this issue

2026-05-09 14:06:14 +02:00

run multiple CI runner instances with resource limits (#362) #919

kiara referenced this issue from a commit

2026-05-09 14:06:27 +02:00

run multiple CI runner instances with resource limits (#362) (#919)

kiara commented

2026-05-09 14:36:44 +02:00

Author

Owner

ci-multi-runner deployed again thru #919, let's see

`ci-multi-runner` deployed again thru #919, let's see

kiara referenced this issue

2026-05-11 09:32:10 +02:00

enqueuing deployment syncs #242

kiara referenced this issue

2026-05-15 18:05:47 +02:00

enqueuing deployment syncs #242

kiara commented

2026-05-24 11:59:57 +02:00

Author

Owner

reflecting on this now, i wonder if this may have just been mostly about OOMs

kiara commented

2026-06-02 21:51:38 +02:00

Author

Owner

with #13 this seems to just work, implying this indeed had been about resource constraints rather than a software issue. an anti-climatic resolution, but glad to see this finally resolved.

kiara closed this issue

2026-06-02 21:51:38 +02:00

kiara commented

2026-06-02 22:38:39 +02:00

Author

Owner

seems not fixed, e.g. https://git.fediversity.eu/fediversity/fediversity/actions/runs/48123/jobs/0/attempt/1 still fails on 'Bad file descriptor'

kiara reopened this issue

2026-06-02 22:38:39 +02:00

kiara removed a dependency

2026-06-06 13:47:50 +02:00

#811 scale up CI runners