Evaluation of secret management schemes

2024-12-03 Robert, Nicolas, Valentin

Requirements

Store and manage secrets in a central place
Must be able to rotate keys (some state management)
Minimal state on contributors' end, ideally exactly one per-user credential or even SSO

Don't need (or need only very basic) RBAC, all contributors are equal (maybe infra admins have special access)
Components which require secrets don't have to be a secret (this would be a requirement for personal setups, where we don't want to leak e.g. which accounts exist)
No need to retrieve secrets for very old versions
No need for forward secrecy (thoroughly destroying keys as required by e.g. secure messaging protocols)

Storing secrets

Some secrets need to be persisted, and there are multiple formats and technologies to do that.
Managing secrets

Secrets need to be shared with contributors, and changed or rotated. Different systems have different degrees of comfort for these operations.
Deploying secrets

Secrets need to be made available to programs and services.
Versioning

For key rotation we need at least two versions: old to access the machine, new for rotating in
Setup complexity

Different systems have different requirements to get going, and may require more or less manual intervention for new contributors. This distinguishes:
- complexity to set up for experts
- complexity to contribute as a beginner
Scalability, sustainability

Questions to consider:
- What if a contributor works on 100 such projects?
- What if a project has 100 contributors?
- What if a project runs over 10 years, how much effort does secret handling incur?
- What if someone messes up the central server?
- How fast can we set up a working system?
- How hard is it to migrate from one scheme to another?

Name	management	deployment	storage	versioning	setup	scalability
agenix	yes (CLI)	yes (tempfiles)	repo (age)	Git	partially manual	details
sops-nix	yes (CLI)	yes (tempfiles)	repo (SOPS)	Git	partially manual	more moving parts than agenix
Vaultwarden	yes (web GUI)	no	database	yes, on demand	details	more up-front effort
ssh/scp	yes (manual)	yes (manual)	per-user	manually	details	details

allows reusing password handling workflows (typically better automation than for ssh keys)
more up-front work for initial deployment
disaster recovery needs special care, doesn't implicitly distribute copies to contributors
less interaction for managing contributor access
separate source of truth (workflows, audit log, etc.) as opposed to everything in the Git repo
adds an extra security boundary; encrypted secrets are not world-readable

requires taking care of distributing keys
per-user key management typically not automated, requires taking care of that separately

Managing the interface between public confiuration and secrets is a concern of the code
- For a scalable setup you want something like modules that take secrets as settings
It is possible to split the git-stored secret schemes into private repositories
- Then you have to handle synchronisation, e.g. by importing the public part from the secret part
- This would incur extra overhead for managing access, but that would be the same workflow as managing access to the rest of the Git server
With secrets stored in Git there's a potential for running into merge conflicts, which can be avoided but requires extra care
- Probably you want a monorepo for the entire organisation
  - Separating public and private parts through git subtrees is possible but requires even more care and automation/tooling when managing outside contributions
  - The upfront effort may be similar (but different in nature) to deploying and maintaining a Vaultwarden server
There's an experience and skill issue involved in maintaining a sophisticated Git repo or a live server, and what is more appropriate will depend on who will be responsible for the setup long-term