gatus: replace uptime-kuma with declarative config-as-code monitoring #23

Open

hermes wants to merge 1 commit from feat/gatus-replaces-kuma into main

pull from: feat/gatus-replaces-kuma

merge into: repomaa:main

repomaa:main

repomaa:hermes-dashboard-auth

repomaa:fix/ci-deploy-polkit

repomaa:pr/hastebin-client-sops

repomaa:ci/fqdn-deploy-trigger

repomaa:hermes-dashboard

repomaa:hermes-fallback

repomaa:fix/apu-wake-ifd

repomaa:chore/remove-stray-files

repomaa:chore/turny-use-nix-aspect

repomaa:chore/colmena-drop-dead-ssh-specialarg

repomaa:fix/mkenableoption-descriptions

repomaa:chore/dedupe-flake-file-inputs

repomaa:fix/dedupe-unstable-overlay

repomaa:forgejo-sse-buffering

repomaa:hermes-free-response-room

repomaa:hermes-rbw

repomaa:hermes-webhooks

repomaa:feat/dendritic-refactor

repomaa:tech/native-immich-module

repomaa:refactor

Conversation 0 Commits 1 Files changed 6 +218 -44

hermes commented 2026-09-27 10:44:17 +03:00

Collaborator

Copy link

Replaces Uptime Kuma with Gatus, which keeps all monitoring config in version control — the whole reason for the swap: kuma stored every monitor in its SQLite DB via the web UI, unreviewable and unreproducible from the flake.

What's in the new webservices/gatus aspect

• Active endpoint checks for every vhost the fleet serves (the same set the glance monitor widget listed: public freun.dev services, in.freun.dev internal ones, and the alderaan.space *arr stack), conditions [STATUS] < 500 + [CERTIFICATE_EXPIRATION] > 48h, 1m interval.
• External (push) endpoints with heartbeat.interval = 26h — dead man switches for:
  • automatic-upgrade-<host> for apu/freun-dev/radish/turny/pumpkin (replaces the kuma push tokens; auto-upgrade now pushes success=true/false to /api/v1/endpoints/jobs_automatic-upgrade-<host>/external with a per-host bearer token),
  • backup for radish's borg job (replaces UPTIME_KUMA_TOKEN with GATUS_TOKEN_BACKUP).
• Email alerting through the existing smtp relay (modules.smtp), reusing grafana's smtp_password sops key — no duplicate credential. Default alert: failure-threshold 2, send-on-resolved.
• Dashboard auth: gatus security.basic with username jokke, bcrypt hash served from sops via env interpolation so it never enters the store. The push API (/api/v1/endpoints/*/external) sits outside basic auth upstream and is token-authenticated.
• SQLite storage in the StateDirectory, behind services.webserver on the same vhost as before (status.freun.dev, port 3007).

Verified

• nix eval ...toplevel.drvPath succeeds for freun-dev, apu, turny, pumpkin (radish's eval failure — private tree_hugger github input — is pre-existing on main).
• nix build ...toplevel succeeds for freun-dev with stubbed secrets; the generated gatus.yaml and unit were inspected (env files attached, tokens interpolated from sops env).

⚠ Required before merge (sops rekey, needs your yubikey)

1. secrets/auto-upgrade.yaml: rename key tree kuma-push/<host> → token/upgrade/<host> (same per-host values), and add token/backup (copy the value currently in secrets/backup.yaml as UPTIME_KUMA_TOKEN — the backup heartbeat token can literally be the same secret, kuma tokens were just opaque strings).
2. secrets/backup.yaml: replace UPTIME_KUMA_TOKEN with GATUS_TOKEN_BACKUP (same value is fine).
3. Create secrets/gatus.yaml with key basic_auth_bcrypt: a bcrypt hash (cost ≥ 9) of the dashboard password, base64-encoded after hashing, e.g.:

   htpasswd -bnBC 9 "" '<password>' | tr -d ':\n' | base64
   

   (the fallback creation rule already covers secrets/gatus.yaml — yubikey + freun-dev age key; add other hosts only if they'll ever need to read it, they don't).

Until those land, hosts will fail activation on sops-install-secrets (verified: missing keys are the only remaining failure).

Reverting

Single revert commit restores kuma; the old kuma SQLite state dir (/var/lib/uptime-kuma) is untouched by this PR, so no monitor data is lost either way.

Replaces Uptime Kuma with [Gatus](https://gatus.io), which keeps all monitoring config in version control — the whole reason for the swap: kuma stored every monitor in its SQLite DB via the web UI, unreviewable and unreproducible from the flake. ## What's in the new `webservices/gatus` aspect - **Active endpoint checks** for every vhost the fleet serves (the same set the glance monitor widget listed: public freun.dev services, `in.freun.dev` internal ones, and the alderaan.space *arr stack), conditions `[STATUS] < 500` + `[CERTIFICATE_EXPIRATION] > 48h`, 1m interval. - **External (push) endpoints** with `heartbeat.interval = 26h` — dead man switches for: - `automatic-upgrade-<host>` for apu/freun-dev/radish/turny/pumpkin (replaces the kuma push tokens; `auto-upgrade` now pushes `success=true/false` to `/api/v1/endpoints/jobs_automatic-upgrade-<host>/external` with a per-host bearer token), - `backup` for radish's borg job (replaces `UPTIME_KUMA_TOKEN` with `GATUS_TOKEN_BACKUP`). - **Email alerting** through the existing smtp relay (`modules.smtp`), reusing grafana's `smtp_password` sops key — no duplicate credential. Default alert: failure-threshold 2, send-on-resolved. - **Dashboard auth**: gatus `security.basic` with username `jokke`, bcrypt hash served from sops via env interpolation so it never enters the store. The push API (`/api/v1/endpoints/*/external`) sits outside basic auth upstream and is token-authenticated. - SQLite storage in the StateDirectory, behind `services.webserver` on the same vhost as before (`status.freun.dev`, port 3007). ## Verified - `nix eval ...toplevel.drvPath` succeeds for freun-dev, apu, turny, pumpkin (radish's eval failure — private `tree_hugger` github input — is pre-existing on `main`). - `nix build ...toplevel` succeeds for freun-dev **with stubbed secrets**; the generated `gatus.yaml` and unit were inspected (env files attached, tokens interpolated from sops env). ## ⚠ Required before merge (sops rekey, needs your yubikey) 1. **`secrets/auto-upgrade.yaml`**: rename key tree `kuma-push/<host>` → `token/upgrade/<host>` (same per-host values), and add `token/backup` (copy the value currently in `secrets/backup.yaml` as `UPTIME_KUMA_TOKEN` — the backup heartbeat token can literally be the same secret, kuma tokens were just opaque strings). 2. **`secrets/backup.yaml`**: replace `UPTIME_KUMA_TOKEN` with `GATUS_TOKEN_BACKUP` (same value is fine). 3. **Create `secrets/gatus.yaml`** with key `basic_auth_bcrypt`: a bcrypt hash (cost ≥ 9) of the dashboard password, base64-encoded *after* hashing, e.g.: ``` htpasswd -bnBC 9 "" '<password>' | tr -d ':\n' | base64 ``` (the fallback creation rule already covers `secrets/gatus.yaml` — yubikey + freun-dev age key; add other hosts only if they'll ever need to read it, they don't). Until those land, hosts will fail activation on `sops-install-secrets` (verified: missing keys are the *only* remaining failure). ## Reverting Single revert commit restores kuma; the old kuma SQLite state dir (`/var/lib/uptime-kuma`) is untouched by this PR, so no monitor data is lost either way.

hermes added 1 commit 2026-09-27 10:44:17 +03:00

gatus: replace uptime-kuma with declarative config-as-code monitoring ... c7a9e60e37

- new webservices/gatus aspect: sqlite storage, basic auth for the
  dashboard (bcrypt from sops), email alerting over the existing smtp
  relay, active endpoint checks for every public/internal/media vhost,
  and external (push) endpoints with heartbeats for the per-host
  automatic-upgrade dead man switches and the radish backup job
- auto-upgrade heartbeats now push success/failure booleans to
  /api/v1/endpoints/*/external with per-host bearer tokens
- radish backup postCreate pushes its heartbeat the same way
- fqdns.uptime-kuma -> fqdns.gatus (same vhost, status.freun.dev)

requires rekeying secrets before merge: rename kuma-push/<host> to
token/upgrade/<host>, add token/backup, create secrets/gatus.yaml with
basic_auth_bcrypt; see PR comment

hermes force-pushed feat/gatus-replaces-kuma from c7a9e60e37 to f54fe60342 2026-09-27 10:45:35 +03:00 Compare

hermes force-pushed feat/gatus-replaces-kuma from f54fe60342 to 0a9cf23c4a 2026-09-27 10:53:18 +03:00 Compare

repomaa force-pushed feat/gatus-replaces-kuma from 0a9cf23c4a to 8bb372ad91 2026-09-27 22:48:50 +03:00 Compare

repomaa force-pushed feat/gatus-replaces-kuma from 8bb372ad91 to e6db2174ea 2026-09-27 22:56:23 +03:00 Compare

All checks were successful

Hide checks

Build Images / build (pull_request) Successful in 40s

Details

Check / check (pull_request) Successful in 3m54s

Required

Details

This pull request is blocked because it's outdated.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/gatus-replaces-kuma:feat/gatus-replaces-kuma

git switch feat/gatus-replaces-kuma

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

repomaa/nixos!23

No description provided.