Skip to content

Renovate Automation Roadmap

The goal: stop hand-merging every Renovate PR from issue #1 and only spend attention on the updates that actually break things.

This is a staged rollout. We expand the auto-merge blast radius one trust boundary at a time, only after the prior tier has been quiet for long enough to trust it.

Why not auto-merge everything today

Three concrete failure modes:

  1. Sidecar coupling. home-assistant runs code-server as a sidecar in the same pod, so a code-server patch bumps and restarts Home Assistant. Any rule that touches home-automation/ has to treat the whole pod as one unit.
  2. Apps that must move together. ha-mcp is meaningless without a matching home-assistant version — bumping it alone is a footgun. Z2M updates can broadcast devices that HA misses on startup, leaving Zigbee entities unavailable until HA is restarted.
  3. Stateful operators drift on upgrade. rook-ceph and cnpg upgrades regularly need hands-on recovery (PVC/pod deletion to re-seed CNPG replication, CRD/deprecation fixes for Rook). These must stay manual until we have a health gate that can detect drift and either roll back or page.

There is also no PR-gating CI today (.github/workflows/ only builds peloton-scraper and publishes mkdocs). So even if we flipped automerge: true right now, nothing would actually validate the change before it lands. Fixing the CI gap is the precondition for everything below.

What we're borrowing from onedr0p/home-ops and bjw-s-labs/home-ops

Two well-respected home-ops repos have solved most of this. We pull from both — onedr0p where the patterns are conservative and battle-tested, bjw-s where they show what "more aggressive once you trust it" looks like. The pieces worth porting:

  • flux-local PR action (from onedr0p's .github/workflows/flux-local.yaml). Runs flux-local test on every PR to validate that all HelmReleases and Kustomizations actually render, and runs flux-local diff helmrelease/kustomization to post the rendered diff as a sticky PR comment. This is the real auto-merge gate — without it, automerge: true is uninspected. We use onedr0p's docker:// invocation pattern (no runner pre-install needed) rather than bjw-s's shell-command pattern (which assumes flux-local is on the runner image).
  • Trusted-package auto-merge, not path-based. Both repos do this. The shape: whitelist specific packages by name/prefix, never auto-merge an entire directory. onedr0p uses home-operations/* digests + a handful of charts; bjw-s uses a broader prefix list (ghcr.io/home-operations, ghcr.io/onedr0p, ghcr.io/bjw-s, ghcr.io/bjw-s-labs). For us: start narrow with ghcr.io/thaynes43/* digests + kube-prometheus-stack minor/patch, expand later. Deliberate deviation (owned 2026-06-09): our Tier 2 also trusts whole leaf-app domains by path (media, ai, downloads, frontend, office, photos) — we trust the domain-placement decision itself: those directories only hold stateless single-pod apps. The cost is that the trust is implicit for future deps: any new app or sidecar added under those paths is auto-merge-trusted from day one. Standing rule: when adding an app to an allowlisted domain, decide at that moment whether it needs an Immich-style carve-out in .renovate/autoMerge.json5.
  • groupName + minimumGroupSize for must-move-together components. Used by both repos for kubernetes (5), flux-operator (3), rook-ceph (2), talos (2). Important semantic gotcha — see "Tier 3" below.
  • minimumReleaseAge: 3 days (onedr0p) to bake third-party tags before auto-merging. bjw-s has dropped this — they've earned the trust. We start with onedr0p's bake time and revisit later.
  • Renovate runtime options: onedr0p self-hosts Renovate via a GitHub Actions cron (.github/workflows/renovate.yaml); bjw-s runs Renovate in-cluster as renovate-operator (a GitOps-managed HelmRelease at kubernetes/apps/renovate/renovate-operator/). Both beat the hosted Renovate app for control and logs. Not Tier 1; revisit at Tier 2 or later. The in-cluster option is interesting because it puts the bot in the same lifecycle as the rest of the cluster.
  • Split config into .renovate/*.json5 files (autoMerge.json5, groups.json5, customManagers.json5, etc.) extended from the root. Both repos do this. Makes future tier work land as small reviewable diffs against individual files.

Tiers

Tier Scope Mode Status
0 github-actions minor/patch auto-merge ✅ live
1 flux-local PR gate required check on all Renovate PRs ✅ live (.github/workflows/flux-local.yaml)
2 Curated allowlist: own ghcr.io/thaynes43/*, kube-prometheus-stack, safe stateless leaf-app domains (media, ai, downloads, frontend, office, photos, observability since 2026-06-15) + curated cluster-infra leaves by subpath (kube-system/{metrics-server,reloader,reflector,k8tz,spegel}, network/cloudflare-ddns) on minor/patch auto-merge, flux-local-gated + bake ✅ live (.renovate/autoMerge.json5) — trust clock starts 2026-06-08 (see exit criteria)
3 Grouped multi-component apps: home-assistant (HA + code-server + ha-mcp), Z2M symmetric dashboard-approval groups (manual phase) ✅ live 2026-06-09 (.renovate/groups.json5)
4 rook-ceph, cnpg, Talos, Flux dashboard-approval + post-reconcile health-gate agent + shepherd + guardrails 🟢 Phase D live 2026-07-03 — scheduled shepherd auto-merge (daily 09:00 ET, one PR/run) for pure bumps of the clean-tier ramp set (see "Phase-D auto-merge ramp" below); everything else gets a shepherd-authored PR for human merge or stays untouched. Phases A–C (Kyverno enforce, diff-scope required, push protection) live 2026-07-02. See audit + handoff.

Resuming this roadmap (next session): Tiers 0–3 are live; Tier 3 runs in its manual dashboard-approval phase (updates for the HA pod and Z2M show up as checkboxes on the Dependency Dashboard, not as auto-opened PRs — tick to release them as one group PR). Next milestones: (a) Tier 2 promotion review on/after 2026-07-06 (four quiet weeks from the 2026-06-08 deadlock fix), (b) build the Tier 4 phase-4a health-gate agent — its runtime/credential decisions are locked in the Tier 4 section.

Tiers 0–2 are live. Tier 2 carve-out: immich-app/* is excluded from auto-merge (breaking schema/DB migrations even on minor) despite living in photos/. Ramp the allowlist by adding packages/domains as each proves quiet. The end state is 100% hands-off, with a Tier 4 health-gate agent shepherding the risky merges (the manual merge→reconcile→verify→rollback loop, automated). That agent needs GitHub Actions (a GITHUB_TOKEN-merged PR does not fire downstream workflows) + a scoped cluster credential.

Tier 1 — flux-local PR gate (next)

Why first: Without this, every other tier is auto-merging unchecked YAML. With it, even the manually-merged PRs get a rendered diff comment, which makes review faster.

What to port from onedr0p:

  • .github/workflows/flux-local.yaml, with two adaptations for this repo:
  • Two flux roots, not one. Main lives at kubernetes/main/flux (Kustomization cluster-apps → ./kubernetes/main/apps). Edge lives at kubernetes/edge/flux with the same shape. Both go in a matrix so each PR validates both clusters. Edge can be in the matrix even while it's powered off — flux-local test only validates that the YAML renders, it doesn't talk to the cluster.
  • Token strategy — see below. Start with GITHUB_TOKEN to avoid blocking on App registration; flip to a GitHub App once Tier 1 is proven.
  • The bjw-s-labs/action-changed-files filter step is what makes this cheap — only runs when kubernetes/** actually changed.

Token strategy: GITHUB_TOKEN vs GitHub App

onedr0p uses a GitHub App (BOT_APP_ID / BOT_APP_PRIVATE_KEY) instead of the default GITHUB_TOKEN. The reasons that matter for haynes-ops:

GITHUB_TOKEN GitHub App
Commits trigger downstream workflows ❌ blocked by GH ✅
Comment author identity github-actions[bot] your bot's name
Cross-repo install one repo only many repos, one credential
Rate limit 1k/hr/repo 5k/hr/install
Setup cost none ~10 min app registration

The killer feature is #1: with GITHUB_TOKEN, when Renovate auto-merges a PR, the resulting merge commit on main will not fire any push-triggered workflow (GitHub blocks this to prevent loops). With an App token, the merge looks like a real user push and downstream workflows run normally. For one cluster it's annoying-but-livable; for main + edge + future expansion it gets painful.

Plan: ship Tier 1 with GITHUB_TOKEN so we're not blocked on app registration, then register a haynes-ops-bot GitHub App and flip the secrets in once flux-local is proven green. Document the app registration steps in this file when we do it.

Exit criteria: the action runs green on at least one real Renovate PR on each of main and edge clusters, and the diff comment is useful enough to make the merge decision from the PR page alone. Then Tier 2.

Tier 2 — Trusted-package auto-merge

After Tier 1 is green, mirror onedr0p's autoMerge.json5 with a haynes-ops twist:

  • Own images on digest: automerge: true for docker digests where matchPackageNames: ["/thaynes43/"]. Rationale: when we bump our own image tag (like today's appdaemon 1.0.1), we tested it before pushing — Renovate picking up the digest is a no-brainer.
  • kube-prometheus-stack on minor/patch. Direct lift from onedr0p, observability has been quiet here for months. Datasource gotcha: it's sourced via OCIRepository (oci://ghcr.io/.../charts/kube-prometheus-stack), which Renovate tracks as the docker datasource — not helm. The rule must matchDatasources: ['docker'] (we use ['docker', 'helm'] to be source-agnostic); a ['helm']-only match silently never fires. This applies to any OCI-sourced chart added to the allowlist later.
  • Use ignoreTests: false on every rule so flux-local actually gates the merge. (Our existing GH-actions auto-merge rule sets ignoreTests: true because there's no test today — once Tier 1 lands we should flip it.)
  • minimumReleaseAge: 3 days on third-party packages, 1 minute on our own images.

Exit criteria: four consecutive weeks with no auto-merge regression traced to a Tier 2 rule. Each new package added to the trust list resets the clock for that package only, not the tier.

Trust clock starts 2026-06-08, not 2026-06-04. The rules existed from 06-04 but were deadlocked (branch-mode automerge vs a PR-only flux-local check — see the note atop autoMerge.json5) and never fired until the 06-08 fix; first confirmed auto-merge was #1829 on 06-09. Four quiet weeks of rules actually firing puts the promotion review at ≥ 2026-07-06.

Allowlist expansion 2026-06-15 — observability domain + curated infra leaves. Added observability/** as a whole leaf domain (every app is a plain Deployment — no StatefulSets, no operator CRs; grafana state lives in external CNPG, loki's only PVC is local log storage) and a second rule for curated cluster-infra leaves matched by explicit subpath: kube-system/{metrics-server,reloader,reflector,k8tz,spegel} and network/cloudflare-ddns. The subpath rule is deliberate: kube-system and network are not safe as whole domains — each holds a cluster-killer (cilium/coredns, traefik, authentik, multus, device-plugins) that GitOps can't self-heal if a bad push lands overnight, so only the stateless single-pod utilities are listed by name. Two things to keep an eye on as this bakes: prometheus-operator-crds (CRD bumps — additive on minor/patch, which is all the rule allows) and the meta-risk that observability is the safety net for the other Tier 2 merges (mitigated by the independent Gatus/Pushover/watchdog alert path). Per-package trust clocks for the new entries start 2026-06-15. Still manual (the irreducible core, → Tier 4): database operators (CNPG/dragonfly/EMQX), cilium, coredns, traefik, authentik, multus, device-plugins.

Tier 3 — Grouped multi-component apps

Home automation can't tier up by namespace because of sidecar coupling and companion-image coupling. The unit of update is the pod, not the file or the namespace.

Important semantic gotcha: minimumGroupSize does not mean "always bundle these together." It means "only form the group PR if N+ matching deps have updates available in the same Renovate scan." If only one matches, it ships as an individual PR under the normal rules. Both onedr0p and bjw-s use it, and that's fine for things like kubernetes-component bumps where the components naturally release together. It is the wrong tool for sidecar coupling.

Why the original two-layer design was scrapped (2026-06-09). The first draft put dependencyDashboardApproval: true on the satellites only (code-server, ha-mcp) plus a groupName across all three, expecting an HA bump to "sweep in" pending satellite updates. It can't work: package rules merge per-dependency, so each satellite ends up with both the approval flag and the group name — and Renovate holds approval-gated deps out of the group branch until each is individually approved. The group PR would ship HA alone while the satellites strand behind un-ticked dashboard checkboxes; worst case, ha-mcp silently drifts incompatibly behind HA — the exact footgun the group exists to prevent.

Locked design (live in .renovate/groups.json5): one group, symmetric approval. Every member of the HA pod group — home-assistant, coder/code-server, homeassistant-ai/ha-mcp — carries dependencyDashboardApproval: true and an explicit automerge: false.

Day-to-day workflow: member updates accumulate as checkboxes on the Dependency Dashboard instead of auto-opening PRs. Tick what you want to release → Renovate opens one group PR with everything ticked → flux-local renders the diff → manual merge → the HA pod restarts once with all bumps.

Accepted edge case: approving a satellite-only update produces a group PR containing just that satellite — one HA pod restart for a code-server bump. That's a restart-churn cost, not a compatibility risk, and a human chooses when to take it. (Two non-issues verified 2026-06-09: ha-mcp is third-party ghcr.io/homeassistant-ai/ha-mcp, not ours, so the Tier 2 thaynes43/* wildcard can't catch it; and the Tier 3 rules live in groups.json5, which extends after autoMerge.json5 — later rules win per-dependency — with automerge: false set explicitly as a belt-and-braces guard.)

Promotion rule — all-or-none. When Tier 3 graduates from manual approval to automation (after the Tier 4 health gate exists), remove dependencyDashboardApproval from all group members in the same commit and add a schedule. Never go asymmetric (HA automated, satellites gated): that recreates the strand problem the two-layer design died of.

Zigbee2MQTT gets its own approval-gated rule — no group (it has no companions). The risk is the HA-restart race: a Z2M update can broadcast devices while HA misses them on startup, leaving Zigbee entities unavailable until HA restarts. During the manual phase, approval is the schedule — you tick the box when you can babysit the reconcile; a schedule: would only add delay after an explicit approval, so it joins at the automation phase instead (before 6am on Monday, the bjw-s pattern). Post-merge check (manual now, exactly what the Tier 4 gate automates later): verify Zigbee entities are available; restart HA if not.

Candidate for the same babysat pattern later: zwave-js-ui (HA's zwave-js integration has a server-schema compatibility window). Left manual-by-default for now — it's in no allowlist, so it still opens ordinary PRs.

Even with grouping, Tier 3 stays dashboard-approval until the Tier 4 health gate exists — the Z2M/HA race is exactly the kind of thing the gate needs to catch automatically.

Tier 4 — Stateful operators with a health gate

Build status (2026-06-30): in progress. The keystone pieces are live — the haynes-ops-bot GitHub App (push/PR/merge identity + scripts/github-app-token.sh), the holds registry, and the three runbooks that encode the agent: the upgrade-shepherd manual (loop + 3 modes), the health gate checks, and the per-component playbooks + rollback. Remaining: stand up the gate runtime (cloud scheduled routine / in-cluster CronJob) and wire the shepherd as an invocable agent (phase 4a → 4b).

rook-ceph, cnpg, Talos, and Flux itself never auto-merge on tag alone. The plan is:

  1. Renovate opens the PR with dependencyDashboardApproval: true (no automatic merge ever).
  2. flux-local renders the diff in the PR comment, human approves the merge.
  3. A scheduled agent (cron trigger or /loop) watches Flux Kustomization status, cnpg cluster health, rook-ceph health, and HA Zigbee entity availability after every reconcile.
  4. On regression, the agent either rolls the HelmRelease back to the prior chart version or pages via the existing notification path.

Pre-merge gating cannot solve this — the failure modes only show up after reconcile. The agent is doing the work that no PR check can.

Decisions locked 2026-06-09:

  • Phase 4a — watch + page (build first): a Claude Code scheduled agent (cron trigger). Fastest to stand up, and the cluster-credential question is already solved: the read-only Omni service-account kubeconfig (runbook) gives it headless cluster access with no browser-OIDC dance. It runs the post-reconcile health checks above and pages (Pushover, the existing notification path) on regression. Rollback stays human in this phase — the agent's job is to make sure a bad merge never goes unnoticed, not yet to act on it.
  • Phase 4b — the hands-off endgame: once the checks have proven themselves, move the loop in-cluster (CronJob) and automate the rollback (revert commit or chart re-pin, pushed for Flux to reconcile). Automated rollback requires the haynes-ops-bot GitHub App — a GITHUB_TOKEN-pushed revert fires no downstream workflows (the long-standing Tier 1 token-strategy item), so the bot's pushes must look like user pushes.
  • Preconditions for 4a: Tier 3 quiet in manual mode, and the alert-noise cleanup done — the gate's signal is the alert/metric stream, and a noisy phone channel is how a regression gets missed. (This is already true today: Alertmanager is the de-facto runtime health gate for Tier 2's unsupervised overnight auto-merges.)

The upgrade-shepherd agent — three invocation modes

The Tier 4 agent is not only the scheduled health gate. It is one agent with a shared toolset — read-only cluster access via the Omni service-account kubeconfig (runbook), gh, repo read/write, flux-local, the Pushover page path, and the renovate-upgrade-batches runbook — invoked three ways:

  1. Scheduled gate (phase 4a, build first). Cron / /loop trigger. After every reconcile it runs the post-merge health checks (Flux Kustomization status, CNPG / Rook-Ceph / EMQX health, HA Zigbee availability) and pages on regression. Rollback stays human in 4a, automated in 4b.

  2. Summoned remediation (on-demand). When an upgrade fails — an alert fires, a Kustomization is stuck not Ready, or a post-merge regression is spotted — the agent is invoked directly (a chat session, or a page-reply / RemoteTrigger hook) to diagnose, attempt the documented remediation (revert the HelmRelease to the prior chart, re-pin a version, delete-and- reseed per the component runbook), and report back. This is the manual Rollback section of renovate-upgrade-batches.md, automated and on-call. This is the "summoned if an upgrade failed" capability.

  3. Breaking-change shepherd (the manual-tier upgrades). For the irreducible-manual set that will never blind-auto-merge — database operators, traefik, authentik, cilium, Talos, Flux, Rook/Ceph — the agent runs the renovate-upgrade-batches.md Tier 3 process: read the release notes / UPGRADING guide, grep our usage for the affected features, make the required values edits in a commit, merge one at a time, reconcile, verify, move on. This is the "reading release notes and making other changes to support breaking changes" capability — work a pre-merge PR check structurally cannot do.

Modes 2 and 3 are the second reason (besides automated rollback) the haynes-ops-bot GitHub App is required: the agent must push commits (reverts, value edits) that fire downstream workflows, which a GITHUB_TOKEN-merged PR does not. Until the App exists, modes 2/3 run as a supervised Claude Code session (the human is the push identity, exactly as in this 2026-06-15 backlog sweep); only the scheduled gate (mode 1) is purely read+page, so it can ship on the Omni SA kubeconfig alone.

Phase-D auto-merge ramp

The scheduled shepherd (CronJob upgrade-shepherd, daily 09:00 ET, MODE=auto) may auto-merge pure version/chart/digest bumps (plus the one typed supporting-edit shape below), for the components listed here. Since 2026-07-06 the ramp is one env var in shepherd/app/helmrelease.yaml: UPGRADE_AGENT_RAMP (component list). The component→path map lives in run-shepherd.sh (ramp_globs_for) and derives the pre-filter globs; a run-start consistency check asserts every ramp component is also named in UPGRADE_AGENT_PROMPT — drift makes the scheduled run refuse to vet (fail closed) and the gate pages shepherd/ramp-mismatch (page path proven 2026-07-06). Widen one component at a time, operator-approved, /kyverno-verify after each.

Cost control — the pre-filter + vet-once: before the scheduled run spends a cent, a deterministic check (prefilter_should_run in run-shepherd.sh) lists open Renovate PRs and matches their changed files against the ramp path prefixes. No in-scope PR ⇒ it exits $0 with no clone and no LLM call — most days cost nothing. It never decides mergeability (the LLM still fully vets every in-scope PR); it only gates whether there is work worth looking at, and fails open (runs the LLM) on any error. Vet-once (2026-07-06): the shepherd records each vet as a <!-- shepherd-vet sha=<head> --> PR comment; the pre-filter skips PRs already vetted at their current head SHA (a Renovate rebase re-arms), so a PR baking through minimumReleaseAge is paid for once, not daily. The scheduled run also refreshes the orphan-PR digest (open Renovate PRs

7d → upgrade-orphan-report CM → the gate pages weekly) so unowned PRs never age silently.

Typed supporting-edit auto-merge (2026-07-06, pattern 1): diff-scope admits exactly one supporting-edit shape — restoring automountServiceAccountToken: true (+ bare defaultPodOptions: parent) in a kubernetes/** helmrelease.yaml whose base already declares serviceAccount: (the app-template-v5 landmine). Worst case admitted: a pod mounts the SA it already declares — identity-minting and RBAC edits still hard-fail, an explicit false→true flip still gates, adversarial suite 26/26 (scripts/diff-scope-test.sh). The generic shape stays closed; future patterns are added one at a time, each with its own adversarial cases.

Structural backstops regardless of prompt: diff-scope (bot PRs must be pure bumps in kubernetes/**), required checks, non-admin bot, Kyverno enforce, the $50/mo spend guard, and the health gate + guardrail alerts.

Component Class In ramp since Notes
coredns stateless 2026-07-03 clean rollback
traefik stateless 2026-07-03 clean; internal→external order per playbook
multus stateless 2026-07-03 clean, node-wide blast on breakage (gate catches)
device-plugins stateless 2026-07-03 clean; Plex-unschedulable failure mode
flux stateless 2026-07-03 first proven auto-merge (#1917, v2.9.0, 2026-07-03)
immich majors stateful 2026-07-05 shepherd owns immich MAJORS only — backup-gated on postgres16-pgvecto (@daily ScheduledBackup), auto-merge a pure major / author-only if it needs a supporting edit. immich minor/patch auto-merge via Renovate Tier-2 (the old "immich=manual" carve-out was removed 2026-07-05).
rook-ceph (incl. ceph-csi) revertible cluster-infra 2026-07-08 chart/operator/csi PATCH + safe minor auto-merge IFF cephVersion pinned-unchanged (no Ceph daemon major) + Ceph HEALTH_OK; moves operator→csi→cluster as a unit. Ceph daemon MAJOR = one-way → authored 'ONE-WAY — human confirm'.
cnpg revertible cluster-infra 2026-07-08 OPERATOR chart bump auto-merge IFF PG imageName unchanged (no PG major). PG major = one-way (pg_upgrade) → authored for confirm.
dragonfly-operator revertible cluster-infra 2026-07-08 operator patch/minor auto-merge (in-memory flush on restart re-enqueues).
cilium revertible cluster-infra 2026-07-08 PATCH auto-merge; MINOR/MAJOR = one-way eBPF map layout → authored for confirm.
authentik revertible cluster-infra 2026-07-08 PATCH within same YYYY.M auto-merge; YYYY.M major = forward-only migration (one-way) → authored for confirm.
cert-manager revertible cluster-infra 2026-07-09 PATCH only auto-merge. MINOR/MAJOR authored for human review — the Helm schema is strict (additionalProperties:false) and minors REMOVE values keys (v1.21.0 dropped prometheus.servicemonitor.*), so they need release-note + values vetting.

Operating principle (2026-07-08): the split is REVERTIBLE vs ONE-WAY, not cluster-breaker vs not. "Cluster-breaker stays manual" was the wrong frame — a supervised Claude Code session merges these today with the exact steps the shepherd has (read notes, grep usage, backup, merge, verify, revert). The only real floor is recoverability: - Revertible (chart/operator/CSI bump, data plane untouched, pins intact) → auto-merge, backup/health-gated, auto-revert on regression. Zero human. This now includes rook/cnpg/dragonfly/cilium/authentik patches (see table) — not just the stateless tier. - One-way (Ceph daemon major, PG major, cilium eBPF minor/major, authentik YYYY.M major, Talos) → the shepherd still does 100% of the work and authors a one-click-ready ONE-WAY — human confirm PR, but does NOT blind-merge the irreversible step. On regression it can't auto-revert, so a human is the break-glass contact for irreversible changes only — the same exposure you have with a supervised agent. - Renovate Tier-2 still auto-merges minor/patch/digest for the safe leaf-app domains independently of the shepherd.

The only genuinely non-auto set is now: one-way majors (author + human-confirm), emqx (held), Talos (not a Flux flow), and supporting-edit PRs (diff-scope security gate — except the one typed pattern it admits; see scripts/diff-scope-test.sh). Everything reversible auto-merges.

Kill switch: kubectl -n upgrade-agent patch cronjob upgrade-shepherd -p '{"spec":{"suspend":true}}'.

Holds registry — reasoned release blacklist

.renovate/holds.json5 is a durable, reasoned blacklist of releases that must not be applied, with the WHY recorded next to the enforcement. It exists because a held upgrade otherwise churns (Renovate re-opens the PR every run) and the reason ends up living only in a human's head or a memory file — so the next person (or agent) re-investigates it from scratch.

Each hold is a packageRule that does two things:

  • Enforces via allowedVersions — a semver range that excludes the bad version(s), auto-resuming at the fixed release where one is known.
  • Documents via a structured description array (summary / Reason: / Issue: / Resume: / Recorded:), which Renovate surfaces in logs/PRs and the Tier-4 shepherd greps before working a PR.

The file extends last in renovate.json5, so a hold is the final word for its package even over an auto-merge allowlist.

Two enforcement shapes:

  • Auto-resume when the fix is a version of the same package: allowedVersions: '<2.3.2 || >=3.0.0' skips 2.3.2–2.x and re-opens at 3.0.0.
  • Manual lift when the resume condition is something else (e.g. "after a different component is upgraded"): a plain upper bound ('<6.2.1') that a human or the agent widens once the condition is met.

Add a hold: append a packageRule per the in-file convention, set the tightest allowedVersions, run npx --yes --package renovate renovate-config-validator .renovate/holds.json5, commit. The Tier-4 shepherd does this automatically when it hits a blocker. Lift a hold: delete the rule (or widen allowedVersions) in a commit referencing the upstream fix; Renovate re-proposes. Seeded 2026-06-30 with the two EMQX holds (operator + broker, emqx/emqx#17600).

Decisions made

  • Tier 1 starts with GITHUB_TOKEN, swap to a GitHub App after flux-local is proven. App registration is the second step, not the first.
  • Renovate config will be split into .renovate/*.json5 files (mirror onedr0p / bjw-s) as a pure refactor — but in a follow-up PR after Tier 1 merges, not in the Tier 1 PR itself. Reason: Renovate's extends resolves referenced files from the default branch of the repo, so adding both the new files and the extends pointing at them in the same PR breaks Renovate until merge. Phase order: Tier 1 PR → merge → Phase 1.5 split PR → merge → Tier 2.
  • Edge cluster goes in the flux-local matrix from day one, even while it's powered off — validation is YAML-only.
  • Branch protection stays strict: false (2026-06-09, conscious trade-off). PRs don't have to be up-to-date with main to merge, so two PRs can each pass flux-local against an older base and merge in sequence with the combined state never rendered. For independent image bumps the risk is near zero, and the fix (strict: true + rebaseWhen: behind-base-branch) would serialize every merge behind a rebase→re-check cycle — a latency tax shaped suspiciously like the deadlock we just escaped. Revisit only if a merge-order incident actually happens.
  • Auto-merges deploy unsupervised overnight — by design (2026-06-09). The schedule gates PR creation; GitHub platform auto-merge fires whenever the required check goes green, and Flux applies within the hour — typically while we're asleep. Acceptable for the Tier 2 allowlist precisely because it's stateless single-pod apps; Alertmanager is the safety net (hence the alert-noise cleanup being a Tier 4 precondition). automergeSchedule exists if this ever needs to change.
  • Tier 3 uses symmetric dashboard-approval (2026-06-09): every HA-pod group member approval-gated, promotion to automation is all-or-none. The asymmetric two-layer design is unimplementable — see Tier 3.

Open questions

  • Renovate runtime — urgency downgraded 2026-06-09: platformAutomerge decoupled merging from Mend's hosted run cadence (GitHub merges the moment the check passes), which was the main pressure to self-host. The remaining motives are logs/control only. Stay on the hosted app; revisit only if its limits actually bite. (Options if they do: GitHub Actions self-hosted — onedr0p pattern; in-cluster renovate-operator — bjw-s pattern.)
  • Document the rollback procedure for each Tier 4 component in docs/observability/ so the agent (and a human at 2am) has a runbook. Do this as part of building phase 4a — the agent's page should link the runbook for the failing component.

Changelog

  • 2026-07-09 — cert-manager added to the ramp (PATCH only). Sixth revertible cluster-infra component. cert-manager minors are deliberately NOT auto-merged: the chart's Helm schema is additionalProperties:false and minors remove values keys (v1.21.0 dropped prometheus.servicemonitor.*), which hard-fails an upgrade if a stale key lingers — so minors get release-note + values vetting and a human-review PR. Also handled the stranded #2007 by hand (v1.20.3→v1.21.0 minor): vetted the three v1.21 breaking changes (none applied — we're already on v1.20.3 for the RBAC one, don't use the token-RBAC pattern, and set none of the removed metrics values), merged, rolled clean (all 5 pods Running, both ClusterIssuers Ready). Also fixed alert-responder noise (PR #2010): it now only pages when its verdict is action-needed AND the alert is still firing, and denylists the appdaemon-owned Protect.* class (was double-paging a self-healed UniFi Protect freeze + a documented ceph-mgr OOM cycle). GHA majors now auto-merge too (revertible + fail-safe: a breaking major reddens its own required check and can't merge).

  • 2026-07-08 — Cluster-breaker ramp: revertible-vs-one-way, + cadence. Reframed the floor from "cluster-breaker vs not" to "git-revertible vs one-way" (a supervised session merges these with the same steps the shepherd has). Widened the ramp to the revertible cluster-infra — rook-ceph (incl. ceph-csi), cnpg operator, dragonfly, cilium, authentik — auto-merging PATCH/safe-minor behind the backup+Ceph-health gate with pin-integrity guards (cephVersion/PG-imageName/authentik-YYYY.M unchanged); rook moves operator→csi→cluster as a unit. One-way majors (Ceph daemon major, PG major, cilium eBPF, authentik major, Talos) are authored as one-click ONE-WAY — human confirm PRs — the human is the break-glass contact for irreversible changes only. Cadence: once-daily → every 4h (quiet runs are $0 via the pre-filter) + a DRAIN rule (all class-1 stateless pure bumps per run; one stateful/infra unit per run, verify between). Fixes the pile-up: PRs no longer wait up to 24h, and the one-per-run cap no longer strands a backlog for days. Disposition labeler + ramp map/prompt updated in lockstep (consistency check green). First live rook auto-merge proven supervised before trusting the cadence.

  • 2026-07-06 — Agent-ops batch (PRs #1980/#1981/#1983, all live + drilled). (1) Vet-once markers: the shepherd records each vet as a shepherd-vet sha=<head> PR comment; the pre-filter skips already-vetted head SHAs (rebase re-arms) — kills the ~$0.5–0.8/day re-vet while a PR bakes. (2) Durable verdicts: the run's final summary line flows run-shepherd → triage (note on the coordination state) → the gate's page body — BREAK-GLASS reasons now reach the phone. (3) Single ramp definition: UPGRADE_AGENT_RAMP + an in-script path map replace the two-place glob/prompt lockstep; drift = fail-closed run + shepherd/ramp-mismatch page (drilled live, incl. dedupe). (4) Orphan-PR digest: scheduled runs write aging-PR reports; the gate pages weekly. (5) Shared coordination lib single-sourced as the upgrade-coordination-lib CM (byte-identical-block convention retired). (6) Remediate → opus (UPGRADE_AGENT_REMEDIATE_MODEL). (7) alert-responder (Track B1, runbook): in-cluster read-only Claude Code diagnosis of ANY critical alert, paged as a priority-0 follow-up — enrich-don't-act; E2E drill proved pickup→claim→diagnosis→page at $0.18 + at-most-once on re-run. (8) diff-scope typed pattern 1: the app-template automount restore is auto-mergeable when the base HR already declares serviceAccount:; suite 26/26. All four cronjobs live; every new page path was proven able to FIRE (the 07-03 lesson).

  • 2026-07-04 — Phase D cost control: deterministic pre-filter (quiet day = $0). The first unattended 09:00 ET run behaved correctly (surveyed 11 PRs, none in the ramp, auto-merged nothing) but cost ~$0.77 to conclude "nothing to do" — ~$23/mo of no-op LLM surveys eating the $50 cap. Added prefilter_should_run to run-shepherd.sh: a scheduled auto run now lists open Renovate PRs and matches their changed files against the ramp path prefixes (UPGRADE_AGENT_PREFILTER_GLOBS, new HR env) before cloning or calling the LLM — no in-scope PR ⇒ exit $0, no clone, no spend. It never decides mergeability (the LLM still fully vets every in-scope PR) and fails open (runs the LLM) on any gh/jq error. The ramp is now two lockstep env vars (prompt names + filter globs) — widen both together. Matcher proven both ways against the live queue (real ramp → skip; probe glob at #1921's path → summon). Targeted manual summons must clear the globs var (documented in the summon recipe).

  • 2026-07-03 — Tier-4 Phase D live: scheduled hands-off auto-merge (clean tier). Proved the chain supervised first: shepherd dry-run E2E ($0.66, correct triage of 12 open PRs), then a targeted MODE=auto summon vetted flux v2.8.8→v2.9.0 (#1917) (release notes, playbook, holds; confirmed the v1beta2 CRD removals don't apply) and auto-merged it as haynes-ops-bot through the required Flux Local + Diff Scope checks; negative proof: an out-of-scope stateful PR (cnpg #1910) was declined, no auto-merge armed. Then flipped the upgrade-shepherd CronJob live via git: daily 09:00 ET, MODE=auto, scope pinned by prompt to the clean-tier ramp set (see "Phase-D auto-merge ramp"). Spend guard verified recording ($1.84/$50 MTD). One backlog artifact handled: PRs opened before Phase B lack the now-required Diff Scope - Success on their head SHA — close/reopen re-fires it (new PRs get it automatically). Triage auto-summon (upgrade-shepherd-triage) enabled on */30 after validating both paths with one-off jobs: healthy path exits pre-LLM at $0 (recent merges + green cluster), and an induced benign regression (throwaway-ns ImagePullBackOff persisted >10m right after a merge) summons MODE=remediate. During the flux upgrade a REAL wedge surfaced — the cluster's image.toolkit CRDs still stored v1beta2 (etcd bookkeeping, invisible to repo-side vetting) — the flux ks failed Ready fail-safe (nothing applied), remediated with the official flux migrate (operator-approved break-glass); pre-flight check added to the flux playbook.

The induced-regression drill caught two REAL monitoring bugs (both fixed in the same PR, both of the "silent false-negative" class): (1) the persisted-pods PromQL in gate.sh/triage.sh used (expr) offset 10m — a parse error; Prometheus 400'd, curl -sf returned empty, and the check read as healthy forever (offset must follow a selector). (2) Worse: the agents' CiliumNetworkPolicy DNS allowlist (*.cluster.local) never matched multi-label service FQDNs (Cilium * doesn't cross dots), so the live health gate's Prometheus (pods/ESO/alerts/Ceph) and HA checks had been silently blind since deploy — it was effectively a 2-check (Flux-only) gate. Fixed with exact matchName entries (keeps DNS-exfil closed) + "blind is NOT green" warnings whenever Prometheus/HA are unreachable or a query fails. Lesson (same class as the Phase-C result==error miss): a monitoring check must be proven able to FIRE, not just observed quiet — the drill exists for exactly this.

  • 2026-07-02 — Tier-4 Phase C: all three Kyverno policies now ENFORCE. After building the exception set to audit-clean (registry rewrite for normalized bare-docker.io + reg.kyverno.io/mirror.gcr.io/nvcr.io; RBAC exceptions for K8s built-ins by bootstrap label + infra roles/bindings by name; PSS exceptions for privileged infra + VolSync movers by the app.kubernetes.io/created-by: volsync label), flipped failureAction: Audit → Enforce one at a time (registry → rbac → pod-security-baseline), each verified with a deliberate-violation --dry-run=server (evil registry / new cluster-admin binding / wildcard role / privileged pod all DENIED; legit + excepted admitted) and a clean reconcile. Kyverno enforcement isolated from native PSA by testing in the ai namespace (PSA=privileged). The VolSync mover exception was verified against a live mover via a controlled probe sync. Caught + fixed a latent bug: the RBAC policy errored (fail-open = admit) on null-rules roles (not_null(request.object.rules, [])). fail-open (failurePolicy: Ignore) preserved throughout; kube-system/flux-system never blanket-excluded; kustomize-controller never excluded. Phase C hard-prereq for Phase D (auto-merge) is now met (both diff-scope-required and Kyverno-enforce).

  • 2026-07-02 — Tier-4 Phase-1 audit + Phase B (push protection) live. Skeptical audit (tier4-audit-2026-07-02.md) with a 22-case diff-scope adversarial suite + live Kyverno/GitHub inspection. Fixed two holes (commit 2dfc71ed): diff-scope GATE B waved through version-shaped edits to config files (.renovate/holds.json5 hold-widening read as a "pure bump") — GATE A now blocks any changed path outside kubernetes/**; and the Kyverno reports-controller was audit-blind on RBAC (forbidden to list bindings) — added an aggregated read role. Phase B: Diff Scope - Success is now a required status check on main (alongside Flux Local - Success); the Main ruleset (14013135) gained a require-PR rule (required_approving_review_count: 0, admin RepoRole-5 bypass preserved); a new Edge ruleset (18431432) requires a PR + Diff Scope - Success on refs/heads/edge (edge had zero protection; flux-local does not run on edge PRs so it is deliberately not required there). Proven E2E: the real bot token was rejected direct-pushing to main ("Changes must be made through a pull request" + "2 of 2 required status checks are expected") while its feature-branch push succeeded, and a green Renovate PR stays MERGEABLE/CLEAN with no review deadlock. No auto-merge yet — that's Phase D, gated on Phase C (Kyverno enforce).

  • 2026-06-30 — Tier 4 build started: haynes-ops-bot GitHub App + holds registry. Registered the least-privilege haynes-ops-bot App (Contents R/W, Pull requests R/W, Checks read; single-repo install; webhook off) as the unattended shepherd's push/PR/merge identity — click-by-click setup + the adversarially-verified rationale (minimal perms, no branch-protection change, no flux-local.yaml change, 1h token mint) in tier4-bot-setup.md; a credential-source-agnostic mint helper at scripts/github-app-token.sh. Added the holds registry (.renovate/holds.json5, see Holds registry) — a reasoned release blacklist Renovate enforces and the agent reads — seeded with the two EMQX holds (operator 2.3.2 → auto-resume at 3.0.0; broker 6.2.1 → manual lift after the operator reaches 3.0.0). Those PRs stop churning and the WHY (emqx/emqx#17600) now lives next to the enforcement, not just in memory. Also authored the three Tier-4 runbooks that encode the agent — upgrade-shepherd.md (the loop + 3 modes), upgrade-health-gate.md (cross-cutting post-reconcile checks), and tier4-component-playbooks.md (per-component supporting edits, health checks, and adversarially-verified rollback for all 12 manual-tier components) — built from a multi-agent spec-then-adversarial-rollback-verification pass. Remaining for Tier 4: the gate runtime + invocable shepherd agent.

  • 2026-06-28 — Digest updates now auto-merge in the path-based Tier 2 rules. Added 'digest' to matchUpdateTypes on both matchFileNames rules (leaf-app domains + curated infra leaves) in .renovate/autoMerge.json5. They were ['minor', 'patch']-only, so every SHA-pinned image refresh in an allowlisted domain stranded forever — cloudflare-ddns #1884 and postgres-init #1889/#1890 sat unmerged purely for this. A digest bump on the same tag is at least as safe as minor/patch; the Immich carve-out (immich-app/*, package-name match, last rule wins per-dep) still blocks immich digests. Found while clearing a ~24-PR manual backlog in risk-tiered batches (safe leaves → infra leaves → breaking one-at-a-time: prom-operator-crds v30 → kube-prometheus-stack v87, traefik v41, cilium v1.19.5, zwave-js-ui, gh-action majors → rook v1.20.1 storage last; all verified). EMQX held (#1859 broker, #1860 operator): the 6.2.1 single-node boot crash is a maintainer-confirmed emqx-operator blue-green bug (not the broker); fix is operator 3.0.0, still pre-release — hold both until it GAs, then operator → broker.

  • 2026-06-15 — Tier 2 allowlist expansion + Tier 4 agent scope codified. Added observability/** as a whole leaf domain and a curated cluster-infra-leaves rule (kube-system/{metrics-server,reloader,reflector, k8tz,spegel}, network/cloudflare-ddns) matched by explicit subpath — see the Tier 2 "Allowlist expansion 2026-06-15" note for the why (and why kube-system/network are not safe whole-domain). Per-package trust clocks for the new entries start today. Codified the Tier 4 upgrade- shepherd agent's three invocation modes (scheduled health gate / summoned on-failure remediation / breaking-change shepherd) — the "summon on failure" and "read release notes + make supporting edits" capabilities now have a home in the Tier 4 section. Cleared the standing 17-PR manual backlog in risk-tiered batches (home-automation first under supervision → safe leaves → shared restic → network infra → database operators).

  • 2026-06-09 — Tier 3 design locked + implemented (manual-approval phase) in .renovate/groups.json5: scrapped the unimplementable two-layer sweep design for the symmetric-approval HA-pod group + a babysat Z2M rule (see Tier 3 for the post-mortem and the all-or-none promotion rule). Tier 4 decisions locked: phase 4a = Claude Code scheduled agent with the read-only Omni service-account kubeconfig (watch + page, rollback stays human); phase 4b = in-cluster CronJob + haynes-ops-bot GitHub App for automated rollback. Owned the Tier 2 path-based-domains deviation (+ standing carve-out rule for new apps); restarted the Tier 2 trust clock at 2026-06-08; recorded the strict: false and overnight-merge trade-offs. Fixed the restic duplicate-PR pair (#1830/#1831 — # renovate: hints made the regex manager double-track images the kubernetes manager already covers; hints removed). Cleared the 9-PR manual backlog in risk batches (leaf/infra → restic → HA pod as one unit → traefik last, verified end-to-end).
  • 2026-06-08 — Fixed the auto-merge deadlock: Tier 0/2 rules used branch-mode automerge, but flux-local runs on: pull_request only, so no branch ever went green and every eligible PR sat unmerged (e.g. #1816). Flipped to automergeType: 'pr' + platformAutomerge: true — GitHub merges on check-pass, independent of Mend's run cadence. First confirmed live auto-merge: #1829 (2026-06-09).
  • 2026-06-04 — Tiers 1 + 2 landed. Tier 1 (flux-local PR gate) was already live. Split config into .renovate/*.json5 (the Phase 1.5 refactor, superseding the stale PR #1673), fixed the ignorePaths .archive glob (Renovate had been scanning archived apps), flipped the github-actions auto-merge to ignoreTests: false now that the gate exists, and added the Tier 2 curated-allowlist auto-merge (own images + kube-prometheus-stack + safe leaf-app domains, minor/patch, minimumReleaseAge bake, immich-app/* carved out). Decision: start with the curated allowlist and ramp.
  • 2026-04-14 — Roadmap created. No config changes yet — Tier 1 (flux-local) is the precondition for the auto-merge tiers and lands first.