Skip to content

Tier-4 hands-off upgrades — PART 1 hardening audit (2026-07-02)

Independent, skeptical audit of everything built so far (bot / holds / gate / shepherd / Kyverno Phase A / diff-scope), per HANDOFF-2026-07-01-tier4-hands-off.md PART 1. Method was empirical over reasoning: 22 crafted malicious diffs run through diff-scope.sh, live Kyverno report/RBAC/webhook inspection on the running cluster, and live GitHub ruleset/branch-protection/repo-settings queries.

Bottom line: the architecture is sound and the two primary controls do what they claim — with two real holes I found and fixed, and a set of Phase-C landmines that would have broken workloads on a blind enforce. Details, ranked by severity.

Fixes already landed (commit 2dfc71ed, verified live)

F1 — [HIGH] diff-scope waved through version-shaped edits to config files

The PRIMARY gate had a bypass. diff-scope.sh GATE B proves a diff is a "pure version bump" by masking version tokens (1.40.0→VER, sha256:…→DIGEST) and requiring the removed-line multiset to equal the added-line multiset. GATE A's path-denylist covered the Kyverno tree, rbac-named files, scripts/diff-scope.sh, and .github/**-non-workflow — but not .renovate/** or arbitrary root files.

Because GATE B masks any version token on any line, an in-place edit to a config file that only changes a version range slips through as a "pure bump":

# proven before the fix — exit 0 (auto-mergeable):
-      allowedVersions: '<6.2.1'
+      allowedVersions: '<99.0.0'      # .renovate/holds.json5 — widens/lifts a hold

Failure scenario: a prompt-injected shepherd (it reads attacker-influenceable release notes) in the future auto-merge mode edits .renovate/holds.json5 to lift the EMQX hold (or any hold), diff-scope reports "pure version bump → safe to auto-merge", and the known-broken upgrade it was protecting against flows in unattended. Same shape lets it edit any version-pinned value in a non-manifest file.

Fix: GATE A now blocks any changed path (add/modify/delete) outside kubernetes/**. Only image-tag/digest + chart-version bumps to kubernetes/** manifests are auto-mergeable; .renovate/**, .github/**, scripts, and root files fall to human review. diff-scope is author-scoped to the bot, so this never gates Renovate/human PRs, and the bot has no legitimate auto-merge edit outside kubernetes/** (the holds-protocol PR is meant to be human-reviewed).

Proof: 22-case adversarial suite — the widen-hold case now BLOCKs, .renovate/, .github/, and root-file edits all BLOCK, and a legit kubernetes/** tag bump still passes SAFE. 18/19 core cases pass; the 1 "miss" is a benign false-positive (adding a digest pin to a previously-unpinned tag → human review), which the shepherd never does.

F2 — [MEDIUM] restrict-rbac-escalation was audit-blind

The restrict-rbac-escalation policy matches ClusterRoleBinding/ClusterRole/ RoleBinding/Role, but the Kyverno chart's reports-controller role lacks read on rbac.authorization.k8s.io. Live logs: clusterrolebindings … is forbidden … cannot list + failed to start watcher. Result: zero PolicyReports for RBAC despite 3 cluster-admin bindings existing on the cluster — the policy's audit was silently empty.

Important: the admission (enforce) path is unaffected — the resource webhook already intercepts CRB/CR/RB/Role, so enforce will deny a bad binding. This was an audit-visibility gap, not an enforcement gap. But it hid exactly the data Phase C needs (the violator set) and any future RBAC drift.

Fix: added an aggregated ClusterRole (kyverno:reports-controller:rbac-read, label rbac.kyverno.io/aggregate-to-reports-controller: "true") granting get/list/watch on those four kinds. Proof: after reconcile the aggregate now lists the RBAC resources, the forbidden errors are gone, and restrict-rbac-escalation results began populating (was 0).

Open findings — surfaced for your call (not yet actioned)

O1 — [Phase C blocker] restrict-image-registries: 101 audit fails are all legit

The 101 "fail"s are not attackers — they're real images whose string doesn't match the host pattern: - reg.kyverno.io/* (Kyverno's own images), mirror.gcr.io/* — not in the allowlist. - bare/implicit-docker.io images — busybox:latest, node:lts-alpine. Kyverno's pattern match is literal; busybox does not start with docker.io/, so it fails even though it is a docker.io image.

Consequence: flipping this policy to Enforce as-is would deny those workloads at apply. Two options for Phase C, your call: - (a) Fix the allowlist — add reg.kyverno.io, mirror.gcr.io, and handle bare-image normalization (Kyverno can't auto-prefix docker.io in pattern; the clean fix is a small mutate-normalize or switching to an image-name CEL check). - (b) De-prioritize it. Be honest about its value: a host-level allowlist of ghcr.io/* | docker.io/* | quay.io/* barely constrains an attacker (anyone can push to those hosts). The org-swap threat (ghcr.io/home-operations/x → ghcr.io/attacker/x) is caught by diff-scope, not this policy. Its real residual value is only blocking a brand-new registry host. I lean (a)-lite: allow the two missing legit registries and enforce, but treat it as the weakest of the three policies and rely on diff-scope + (recommended) cosign for the actual image trust.

O2 — [Phase C] rbac-escalation enforce needs 2 name-scoped exceptions or it wedges Flux

Enforce fires on CREATE/UPDATE. Two existing privileged bindings are re-applied by Flux and would be denied on the next reconcile: - cluster-reconciler-flux-system → cluster-admin, managed by kustomize-controller (the Flux bootstrap binding). Deny → the flux Kustomization wedges → GitOps stops. - headlamp-admin → cluster-admin, managed by helm-controller (chart RBAC).

Fix in Phase C: PolicyExceptions by resource name (cluster-reconciler-flux-system, headlamp-admin) — not a requester/subject exclude (invalid under background:true) and not excluding kustomize-controller (that's the apply path a malicious PR takes; must stay covered). 0 namespaced RoleBindings to admin/edit/cluster-admin exist, so no RB exceptions needed.

O3 — [Phase C] pod-security-baseline: 361 fails = the exception set to build

Expected Phase-A output. Big buckets: rook-ceph (251), kube-system (device-plugins, coredns), observability, ai/comfyui, plus invisible subchart pods. Phase C converts these into per-workload PolicyExceptions — critically the VolSync privileged movers by their label (volsync.backube/privileged-movers: "true") across ~18 namespaces, not a name glob. This is enumeration work, not a decision.

O4 — [your decision] Anthropic spend cap

The shepherd run caps at --max-budget-usd 5.00 / --max-turns 40 per run, but there's no hard monthly ceiling on the Anthropic key. For an unattended auto-summon loop (Phase D), recommend setting a console-level monthly spend limit as a backstop against a runaway summon loop. Your call on the number.

O5 — [recommend, not required] cosign image provenance for ghcr.io/thaynes43/*

The one gap neither diff-scope nor the registry policy closes: a pure tag bump to a malicious image on an already-allowed repo path. For the highest-trust/highest-risk set — our self-built images (upgrade-agent, upgrade-shepherd, appdaemon) — Kyverno verifyImages with keyless (GitHub OIDC) cosign attestation would prove the image was built by our CI, not swapped. Renovate minimumReleaseAge is the current partial mitigation. Recommend for Phase C+ once enforce is stable.

Checklist items verified OK (no change needed)

  • Author gate is sound. github.event.pull_request.user.login is the immutable PR creator; only the bot's installation token can author as haynes-ops-bot[bot]. A human can't spoof it and the bot can't author as someone else.
  • Auto-merge cannot bypass a red check even without --auto. The bot is non-admin, non-bypass. An immediate gh pr merge (no --auto) fails on a pending/red required check; gh pr merge --admin fails (not admin). --auto is a robustness nicety (queue-then-merge), not the safety boundary — the required check + non-admin token is. So the allowlist not being able to require --auto (prefix glob) is acceptable.
  • allow_auto_merge: true at the repo — Phase D --auto and Renovate platformAutomerge both work.
  • Admin bypass = RepositoryRole id 5, already the lone bypass_actors entry on the Main ruleset (14013135). The bot is not in it. Matches the Phase-B plan.
  • The gate is genuinely read-only. ClusterRole is get/list/watch only — no secrets, no pods/exec, no pods/log, no write verb anywhere. It pages Pushover through its own egress CNP (api.pushover.net), independent of the Alertmanager it monitors.
  • PEM isolation holds. Two separate ExternalSecrets; the PEM is only in the init-container env; run-shepherd.sh asserts GITHUB_BOT_APP_PRIVATE_KEY is unset in the LLM container and refuses to run otherwise. Token is 1h TTL, single-repo, contents+PR:write + checks:read.
  • Kyverno admission webhook covers RBAC + workloads (clusterrolebindings, clusterroles, rolebindings, roles, pods, deployments, …) with failurePolicy: Ignore (fail-open, as designed — never wedge Flux). PolicyExceptions are locked to the kyverno namespace (GHSA-qjvc-p88j-j9rm), so the shepherd can't commit a self-exempting exception elsewhere.
  • diff-scope GATE B is robust against injection. Multiset-equality means you cannot add a net-new line without removing a line that masks identically; malicious tokens (command:, hostNetwork:, privileged:, a new sidecar, an env var) never mask to a version token, so they always break equality → BLOCK. Mid-line injection (tag: 1.40.1 hostNetwork: true) also breaks equality. Org-swap on the same host, new ClusterRoleBinding, new Deployment file, serviceAccountName, privileged: true, replicas: 3 — all BLOCK.

Known gaps that are by-design, not bugs

  • Edge cluster has no branch protection and flux-local does not run on edge PRs (flux-local.yaml is branches: [main]). diff-scope does run on edge. Phase B must add an edge ruleset (require-PR + require diff-scope; not flux-local yet — a skipped required check hangs forever). Tracked in the Phase-B plan.
  • Flux Local - Success passes on skipped jobs (a bot PR touching only non-kubernetes/** earns a trivially-green flux-local). This is why diff-scope must be a required check and why F1's path-tightening matters — together they close it.
  • Backdoored-image gap (O5) — a malicious pure-tag bump on an allowed path. Mitigated by minimumReleaseAge; cosign recommended.

Alternatives weighed

  • Kyverno vs native Validating Admission Policy (CEL, GA on k8s 1.35): stay on Kyverno. VAP saves a controller but loses PolicyReports (the audit-then-enforce workflow this whole plan is built on), PolicyExceptions, and autogen-for-controllers. Not worth a rewrite. Note VAP as a future option if Kyverno ever becomes a liability.
  • Cluster-wide default-deny egress: real value but a large, breakage-prone project. The two agents already have tight egress CNPs; a non-privileged exfil workload is a broader threat than this upgrade pipeline. Defer to the platform phase (below).
  • Canary auto-merge to edge first: edge is powered off, so it can't runtime-validate. The health-gate + one-at-a-time + Flux HR strategy: rollback is the canary here.
  • Branch-restricted bot (shepherd/* only): App tokens can't be branch-scoped; a ruleset could restrict branch creation, but push-protection already forces the PR path, so marginal. Optional.
  • diff-scope as Kyverno/Conftest policy instead of bash regex: the bash multiset approach tested out strong (22 cases). A policy engine on the rendered PR is more work for similar guarantees. Keep bash; revisit only if it grows unwieldy.

Scalability toward the "cluster bot" vision

The bigger question you raised: is this on the path to an in-cluster, conversational agent that can answer "why is Pushover firing", "why is this HA sensor broken", "add an automation per hass-sandbox rules" — one or more products, maybe its own repo?

What's already right and generalizes: - The deterministic-detector / summoned-LLM split is the correct backbone. A cheap always-on LLM-free monitor that pages, plus an expensive LLM summoned on demand, generalizes to any domain (networking, HA, storage). Keep this seam. - The containment primitives are domain-agnostic: read-only-SA-by-default + egress CNP + PEM/secret isolation via init-container + PR-path-only writes + per-run token + Kyverno admission backstop. A second agent reuses all of these.

What does NOT yet reach the vision (the honest gaps): 1. The shepherd is headless one-shot (claude -p … --output-format json), not conversational. "Ask it why X" needs a bidirectional, durable channel — the operator sends a message, the agent replies, back and forth. Today the only operator channel is one-way Pushover. This is the single biggest gap between "today's renovate goal" and "the cluster bot." Closing it means a chat bridge (Pushover replies / Telegram / Slack / this harness's remote-control channel) into a longer-lived agent session, not a CronJob Job. 2. diff-scope is upgrade-shaped and won't transfer. "Pure version bump" is meaningless for an HA-automation or networking agent that makes arbitrary edits. Treat diff-scope as one member of a family of per-capability scope gates — each new domain needs its own "what may this agent auto-apply vs. must a human review" gate, or it defaults to human-review. Don't try to reuse diff-scope for a different domain. 3. The upgrade-agent tree is copy-paste-shaped. A second agent today means duplicating the ns/SA/CNP/init-token/run-script scaffold. Before the second domain lands, factor the shared runtime (token mint, read-only SA + egress CNP templates, the "summon-a-Job-with-a-prompt" mechanism, a Pushover page+reply bridge) into a reusable Kustomize component (kubernetes/shared/components/agent-runtime/) or a dedicated repo, so each new agent is a thin manifest + prompt + tool-allowlist on top.

Separate repo — yes, but not yet. The agent runtime (image, run scripts, tool allowlists, prompt/runbook library, tests, semver tags) is a product with its own release cadence, distinct from the cluster's desired-state — exactly the hass-sandbox → appdaemon shape. Trigger to split: when the agent gains its second domain (beyond renovate upgrades) or its first conversational surface. Until then the shepherd image building out of haynes-ops is fine and not worth the churn.

Recommendation: finish 100%-hands-off renovate on the current design (it's close and the seams are right), and treat "conversational bridge + agent-runtime component" as the first milestone of the platform phase — that's the fork where a dedicated repo earns its keep.