Skip to content

feat(alerts): add etcd encryption key rotation alerts - #37

Open
defo89 wants to merge 1 commit into
cloudoperators:mainfrom
defo89:feat/etcd-encryption-rotation-alerts
Open

defo89 wants to merge 1 commit into
cloudoperators:mainfrom
defo89:feat/etcd-encryption-rotation-alerts

Conversation

@defo89

@defo89 defo89 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Add two PrometheusRule alerts for etcd encryption key rotation CronJob:

EtcdEncryptionRotationOverdue (warning)

  • Fires when CronJob has not succeeded for >120 days (90-day rotation + 30-day grace)
  • Detects: disabled CronJob, persistent failures, schedule misconfiguration

EtcdEncryptionRotationJobFailed (warning)

  • Fires when a rotation Job fails
  • Detects: RBAC issues, image pull errors, KCP patch failures, node readiness problems

Both alerts follow the same pattern as existing controlplane-backup.yaml alerts and use standard kube-state-metrics.

Copilot AI balanced review requested due to automatic review settings October 9, 2026 14:56
@defo89
defo89 requested a review from a team as a code owner October 9, 2026 14:56
@defo89
defo89 force-pushed the feat/etcd-encryption-rotation-alerts branch from d37e57c to ed018ce Compare October 9, 2026 14:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The alert expressions miss never-successful CronJobs and can misclassify retrying Jobs as failed.

2 open findings
What changed in this PR

Adds monitoring for overdue and failed etcd encryption key rotations.

Changes:

  • Adds overdue CronJob alert.
  • Adds failed rotation Job alert.
File Description
charts/​controlplane-operations/​alerts/​controlplane-encryption.yaml Defines both encryption rotation alerts.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +10 to +12
(
time() - kube_cronjob_status_last_successful_time{cronjob="etcd-kcp-backup-encryption-rotate"}
) > 120 * 24 * 3600
# state that needs investigation.
- alert: EtcdEncryptionRotationJobFailed
expr: >
kube_job_status_failed{job_name=~"etcd-kcp-backup-encryption-rotate-.*"} > 0
@defo89
defo89 force-pushed the feat/etcd-encryption-rotation-alerts branch from ed018ce to dd7540e Compare October 9, 2026 15:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants