Repository navigation
Conversation
CD only ran helm upgrade for the app, AG, staff and observability charts. bt-base was pushed but never upgraded, and Mongo and Redis were installed once by hand from infra/init.sh, so the cluster drifted from the repo: Redis runs with no maxmemory and the noeviction policy although infra/redis/values.yaml sets 140mb and volatile-lru. Add "CD: Infra Releases": - pull requests touching these charts show the diff against the cluster - a weekly run fails when a release no longer matches main - Run workflow diffs one release and, with apply and the environment's approval, packages the chart per commit and runs helm upgrade --wait - applying refuses images without a fixed tag or digest, and checks every image is on the node or pullable before anything restarts Releases and their values files live in infra/releases.json. Chart.lock files are regenerated (same versions) because helm dependency build rejected the stale digests. init.sh installs from the local charts with the same values files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
- Mongo prod: request 2 CPUs, limit 4. Its container had no CPU request, so it got the smallest share on the node whenever the node was busy. - Mongo staging: request 100m, limit 500m, so staging cannot take CPU from production. - Mongo readiness probe timeout 5s -> 10s; it timed out during node stalls. - Redis prod: 256Mi/512Mi memory, 100m/500m CPU, maxmemory 350mb. The chart's nano preset (192Mi) got Redis OOM-killed on Oct 22. - Redis staging values file with smaller limits. - image.digest keys for Mongo, its init container and Redis. They are empty: fill them with the digests running now before the first apply; the workflow refuses to apply while any image is `latest`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Staging resource limits move to their own PR so this one only changes production. Staging Mongo nulls out the prod CPU settings it would otherwise inherit and keeps memory only, as deployed today. Staging Redis keeps its env label and inherits the prod resources until then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
The first PR run failed two ways: GitHub releases returned 504 for the
helm-diff plugin download, and the HELM_DIFF_KUBECONFIG secret was
rejected by the API server ("the server has asked for the client to
provide credentials"), so its token has expired or been revoked.
- retry the plugin install up to three times
- check the kubeconfig can read Helm releases before diffing, and fail
with instructions when it cannot
- document how to regenerate the secret for helm-diff-sa
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
- Three of five diff jobs failed because every attempt to download the helm-diff release from GitHub returned 504. Fetch the archive with curl retries and unpack it into the plugin directory instead of relying on `helm plugin install`, which gives up after one try. - The production diff showed the migrations ConfigMap going backwards: infra/mongo/migrations/seed-requirements.js (generated Apr 2) was older than migrations/seed-requirements.js and the deployed copy (May 6), and missed #1263. Copy the current file so applying does not downgrade it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Production: the digests bt-prod-mongo (MongoDB 8.3.4, os-shell) and bt-prod-redis ran on 2026-11-08, so applying keeps the same images. Staging Mongo still runs docker.io/bitnamilegacy/mongodb:7.0.5 from an older chart. Inheriting the production pin would jump it from 7.0 to 8.3, which MongoDB does not support, so staging pins its current tags until it is upgraded on purpose. Staging Redis keeps no data and inherits the production pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Even with ten curl retries over 100 seconds, the helm-diff release download from GitHub kept returning 504 on some runners. Compare `helm get manifest` (what is deployed) with `helm template` (what the repository would deploy) using diff instead, in infra/scripts/release-diff.sh. It needs only read access to the release's Helm secrets, which helm-diff-sa already has, and leaves Kubernetes Secrets out of the output. Exit codes match the old --detailed-exitcode use: 0 no changes, 2 changes, 1 error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
|
@sashakmurray, some redis and helm stuff |
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
| code=0 | ||
| values_file="" | ||
| if [ -n "$VALUES" ]; then values_file="infra/$CHART/$VALUES"; fi | ||
| infra/scripts/release-diff.sh "$RELEASE" "infra/$CHART" $values_file > diff.txt 2>&1 || code=$? |
There was a problem hiding this comment.
the release-diff script uses version numbers from Chart.yaml (0.1.0) whereas the later apply step overrides the version with 1.0.0-g<commit>. I think we need to package the chart with the deployment version before both diffing and applying so that the drift check is accurate
There was a problem hiding this comment.
Agreed, but one thing to note: that version contains the commit SHA, so it changes on every commit, not only when the chart changes.
For example:
- Mon: apply from
abc1234→ cluster labelshelm.sh/chart: bt-base-1.0.0-gabc1234 - Tue: an unrelated commit lands,
mainis nowdef5678 - Next Mon: the weekly diff packages
1.0.0-gdef5678→ the label differs → "drift", although nothing in the chart changed
So the weekly check would fail whenever main moved since the last apply. Two ways to fix it:
- Keep the single packaging step, but derive the version from the last commit that touched the chart (
1.0.0-g$(git log -1 --format=%h -- infra/<chart>)), so it only changes when the chart does. The packaging step needsfetch-depth: 0(the checkout is a shallow sparse checkout). Until the first apply, the diff would still show label-only changes, because the cluster has1.0.0frominit.sh. - Ignore
helm.sh/chartinrelease-diff.sh, next to where Secrets are already dropped. A few lines, and the diff no longer depends on the version at all.
2 seems to be of less headache, though. lemme know what you think
| # | ||
| # The diff runs from GitHub with the HELM_DIFF_KUBECONFIG secret: a kubeconfig | ||
| # for the read-only helm-diff-sa service account (infra/base/templates/helm-diff-role.yaml). | ||
| # To regenerate it, on the server: |
There was a problem hiding this comment.
could you add info about this process in the docs rather than in the comments of this file?
…eneration runbooks.md: - Deploying Infra Releases: how to read a diff, apply a release and roll back - Regenerating the HELM_DIFF_KUBECONFIG Secret, so the steps live in the docs instead of a workflow comment - Note in the /helm-diff section pointing to the new workflow cicd-workflow.md: - Infra Releases: what the workflow deploys and why it diffs before it applies Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi6srcGkgAJ2SDUhxt6
The steps for regenerating the HELM_DIFF_KUBECONFIG secret now live in the docs (runbooks.md). The workflow header comment only links to them, and the "Check Cluster Access" error points there instead of to the top of this file. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi6srcGkgAJ2SDUhxt6
Problem
CD only runs
helm upgradefor the app, AG, staff and observability charts. The other infrastructure charts never deploy from the repo:bt-base(backup CronJob, MetalLB, cert issuer) is pushed to the registry but never upgraded.infra/init.sh.For example: #1154 and #1135 are merged but never deployed on hozer-51.
So the cluster has drifted from the repo, and it has caused outages:
maxmemoryand uses thenoevictionpolicy, thoughinfra/redis/values.yamlsets 140mb andvolatile-lru. With the chart's "nano" preset (192Mi limit), Redis grows until it's killed. It was OOM-killed on Oct 22 (OOMKilled, exit 137), which logged every user out. It's at about 168 MiB of 192 MiB now.Before:

What I have runned this time:
mongodbcontainer has no CPU request, so it gets the smallest CPU share on the node whenever the node is busy (about 0.1 core, inherited from the init container).--copy-props nonewhen promoting the 5 AM backup into the daily and weekly folders) never reached the cluster.latesttag. Any fresh pull can change the version.bitnamilegacy) on an older chart.Change
New workflow
CD: Infra Releases(.github/workflows/cd-infra.yaml), with the release list ininfra/releases.json:infra/base,infra/mongoorinfra/redis: the diff against what's deployed goes to the job log and summary.helm get manifestcompared withhelm template(infra/scripts/release-diff.sh), with Kubernetes Secrets left out of the output. No helm-diff plugin: its GitHub release download returned 504 for minutes at a time, even with 10 retries.HELM_DIFF_KUBECONFIG, a read-onlyhelm-diff-sakubeconfig. That secret had expired; it was regenerated on 2026-11-08, and the steps are documented at the top of the workflow.main, and fails if the cluster no longer matches the repo.applychecked and the environment's approval (productionorstaging), it:1.0.0-g<sha>) and pushes it to the registry;helm upgrade --wait. On failure it prints the pods and the rollback command.infra/scripts/check-images.shrejects any image without a fixed tag or digest.Values:
os-shell)maxmemory 350mb,volatile-lru. Pinned to the running digestbitnamilegacy/mongodb:7.0.5,os-shell:11-debian-11-r96), so it doesn't inherit production's 8.3.4 pin. MongoDB doesn't support skipping a major versionStaging limits are in #1313, stacked on this PR.
Other:
infra/mongo/migrations/seed-requirements.jssynced frommigrations/. The repo copy (generated Apr 2) was older than what's deployed (May 6) and missed [BER-74] Fix CS 61 series incorrectly satisfying GradTrak HSDDT requirement #1263.Chart.lockfor Mongo and Redis regenerated (same chart versions).infra/init.shinstalls from the local charts with the same values files.What the diffs against the cluster show
All five diff jobs pass on the latest commit.
bt-prod-mongomongodbcontainer and thelog-dirinit container. Image moves from:latestto the same image pinned by digest. Migrations ConfigMap updatedbt-prod-redismaxmemory 350mb/volatile-lru/save "". Memory 192Mi → 512Mi limit, CPU 150m → 500m. Image moves from:latestto the same image pinned by digestbt-stage-mongobt-stage-redissha256:5179ef…to production'ssha256:341edafd…bt-base--copy-props none(the R2 fix). Chart labelbt-base-1.0.0→0.1.0on the MetalLB pool, L2 advertisement and issuer (label only)Testing
maxmemory, labels andhostPathland where intended.release-diff.shtested locally against a fake deployed manifest:bash -n. actionlint was not run.Before applying
bt-stage-mongoandbt-stage-redisfirst and watch staging Mongo start on the new chart.helm rollback bt-stage-mongo -n btundoes it.bt-baseany time: it only changes the backup CronJob and labels.SSH_USERNAMEcan runcrictl. It needs root.🤖 Generated with Claude Code
https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6