Skip to content

Deploy bt-base, Mongo and Redis from the repository - #1312

Open
bfliao wants to merge 9 commits into
mainfrom
leo/infra-releases-cd
Open

bfliao wants to merge 9 commits into
mainfrom
leo/infra-releases-cd

Conversation

@bfliao

@bfliao bfliao commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem

CD only runs helm upgrade for the app, AG, staff and observability charts. The other infrastructure charts never deploy from the repo:

  • bt-base (backup CronJob, MetalLB, cert issuer) is pushed to the registry but never upgraded.
  • Mongo and Redis were installed once by hand from infra/init.sh.

For example: #1154 and #1135 are merged but never deployed on hozer-51.

So the cluster has drifted from the repo, and it has caused outages:

  • Redis: the live config has no maxmemory and uses the noeviction policy, though infra/redis/values.yaml sets 140mb and volatile-lru. With the chart's "nano" preset (192Mi limit), Redis grows until it's killed. It was OOM-killed on Oct 22 (OOMKilled, exit 137), which logged every user out. It's at about 168 MiB of 192 MiB now.

Before:
image
What I have runned this time:

kubectl exec -n bt bt-prod-redis-master-0 -- redis-cli CONFIG SET maxmemory-policy volatile-lru
kubectl exec -n bt bt-prod-redis-master-0 -- redis-cli CONFIG SET maxmemory 180mb
kubectl exec -n bt bt-prod-redis-master-0 -- redis-cli CONFIG GET maxmemory*
  • Mongo: the mongodb container has no CPU request, so it gets the smallest CPU share on the node whenever the node is busy (about 0.1 core, inherited from the init container).
  • Backups: the R2 fix from Skip tag copy on R2 server-side copies #1268 and Tempo metrics fix #1282 (--copy-props none when promoting the 5 AM backup into the daily and weekly folders) never reached the cluster.
  • Images use the latest tag. Any fresh pull can change the version.
  • Production and staging run different MongoDB versions: production 8.3.4 on the current chart, staging 7.0.5 (bitnamilegacy) on an older chart.

Change

New workflow CD: Infra Releases (.github/workflows/cd-infra.yaml), with the release list in infra/releases.json:

  • Pull requests touching infra/base, infra/mongo or infra/redis: the diff against what's deployed goes to the job log and summary.
    • The diff is helm get manifest compared with helm template (infra/scripts/release-diff.sh), with Kubernetes Secrets left out of the output. No helm-diff plugin: its GitHub release download returned 504 for minutes at a time, even with 10 retries.
    • It uses HELM_DIFF_KUBECONFIG, a read-only helm-diff-sa kubeconfig. That secret had expired; it was regenerated on 2026-11-08, and the steps are documented at the top of the workflow.
  • Weekly: diffs every release from main, and fails if the cluster no longer matches the repo.
  • Run workflow: diffs one release. With apply checked and the environment's approval (production or staging), it:
    • packages the chart with a per-commit version (1.0.0-g<sha>) and pushes it to the registry;
    • over SSH, checks every image is on the node or can be pulled, then runs helm upgrade --wait. On failure it prints the pods and the rollback command.
  • Unpinned images block the apply: infra/scripts/check-images.sh rejects any image without a fixed tag or digest.

Values:

Release Change
Mongo prod CPU request 2, limit 4. Pinned to the running digests (MongoDB 8.3.4, os-shell)
Redis prod Memory 256Mi/512Mi, CPU 100m/500m, maxmemory 350mb, volatile-lru. Pinned to the running digest
Mongo staging Pinned to what it runs today (bitnamilegacy/mongodb:7.0.5, os-shell:11-debian-11-r96), so it doesn't inherit production's 8.3.4 pin. MongoDB doesn't support skipping a major version
Redis staging Env label. Inherits the production resources and pin; it keeps no data on disk, so moving to production's Redis image is harmless
Mongo (both) Readiness probe timeout 10s. Production already has 10s, so this changes staging only, from 5s

Staging limits are in #1313, stacked on this PR.

Other:

What the diffs against the cluster show

All five diff jobs pass on the latest commit.

Release Changes an apply would make
bt-prod-mongo CPU request and limit on the mongodb container and the log-dir init container. Image moves from :latest to the same image pinned by digest. Migrations ConfigMap updated
bt-prod-redis Adds maxmemory 350mb / volatile-lru / save "". Memory 192Mi → 512Mi limit, CPU 150m → 500m. Image moves from :latest to the same image pinned by digest
bt-stage-mongo Chart upgrade from an older version: read-only root filesystem, a config file from a new ConfigMap, a new PodDisruptionBudget, readiness timeout 5s → 10s. The MongoDB version stays 7.0.5. Untested: whether the 18.6.0 chart works with the 7.0.5 image
bt-stage-redis Same config and resources as production. Image moves from sha256:5179ef… to production's sha256:341edafd…
bt-base Backup CronJob gains --copy-props none (the R2 fix). Chart label bt-base-1.0.0 → 0.1.0 on the MetalLB pool, L2 advertisement and issuer (label only)

Testing

  • I rendered all five releases with Helm 3.16.4. Every image is pinned by digest or a fixed tag (the check passes). The CPU and memory settings, maxmemory, labels and hostPath land where intended.
  • release-diff.sh tested locally against a fake deployed manifest:
    • no changes → exit 0;
    • old Redis config → exit 2, with the expected lines;
    • a Secret in the manifest → not printed;
    • a bad chart path → exit 1.
  • The diff job ran against the cluster for all five releases (results above).
  • The workflow YAML parses, and every embedded script passes bash -n. actionlint was not run.
  • The apply job has not run yet.

Before applying

  1. Apply bt-stage-mongo and bt-stage-redis first and watch staging Mongo start on the new chart. helm rollback bt-stage-mongo -n bt undoes it.
  2. Apply bt-base any time: it only changes the backup CronJob and labels.
  3. Apply production Mongo and Redis at a quiet hour: each restarts, and a Redis restart logs everyone out.
  4. Check SSH_USERNAME can run crictl. It needs root.

🤖 Generated with Claude Code

https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6

CD only ran helm upgrade for the app, AG, staff and observability charts.
bt-base was pushed but never upgraded, and Mongo and Redis were installed
once by hand from infra/init.sh, so the cluster drifted from the repo:
Redis runs with no maxmemory and the noeviction policy although
infra/redis/values.yaml sets 140mb and volatile-lru.

Add "CD: Infra Releases":
- pull requests touching these charts show the diff against the cluster
- a weekly run fails when a release no longer matches main
- Run workflow diffs one release and, with apply and the environment's
  approval, packages the chart per commit and runs helm upgrade --wait
- applying refuses images without a fixed tag or digest, and checks every
  image is on the node or pullable before anything restarts

Releases and their values files live in infra/releases.json. Chart.lock
files are regenerated (same versions) because helm dependency build
rejected the stale digests. init.sh installs from the local charts with
the same values files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
- Mongo prod: request 2 CPUs, limit 4. Its container had no CPU request,
  so it got the smallest share on the node whenever the node was busy.
- Mongo staging: request 100m, limit 500m, so staging cannot take CPU
  from production.
- Mongo readiness probe timeout 5s -> 10s; it timed out during node stalls.
- Redis prod: 256Mi/512Mi memory, 100m/500m CPU, maxmemory 350mb. The
  chart's nano preset (192Mi) got Redis OOM-killed on Oct 22.
- Redis staging values file with smaller limits.
- image.digest keys for Mongo, its init container and Redis. They are
  empty: fill them with the digests running now before the first apply;
  the workflow refuses to apply while any image is `latest`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Staging resource limits move to their own PR so this one only changes
production. Staging Mongo nulls out the prod CPU settings it would
otherwise inherit and keeps memory only, as deployed today. Staging Redis
keeps its env label and inherits the prod resources until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
The first PR run failed two ways: GitHub releases returned 504 for the
helm-diff plugin download, and the HELM_DIFF_KUBECONFIG secret was
rejected by the API server ("the server has asked for the client to
provide credentials"), so its token has expired or been revoked.

- retry the plugin install up to three times
- check the kubeconfig can read Helm releases before diffing, and fail
  with instructions when it cannot
- document how to regenerate the secret for helm-diff-sa

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
- Three of five diff jobs failed because every attempt to download the
  helm-diff release from GitHub returned 504. Fetch the archive with curl
  retries and unpack it into the plugin directory instead of relying on
  `helm plugin install`, which gives up after one try.
- The production diff showed the migrations ConfigMap going backwards:
  infra/mongo/migrations/seed-requirements.js (generated Apr 2) was older
  than migrations/seed-requirements.js and the deployed copy (May 6), and
  missed #1263. Copy the current file so applying does not downgrade it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Production: the digests bt-prod-mongo (MongoDB 8.3.4, os-shell) and
bt-prod-redis ran on 2026-11-08, so applying keeps the same images.

Staging Mongo still runs docker.io/bitnamilegacy/mongodb:7.0.5 from an
older chart. Inheriting the production pin would jump it from 7.0 to 8.3,
which MongoDB does not support, so staging pins its current tags until it
is upgraded on purpose. Staging Redis keeps no data and inherits the
production pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Even with ten curl retries over 100 seconds, the helm-diff release
download from GitHub kept returning 504 on some runners. Compare
`helm get manifest` (what is deployed) with `helm template` (what the
repository would deploy) using diff instead, in
infra/scripts/release-diff.sh. It needs only read access to the release's
Helm secrets, which helm-diff-sa already has, and leaves Kubernetes
Secrets out of the output.

Exit codes match the old --detailed-exitcode use: 0 no changes,
2 changes, 1 error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
@bfliao

bfliao commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

@sashakmurray, some redis and helm stuff

@bfliao
bfliao requested a review from sashakmurray November 10, 2026 23:25
@bfliao
bfliao marked this pull request as ready for review November 11, 2026 00:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-11T00:33:40.877630Z
We couldn’t complete this review. Comment @codex review to try again.
a7c5b73 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

code=0
values_file=""
if [ -n "$VALUES" ]; then values_file="infra/$CHART/$VALUES"; fi
infra/scripts/release-diff.sh "$RELEASE" "infra/$CHART" $values_file > diff.txt 2>&1 || code=$?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the release-diff script uses version numbers from Chart.yaml (0.1.0) whereas the later apply step overrides the version with 1.0.0-g<commit>. I think we need to package the chart with the deployment version before both diffing and applying so that the drift check is accurate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but one thing to note: that version contains the commit SHA, so it changes on every commit, not only when the chart changes.

For example:

  • Mon: apply from abc1234 → cluster labels helm.sh/chart: bt-base-1.0.0-gabc1234
  • Tue: an unrelated commit lands, main is now def5678
  • Next Mon: the weekly diff packages 1.0.0-gdef5678 → the label differs → "drift", although nothing in the chart changed

So the weekly check would fail whenever main moved since the last apply. Two ways to fix it:

  1. Keep the single packaging step, but derive the version from the last commit that touched the chart (1.0.0-g$(git log -1 --format=%h -- infra/<chart>)), so it only changes when the chart does. The packaging step needs fetch-depth: 0 (the checkout is a shallow sparse checkout). Until the first apply, the diff would still show label-only changes, because the cluster has 1.0.0 from init.sh.
  2. Ignore helm.sh/chart in release-diff.sh, next to where Secrets are already dropped. A few lines, and the diff no longer depends on the version at all.

2 seems to be of less headache, though. lemme know what you think

Comment thread .github/workflows/cd-infra.yaml Outdated
#
# The diff runs from GitHub with the HELM_DIFF_KUBECONFIG secret: a kubeconfig
# for the read-only helm-diff-sa service account (infra/base/templates/helm-diff-role.yaml).
# To regenerate it, on the server:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add info about this process in the docs rather than in the comments of this file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…eneration

runbooks.md:
- Deploying Infra Releases: how to read a diff, apply a release and roll back
- Regenerating the HELM_DIFF_KUBECONFIG Secret, so the steps live in the
  docs instead of a workflow comment
- Note in the /helm-diff section pointing to the new workflow

cicd-workflow.md:
- Infra Releases: what the workflow deploys and why it diffs before it applies

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi6srcGkgAJ2SDUhxt6
@bfliao
bfliao requested a review from sashakmurray November 15, 2026 10:23
The steps for regenerating the HELM_DIFF_KUBECONFIG secret now live in the
docs (runbooks.md). The workflow header comment only links to them, and the
"Check Cluster Access" error points there instead of to the top of this file.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi6srcGkgAJ2SDUhxt6

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants