Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 288 additions & 0 deletions .github/workflows/cd-infra.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,288 @@
# Deploys the Helm releases that CD: Deploy to Production/Staging does not:
# bt-base (backup CronJob, MetalLB, cert issuer), Mongo and Redis.
# Before this workflow they were installed by hand from infra/init.sh, so
# changes to infra/base, infra/mongo and infra/redis never reached the cluster.
#
# - Pull requests that touch those charts: posts the diff against what is
# deployed to the job summary.
# - Weekly: the same diff for every release from main. Fails if the cluster
# no longer matches the repository.
# - Run workflow (manual): diff, then with "apply" checked, deploys one
# release after the environment's approval. Applying Mongo or Redis
# restarts it when its pod spec changes; a Redis restart logs every user out.
#
# Releases, charts and values files are listed in infra/releases.json.
#
# The diff runs from GitHub with the HELM_DIFF_KUBECONFIG secret, a read-only
# kubeconfig. When its token expires, regenerate it: see "Regenerating the
# HELM_DIFF_KUBECONFIG Secret" in the docs
# (https://docs.berkeleytime.com/core/infrastructure/runbooks.html).
name: "CD: Infra Releases"
run-name: >-
${{ github.event_name == 'workflow_dispatch'
&& format('{0} {1}', inputs.apply && 'Apply' || 'Diff', inputs.release)
|| format('Infra diff ({0})', github.event_name) }}

on:
pull_request:
paths:
- "infra/base/**"
- "infra/mongo/**"
- "infra/redis/**"
- "infra/releases.json"
- "infra/scripts/**"
- ".github/workflows/cd-infra.yaml"
schedule:
- cron: "23 17 * * 1" # Mondays, 17:23 UTC
workflow_dispatch:
inputs:
release:
description: "Release to diff or apply"
required: true
type: choice
options:
- bt-base
- bt-prod-mongo
- bt-prod-redis
- bt-stage-mongo
- bt-stage-redis
apply:
description: "Apply after the diff (needs environment approval)"
required: true
type: boolean
default: false

permissions:
contents: read

concurrency:
group: cd-infra-${{ github.event_name == 'workflow_dispatch' && inputs.release || github.ref }}
cancel-in-progress: false

env:
HELM_VERSION: "3.16.4"

jobs:
select:
name: Select Releases
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.select.outputs.matrix }}
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
sparse-checkout: infra

- name: Select
id: select
env:
RELEASE: ${{ inputs.release }}
run: |
if [ -n "$RELEASE" ]; then
releases=$(jq -c --arg r "$RELEASE" '[.[] | select(.release == $r)]' infra/releases.json)
else
releases=$(jq -c '.' infra/releases.json)
fi
if [ "$(jq length <<< "$releases")" -eq 0 ]; then
echo "Release $RELEASE is not in infra/releases.json" >&2
exit 1
fi
echo "matrix={\"include\":$releases}" >> "$GITHUB_OUTPUT"

diff:
name: Diff ${{ matrix.release }}
needs: [select]
# Secrets are not available to pull requests from forks
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.select.outputs.matrix) }}
env:
RELEASE: ${{ matrix.release }}
CHART: ${{ matrix.chart }}
VALUES: ${{ matrix.values }}
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
sparse-checkout: infra

- name: Set up Helm
uses: azure/setup-helm@v5
with:
version: ${{ env.HELM_VERSION }}

- name: Build Chart Dependencies
run: |
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency build "infra/$CHART"

- name: Set up kubeconfig
env:
KUBECONFIG_CONTENT: ${{ secrets.HELM_DIFF_KUBECONFIG }}
run: |
mkdir -p ~/.kube
printf '%s' "$KUBECONFIG_CONTENT" > ~/.kube/config
chmod 600 ~/.kube/config

- name: Check Cluster Access
run: |
# Helm stores releases as Secrets labelled owner=helm; reading them is
# all the diff needs from the cluster.
if ! out=$(kubectl get secrets --namespace bt -l owner=helm -o name --request-timeout=20s 2>&1); then
echo "$out"
echo "::error::The HELM_DIFF_KUBECONFIG secret cannot read Helm releases in namespace bt (expired or revoked token?). Regenerate it: see 'Regenerating the HELM_DIFF_KUBECONFIG Secret' in the docs (https://docs.berkeleytime.com/core/infrastructure/runbooks.html)."
exit 1
fi

- name: Diff Against Cluster
run: |
args=(--namespace bt)
if [ -n "$VALUES" ]; then args+=(--values "infra/$CHART/$VALUES"); fi

{
echo "### $RELEASE"
echo
echo "Images:"
echo '```'
helm template "$RELEASE" "infra/$CHART" "${args[@]}" | infra/scripts/check-images.sh 2>&1 || true
echo '```'
} >> "$GITHUB_STEP_SUMMARY"

# Plain helm get manifest + helm template + diff, no helm-diff
# plugin: its GitHub release download returned 504 for minutes at a time.
code=0
values_file=""
if [ -n "$VALUES" ]; then values_file="infra/$CHART/$VALUES"; fi
infra/scripts/release-diff.sh "$RELEASE" "infra/$CHART" $values_file > diff.txt 2>&1 || code=$?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the release-diff script uses version numbers from Chart.yaml (0.1.0) whereas the later apply step overrides the version with 1.0.0-g<commit>. I think we need to package the chart with the deployment version before both diffing and applying so that the drift check is accurate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but one thing to note: that version contains the commit SHA, so it changes on every commit, not only when the chart changes.

For example:

  • Mon: apply from abc1234 → cluster labels helm.sh/chart: bt-base-1.0.0-gabc1234
  • Tue: an unrelated commit lands, main is now def5678
  • Next Mon: the weekly diff packages 1.0.0-gdef5678 → the label differs → "drift", although nothing in the chart changed

So the weekly check would fail whenever main moved since the last apply. Two ways to fix it:

  1. Keep the single packaging step, but derive the version from the last commit that touched the chart (1.0.0-g$(git log -1 --format=%h -- infra/<chart>)), so it only changes when the chart does. The packaging step needs fetch-depth: 0 (the checkout is a shallow sparse checkout). Until the first apply, the diff would still show label-only changes, because the cluster has 1.0.0 from init.sh.
  2. Ignore helm.sh/chart in release-diff.sh, next to where Secrets are already dropped. A few lines, and the diff no longer depends on the version at all.

2 seems to be of less headache, though. lemme know what you think

cat diff.txt

{
if [ "$code" -eq 0 ]; then
echo "No changes: the cluster matches the repository."
elif [ "$code" -eq 2 ]; then
echo "<details><summary>Changes the repository would make</summary>"
echo
echo '```diff'
head -c 60000 diff.txt
echo
echo '```'
echo "</details>"
else
echo "Diff failed (exit $code):"
echo '```'
tail -n 40 diff.txt
echo '```'
fi
} >> "$GITHUB_STEP_SUMMARY"

if [ "$code" -ne 0 ] && [ "$code" -ne 2 ]; then exit "$code"; fi
# The weekly run fails when the cluster drifted from main
if [ "$code" -eq 2 ] && [ "$GITHUB_EVENT_NAME" = "schedule" ]; then
echo "::error::$RELEASE on the cluster does not match main. Apply it from 'Run workflow'."
exit 1
fi

apply:
name: Apply ${{ matrix.release }}
needs: [select, diff]
if: github.event_name == 'workflow_dispatch' && inputs.apply
runs-on: ubuntu-latest
strategy:
matrix: ${{ fromJSON(needs.select.outputs.matrix) }}
environment: ${{ matrix.environment }}
env:
RELEASE: ${{ matrix.release }}
CHART: ${{ matrix.chart }}
VALUES: ${{ matrix.values }}
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
sparse-checkout: infra

- name: Set up Helm
uses: azure/setup-helm@v5
with:
version: ${{ env.HELM_VERSION }}

- name: Build Chart Dependencies
run: |
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency build "infra/$CHART"

- name: Refuse Unpinned Images
run: |
args=(--namespace bt)
if [ -n "$VALUES" ]; then args+=(--values "infra/$CHART/$VALUES"); fi
helm template "$RELEASE" "infra/$CHART" "${args[@]}" | infra/scripts/check-images.sh

- name: Package and Push Chart
id: chart
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
run: |
# A version per commit, so the release history says what was deployed
version="1.0.0-g${GITHUB_SHA::7}"
helm package "infra/$CHART" --version "$version"
printf '%s' "$DOCKER_PASSWORD" | helm registry login registry-1.docker.io -u "$DOCKER_USERNAME" --password-stdin
helm push "bt-$CHART-$version.tgz" "oci://registry-1.docker.io/$DOCKER_USERNAME"
echo "version=$version" >> "$GITHUB_OUTPUT"
if [ -n "$VALUES" ]; then
echo "values_b64=$(base64 -w0 "infra/$CHART/$VALUES")" >> "$GITHUB_OUTPUT"
else
echo "values_b64=" >> "$GITHUB_OUTPUT"
fi

- name: SSH and Helm Upgrade
uses: appleboy/ssh-action@v1.2.5
env:
CHART_REF: oci://registry-1.docker.io/octoberkeleytime/bt-${{ matrix.chart }}
CHART_VERSION: ${{ steps.chart.outputs.version }}
VALUES_B64: ${{ steps.chart.outputs.values_b64 }}
with:
host: ${{ secrets.SSH_HOST }}
username: ${{ secrets.SSH_USERNAME }}
key: ${{ secrets.SSH_KEY }}
command_timeout: 20m
envs: RELEASE,CHART_REF,CHART_VERSION,VALUES_B64
script: |
set -euo pipefail
cri=unix:///run/containerd/containerd.sock

values=$(mktemp)
rendered=$(mktemp)
trap 'rm -f "$values" "$rendered"' EXIT
printf '%s' "$VALUES_B64" | base64 -d > "$values"

echo "=== Deployed before"
helm history "$RELEASE" --namespace bt --max 3 || true

# Make sure every image is on the node or can be pulled before
# anything restarts, so a missing image fails here instead of
# leaving the database unable to start.
helm template "$RELEASE" "$CHART_REF" --version "$CHART_VERSION" \
--namespace bt --values "$values" > "$rendered"
for image in $(grep -E '^[[:space:]]*-?[[:space:]]*image:[[:space:]]' "$rendered" \
| sed -E 's/^[[:space:]]*-?[[:space:]]*image:[[:space:]]*//; s/["'\'']//g' | sort -u); do
if crictl --runtime-endpoint "$cri" inspecti "$image" > /dev/null 2>&1; then
echo "on node: $image"
else
echo "pulling: $image"
crictl --runtime-endpoint "$cri" pull "$image"
fi
done

if ! helm upgrade "$RELEASE" "$CHART_REF" --version "$CHART_VERSION" \
--namespace bt --values "$values" --wait --timeout 10m; then
echo "=== Upgrade failed. Pods:"
kubectl -n bt get pods -l "app.kubernetes.io/instance=$RELEASE" -o wide || true
echo "Roll back with: helm rollback $RELEASE --namespace bt"
exit 1
fi

echo "=== Deployed after"
helm history "$RELEASE" --namespace bt --max 3
13 changes: 13 additions & 0 deletions apps/docs/src/core/infrastructure/cicd-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,19 @@ The differences between the three environments are managed by each individual wo
| Deployment Count Limit | 8 | 1 | 1 |
| Datapuller `suspend` | `true` | `false` | `false` |

## Infra Releases

The Deploy Phase above covers the app, AG, staff and observability charts. The other Helm releases (`bt-base` with the backups, MetalLB and certificate issuer, plus Mongo and Redis for production and staging) are managed by a separate workflow, `.github/workflows/cd-infra.yaml` (**CD: Infra Releases**). Before it existed they were installed once by hand from `infra/init.sh`, so changes to `infra/base`, `infra/mongo` and `infra/redis` never reached the cluster and the cluster drifted from the repository.

The workflow has two steps:

1. **Diff**: compares what a release has deployed with what the repository would deploy. It runs on pull requests that touch those charts, every Monday for every release (failing if the cluster no longer matches `main`), and on demand.
2. **Apply**: only on demand, only after the diff succeeds, and only after someone approves the `production` or `staging` environment. It packages the chart with a version that identifies the commit, pushes it to the registry, and runs `helm upgrade` on `hozer-51`.

The diff comes first because applying changes the live cluster. Mongo and Redis restart when their pod spec changes, and a Redis restart logs every user out. These releases used to be edited by hand, so the cluster and the repository can differ, and applying makes the cluster match the repository: that can overwrite a manual fix or change an image. The diff shows exactly what will change, and it is what the approver reads before approving. If the cluster cannot be reached (for example, the `HELM_DIFF_KUBECONFIG` secret expired), the run fails at the diff, before anything is changed.

See [Deploying Infra Releases](./runbooks.md#deploying-infra-releases) for how to run it, and [Regenerating the `HELM_DIFF_KUBECONFIG` Secret](./runbooks.md#regenerating-the-helm_diff_kubeconfig-secret) for when the diff cannot reach the cluster.

[^1]: In the past, we have used a self-hosted GitLab instance. However, the CI/CD pipeline was obscured behind a admin login page. Hopefully, with GitHub actions, the deployment process will be more transparent and accessible to all engineers. Please don't break anything though!

[^2]: Development, Staging, and Production
Expand Down
75 changes: 75 additions & 0 deletions apps/docs/src/core/infrastructure/runbooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,81 @@ To use it:

The diff output is formatted as collapsible sections for each resource, with a raw diff available at the bottom for debugging.

> [!NOTE]
> Changes to `infra/base`, `infra/mongo` and `infra/redis` are also diffed automatically on every pull request by the **CD: Infra Releases** workflow. See [Deploying Infra Releases](#deploying-infra-releases).

## Deploying Infra Releases

The app, AG, staff and observability charts are deployed by the `CD: Deploy to ...` workflows. The other Helm releases are deployed by the **CD: Infra Releases** workflow (`.github/workflows/cd-infra.yaml`). For how it works and why it diffs before it applies, see [Infra Releases](./cicd-workflow.md#infra-releases).

| Release | Chart | Environment |
| --- | --- | --- |
| `bt-base` | `infra/base` | production |
| `bt-prod-mongo` | `infra/mongo` | production |
| `bt-prod-redis` | `infra/redis` | production |
| `bt-stage-mongo` | `infra/mongo` (`values-staging.yaml`) | staging |
| `bt-stage-redis` | `infra/redis` (`values-staging.yaml`) | staging |

The list lives in `infra/releases.json`.

### Reading a diff

The workflow compares what a release has deployed (`helm get manifest`) with what the repository would deploy (`helm template`). Kubernetes Secrets are left out of the output.

- **Pull requests** that touch `infra/base`, `infra/mongo`, `infra/redis`, `infra/releases.json` or `infra/scripts`: open the run from the pull request's Checks tab and read the diff in the job summary.
- **Every Monday**: the diff runs for every release from `main`. The run **fails if the cluster no longer matches the repository**.
- **Run workflow**: diffs one release, and applies it if you ask.

The summary shows either `No changes: the cluster matches the repository.` or the changes the repository would make. It also lists every image and whether it is pinned. Applying is refused while any image has no fixed tag or digest (no tag, or the `latest` tag).

### Applying a release

1. Go to **Actions → CD: Infra Releases → Run workflow**. Run it from `main`: the Monday check compares the cluster with `main`, so applying from another branch makes it fail.
2. Pick the release in **Release to diff or apply** and check **Apply after the diff**.
3. When the diff finishes, read it in the job summary. Approve the `production` or `staging` deployment only if the changes are the ones you expect.
4. After approval, the workflow packages the chart with a version that identifies the commit and pushes it to the registry. Then, over SSH on `hozer-51`, it checks that every image is on the node (pulling it if not) and runs `helm upgrade --wait --timeout 10m`.
5. Check that the pods are running: `k get pods -l app.kubernetes.io/instance=<release>`.

> [!IMPORTANT]
> Mongo and Redis restart when their pod spec changes, and a Redis restart logs every user out. Apply production Mongo and Redis at a quiet hour, and apply the matching staging release first.

If the upgrade fails, the job prints the pods and the rollback command. To roll back, run this on `hozer-51`:

```sh
helm rollback <release> --namespace bt
```

> [!NOTE]
> Don't change these releases by hand (`kubectl edit`, `redis-cli CONFIG SET`, ...). The next apply overwrites the change, and the Monday check fails until the repository matches the cluster. Change the chart or values in the repository instead.

## Regenerating the `HELM_DIFF_KUBECONFIG` Secret

The diff jobs (and the `/helm-diff` pull request command) read the cluster from GitHub Actions with the `HELM_DIFF_KUBECONFIG` repository secret. It holds a kubeconfig for the read-only `helm-diff-sa` service account (`infra/base/templates/helm-diff-role.yaml`). The token inside expires after a year (or the API server's maximum). When it does, the jobs fail at **Check Cluster Access** with:

> The HELM_DIFF_KUBECONFIG secret cannot read Helm releases in namespace bt (expired or revoked token?)

To regenerate it, SSH into `hozer-51` and run:

```sh
token=$(kubectl -n bt create token helm-diff-sa --duration=8760h)
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' > /tmp/ca
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
KUBECONFIG=/tmp/helm-diff.kubeconfig kubectl config set-cluster bt --server="$server"
KUBECONFIG=/tmp/helm-diff.kubeconfig kubectl config set clusters.bt.certificate-authority-data "$(cat /tmp/ca)"
KUBECONFIG=/tmp/helm-diff.kubeconfig kubectl config set-credentials helm-diff-sa --token="$token"
KUBECONFIG=/tmp/helm-diff.kubeconfig kubectl config set-context bt --cluster=bt --user=helm-diff-sa --namespace=bt
KUBECONFIG=/tmp/helm-diff.kubeconfig kubectl config use-context bt
```

Then:

1. Copy the contents of `/tmp/helm-diff.kubeconfig` into the `HELM_DIFF_KUBECONFIG` secret (**Settings → Secrets and variables → Actions**).
2. Delete the local files: `rm /tmp/ca /tmp/helm-diff.kubeconfig`.
3. Run **CD: Infra Releases** for any release with **Apply after the diff** unchecked, and check that **Check Cluster Access** passes.

> [!WARNING]
> `helm-diff-sa` cannot change anything, but it can read the Secrets in the `bt` namespace (Helm stores release records in Secrets). Treat the kubeconfig as a credential: don't paste it into chats, issues or logs, and delete the local copies.

## Uninstall ALL development helm releases

```sh
Expand Down
Loading
Loading