Skip to content

Limit staging so it cannot take resources from production - #1313

Draft
bfliao wants to merge 4 commits into
leo/infra-releases-cdfrom
leo/staging-limits
Draft

bfliao wants to merge 4 commits into
leo/infra-releases-cdfrom
leo/staging-limits

Conversation

@bfliao

@bfliao bfliao commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1312. It targets leo/infra-releases-cd, so the diff here is staging only. Merge #1312 first, then retarget this PR to main.

Problem

Staging shares the one node (hozer-51) with production and competes with it:

  • Same schedules: it runs every datapuller job at the same minutes as production, including enrollment every 15 minutes and the monthly five-year rebuilds at 04:00–04:35 PST on the 1st. On Nov 1 at 04:25, health checks for Redis and Mongo timed out node-wide, and pods restarted.
  • No limits: staging datapuller jobs and semantic search have no CPU or memory limits, and staging Mongo has no CPU limit.
  • Measured: at midnight, staging Mongo used 321m CPU against production Mongo's 274m.

Change

Staging app (cd-stage.yaml; takes effect on the next staging deploy, which happens on merge to main):

Part Before After
Datapuller jobs No CPU or memory limits Request 50m / 256Mi, limit 1 CPU / 2Gi
Datapuller schedules Same minutes as production One hour after production's. Enrollment hourly at :07, avoiding production's :00/:15/:30/:45
Backend 2 replicas 1 replica
Semantic search No limits Request 100m / 256Mi, limit 500m / 1Gi

Staging databases (applied with CD: Infra Releases from #1312):

Release After
Mongo CPU request 100m, limit 500m (memory unchanged at 1Gi / 2Gi)
Redis Memory 128Mi / 256Mi, CPU 50m / 250m, maxmemory 150mb, volatile-lru

Previews (cd-dev.yaml; takes effect on the next preview deploy): up to 8 previews run on the same node, each with its own semantic-search pod and no limits. They now get the staging limits: request 100m / 256Mi, limit 500m / 1Gi. With 8 previews that's at most 4 cores and 8Gi in total. Previews already run one backend replica with the datapuller suspended.

Chart: infra/app gains optional datapuller.resources and semanticSearch.resources. Both are empty by default, so production is unaffected.

Testing

  • Rendered bt-prod-app before and after with Helm 3.16.4: byte-for-byte identical.
  • Rendered bt-stage-app with the cd-stage values before and after. The diff contains only the 12 schedules, the 12 datapuller resource blocks, the backend replica count and the semantic-search resources. All 25 rendered documents parse.
  • All 12 staging schedule keys match the chart's job names, so no extra jobs get created.
  • cd-stage values contain no ", $ or backticks, so they survive the echo "…" on the server.
  • Rendered a preview release with the cd-dev values (commit expressions replaced with a dummy hash). Its semantic-search pod gets the intended limits, and the values contain no unsafe characters.
  • Staging Mongo and Redis render the intended limits and maxmemory.
  • Prettier passes.

Risks

  • A staging job could hit its memory limit. If a five-year rebuild in staging needs more than 2Gi, it's OOM-killed and retried, and staging data goes stale. Raise the limit if that happens.
  • Staging enrollment data updates hourly instead of every 15 minutes.
  • A preview's semantic search could hit 1Gi and restart, breaking AI search on that preview only. Staging's semantic search used about 344Mi. Previews also start a little slower with half a core; the startup check allows 10 minutes.

Generated by Claude Code

Staging shares hozer-51 with production. It ran the same datapuller jobs
at the same minutes (enrollment every 15 minutes, the monthly five-year
rebuilds at 04:00-04:35 on the 1st), with no CPU or memory limits, and
staging Mongo used more CPU than production Mongo.

Staging app (deployed by CD: Deploy to Staging):
- datapuller jobs: request 50m/256Mi, limit 1 CPU/2Gi
- datapuller schedules an hour after prod's; enrollment hourly at :07
- backend: 1 replica instead of 2
- semantic search: request 100m/256Mi, limit 500m/1Gi

Staging Mongo: request 100m, limit 500m. Staging Redis: 128Mi/256Mi,
50m/250m, maxmemory 150mb. Both are applied by CD: Infra Releases.

The app chart gains optional datapuller.resources and
semanticSearch.resources, empty by default, so the prod render is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Up to 8 preview deploys run on the same node as production, each with its
own semantic-search pod that loads the embedding model and had no CPU or
memory limit. Give previews the staging limits: request 100m/256Mi,
limit 500m/1Gi. Previews already run one backend replica with the
datapuller suspended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants