Repository navigation
Conversation
Staging shares hozer-51 with production. It ran the same datapuller jobs at the same minutes (enrollment every 15 minutes, the monthly five-year rebuilds at 04:00-04:35 on the 1st), with no CPU or memory limits, and staging Mongo used more CPU than production Mongo. Staging app (deployed by CD: Deploy to Staging): - datapuller jobs: request 50m/256Mi, limit 1 CPU/2Gi - datapuller schedules an hour after prod's; enrollment hourly at :07 - backend: 1 replica instead of 2 - semantic search: request 100m/256Mi, limit 500m/1Gi Staging Mongo: request 100m, limit 500m. Staging Redis: 128Mi/256Mi, 50m/250m, maxmemory 150mb. Both are applied by CD: Infra Releases. The app chart gains optional datapuller.resources and semanticSearch.resources, empty by default, so the prod render is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
Up to 8 preview deploys run on the same node as production, each with its own semantic-search pod that loads the embedding model and had no CPU or memory limit. Give previews the staging limits: request 100m/256Mi, limit 500m/1Gi. Previews already run one backend replica with the datapuller suspended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1312. It targets
leo/infra-releases-cd, so the diff here is staging only. Merge #1312 first, then retarget this PR tomain.Problem
Staging shares the one node (hozer-51) with production and competes with it:
Change
Staging app (
cd-stage.yaml; takes effect on the next staging deploy, which happens on merge tomain):Staging databases (applied with
CD: Infra Releasesfrom #1312):maxmemory 150mb,volatile-lruPreviews (
cd-dev.yaml; takes effect on the next preview deploy): up to 8 previews run on the same node, each with its own semantic-search pod and no limits. They now get the staging limits: request 100m / 256Mi, limit 500m / 1Gi. With 8 previews that's at most 4 cores and 8Gi in total. Previews already run one backend replica with the datapuller suspended.Chart:
infra/appgains optionaldatapuller.resourcesandsemanticSearch.resources. Both are empty by default, so production is unaffected.Testing
bt-prod-appbefore and after with Helm 3.16.4: byte-for-byte identical.bt-stage-appwith thecd-stagevalues before and after. The diff contains only the 12 schedules, the 12 datapuller resource blocks, the backend replica count and the semantic-search resources. All 25 rendered documents parse.cd-stagevalues contain no",$or backticks, so they survive theecho "…"on the server.cd-devvalues (commit expressions replaced with a dummy hash). Its semantic-search pod gets the intended limits, and the values contain no unsafe characters.maxmemory.Risks
Generated by Claude Code