Repository navigation
fix(infra): alert after 2 hours when vcs-sync has no successful run - #3000
Merged
Merged
Conversation
The Ingestion Stalled alert waits 23 hours. That suits workflows that run every 6 hours, but vcs-sync runs every 5 minutes, so 23 hours is about 276 missed runs. Add a no_success_alert_window_hours setting to single_stage_go_workflow (default 23, so other workflows keep their window) and set it to 2 for vcs-sync. Rename the alert from "Ingestion Stalled (No Success in N Hours)" to "No Successful Run in N Hours": it fires when no run has succeeded, which doesn't always mean ingestion stopped. Also drop "7-8 consecutive runs failed" from the alert text, since that number depends on each workflow's schedule.
neilv-g
force-pushed
the
fix/vcs-sync-stalled-alert-window
branch
from
November 20, 2026 00:25
397d779 to
941c5bc
Compare
jrobbins
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Ingestion Stalled" alert fires when a workflow hasn't had a successful run in any region for 23 hours. That fits the other workflows, which run every 6 hours, but vcs-sync runs every 5 minutes, so 23 hours is about 276 missed runs. This PR adds a
no_success_alert_window_hourssetting tosingle_stage_go_workflowand sets it to 2 for vcs-sync. Other workflows keep the default of 23.A day is too long because nothing else would catch a vcs-sync outage. The scan queue alerts when a task waits more than 10 minutes, but if vcs-sync stops publishing, the queue just empties and that alert stays quiet. The failures that stop vcs-sync also tend to break the rest of the pipeline: a deleted topic stops webhook-triggered scans too, and a revoked app key makes the scanner drop every scan, because it treats the 401 as permanent and acks the message.
2 hours is a compromise. Matching the other workflows' 7-8 missed runs would mean about 40 minutes, which would probably fire during routine GitHub or Pub/Sub hiccups. 2 hours (about 24 runs) rides those out and still catches a real break the same day. It's a judgment call rather than something based on an SLO, and it will also fire during a GitHub outage longer than 2 hours.
This also renames the alert to "No Successful Run in N Hours", which is what it actually checks. With #2998, vcs-sync's alert also fires when one repo or installation fails on every run while everything else is still ingested, so "Ingestion Stalled" would be misleading. Anyone filtering alert emails on the old name will need to update the filter.
It also drops "7-8 consecutive runs failed" from the alert text, since that number depends on each workflow's schedule. Applying it updates all 8 alert policies in place in staging and prod. They all get the new name and text, and vcs-sync's also gets the 2-hour window.
Split out of #2998, and should merge after it. Until #2998 lands, a vcs-sync run still succeeds when every publish fails or GitHub rejects the app's key, so a shorter window wouldn't catch those. Once it lands, a run fails on any error, so this alert fires after 2 hours without a successful run. That includes steady rate limiting, one repo that never publishes, or the job not running at all.