Skip to content

fix(infra): alert after 2 hours when vcs-sync has no successful run - #3000

Merged
neilv-g merged 1 commit into
mainfrom
fix/vcs-sync-stalled-alert-window
Nov 20, 2026
Merged

neilv-g merged 1 commit into
mainfrom
fix/vcs-sync-stalled-alert-window

Conversation

@neilv-g

@neilv-g neilv-g commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

The "Ingestion Stalled" alert fires when a workflow hasn't had a successful run in any region for 23 hours. That fits the other workflows, which run every 6 hours, but vcs-sync runs every 5 minutes, so 23 hours is about 276 missed runs. This PR adds a no_success_alert_window_hours setting to single_stage_go_workflow and sets it to 2 for vcs-sync. Other workflows keep the default of 23.

A day is too long because nothing else would catch a vcs-sync outage. The scan queue alerts when a task waits more than 10 minutes, but if vcs-sync stops publishing, the queue just empties and that alert stays quiet. The failures that stop vcs-sync also tend to break the rest of the pipeline: a deleted topic stops webhook-triggered scans too, and a revoked app key makes the scanner drop every scan, because it treats the 401 as permanent and acks the message.

2 hours is a compromise. Matching the other workflows' 7-8 missed runs would mean about 40 minutes, which would probably fire during routine GitHub or Pub/Sub hiccups. 2 hours (about 24 runs) rides those out and still catches a real break the same day. It's a judgment call rather than something based on an SLO, and it will also fire during a GitHub outage longer than 2 hours.

This also renames the alert to "No Successful Run in N Hours", which is what it actually checks. With #2998, vcs-sync's alert also fires when one repo or installation fails on every run while everything else is still ingested, so "Ingestion Stalled" would be misleading. Anyone filtering alert emails on the old name will need to update the filter.

It also drops "7-8 consecutive runs failed" from the alert text, since that number depends on each workflow's schedule. Applying it updates all 8 alert policies in place in staging and prod. They all get the new name and text, and vcs-sync's also gets the 2-hour window.

Split out of #2998, and should merge after it. Until #2998 lands, a vcs-sync run still succeeds when every publish fails or GitHub rejects the app's key, so a shorter window wouldn't catch those. Once it lands, a run fails on any error, so this alert fires after 2 hours without a successful run. That includes steady rate limiting, one repo that never publishes, or the job not running at all.

The Ingestion Stalled alert waits 23 hours. That suits workflows that
run every 6 hours, but vcs-sync runs every 5 minutes, so 23 hours is
about 276 missed runs.

Add a no_success_alert_window_hours setting to single_stage_go_workflow
(default 23, so other workflows keep their window) and set it to 2 for
vcs-sync. Rename the alert from "Ingestion Stalled (No Success in N
Hours)" to "No Successful Run in N Hours": it fires when no run has
succeeded, which doesn't always mean ingestion stopped. Also drop
"7-8 consecutive runs failed" from the alert text, since that number
depends on each workflow's schedule.
@neilv-g
neilv-g force-pushed the fix/vcs-sync-stalled-alert-window branch from 397d779 to 941c5bc Compare November 20, 2026 00:25
@neilv-g
neilv-g added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit d78e2a2 Nov 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants