Repository navigation
fix(gitlab): Fail fast on connect for project webhook calls - #128491
Open
sentry-junior[bot] wants to merge 1 commit into
Open
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
Repo sync restores and reconciles webhooks for up to 100 repositories in one restore_repos_batch task with a 120s deadline. Against an unreachable self-hosted GitLab, each hook call waited the full 10s proxy client connect timeout, so a batch could exceed the deadline. Use a 3s connect timeout for the project hook create, update, and delete calls. Co-Authored-By: Giovanni Guidini <giovanni.guidini@sentry.io>
github-actions
Bot
added
the
Scope: Backend
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #127268,
restore_repos_batchcallson_create_repositoryfor every repository it reactivates. The task handles up to 100 repositories and has a 120s deadline. When the self-hosted GitLab instance can't be reached, each project hook call waits out the full 10sIntegrationProxyClientconnect timeout. That adds up past the deadline, so the task is killed while it's still connecting (ProcessingDeadlineExceededinrestore_repos_batch, with the stack ending increate_project_webhook→sock.connect).The project hook create, update and delete calls now use a 3s connect timeout. The read timeout stays at the 10s client default. These calls go to a host that is either reachable or not, so a long connect wait doesn't help.
This cuts the cost per repository from about 10s to about 3s, but it doesn't fully remove the risk. A batch with enough repositories on a dead host can still go over 120s, because the loop keeps trying each repository. If that turns out to matter, a follow-up can stop reconciling webhooks for the rest of a batch after the first host-unreachable error.
via Gio.
--
View Junior Session [Sentry]