Repository navigation
fix: retry embedded PostgreSQL launch when the probed port is taken - #666
Merged
Merged
Conversation
The PG14 job of the v1.13.3 release run failed in TestApplyCommand_LockTimeoutRetryNotAppliedToConcurrentIndex with "pg_ctl: could not start server" and no further output. The port for the embedded server is reserved by probing it and closing the probe socket before pg_ctl binds it, so another process (such as a parallel test package starting its own instance) can take it in between. With log_min_messages=PANIC the postmaster's bind failure was never printed. StartEmbeddedPostgres now retries the launch up to three times, each on a fresh port and runtime directory, removes the data directory of a failed attempt, and runs the server at log_min_messages=LOG so a failed launch reports its cause in the error. The log still goes to the library's temp file, not to the user's terminal. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
[Medium impact] The changes affect embedded PostgreSQL startup used by plan generation and test setup. The PR appears safe to merge, with a non-blocking request to test the retry limit. Findings
SummaryAdds up to three embedded PostgreSQL launch attempts, each with a fresh port and runtime path.
Reviews (1) · Last reviewed commit: "fix: retry embedded PostgreSQL launch wh..." · Reviewed by Greptile |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Failed-start cleanup can delete a still-running PostgreSQL server's data directory without first ensuring process termination.
1 open issue
What changed in this PR
This PR makes embedded PostgreSQL startup more resilient to port-allocation races during planning and parallel tests.
Changes:
- Retry startup up to three times with fresh ports and runtime directories.
- Remove failed-attempt directories and retain startup diagnostics.
- Add test hooks and a regression test for retrying an occupied port.
| File | Description |
|---|---|
| internal/postgres/export_test.go | Exposes internal helpers for testing. |
| internal/postgres/embedded.go | Adds bounded startup retries, cleanup, and diagnostic logging. |
| internal/postgres/embedded_test.go | Tests successful retry after an occupied-port failure. |
Review effort: Balanced
When "pg_ctl start -w" gives up waiting, embedded-postgres returns an error but the server keeps running, and its Stop refuses to act because the instance was never marked started. Run "pg_ctl stop -m immediate" against the attempt's data directory before removing it, so a retry never deletes a running server's files. Also cover the retry bound: a start that fails on every attempt returns an error after three tries instead of retrying forever. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
The PG14 job of the v1.13.3 release run (38778778375) failed in
TestApplyCommand_LockTimeoutRetryNotAppliedToConcurrentIndexwith:The other four version jobs passed, and the same workflow passed for v1.13.2 ten hours earlier, so this is a flaky launch, not a logic failure.
Cause
findAvailablePortreserves a port by listening on:0and closing the socket; the embedded-postgres library probes the same port once more, then runspg_ctl. Nothing holds the port in between, so another process (five test packages start embedded instances in parallel undergo test ./...) can take it before the postmaster binds. The server ran withlog_min_messages=PANIC, which hides the postmaster's own bind failure (LOG: could not bind ...,FATAL: no socket created for listening). That is why the CI output is silent between "waiting for server to start" and "stopped waiting": any post-config startup failure was swallowed at that level.Fix (
internal/postgres/embedded.go)StartEmbeddedPostgresretries the launch up to 3 times, each on a fresh port and runtime directory.log_min_messages=LOG, so a failed launch reports its cause in the error. Output still goes to the library's temp log file (it is passedio.Discardas logger), so nothing new reaches the user's terminal.findAvailablePortbecomes a package variable so a test can inject a taken port.Test
TestStartEmbeddedPostgres_RetriesWhenPortIsTakenhands the first attempt a port held by a listener and asserts the second attempt comes up on a different port with exactly one retry. A throwaway variant where every attempt gets the taken port fails after 3 attempts, so the test is not vacuous.Test plan
go test ./internal/postgreson PG 18 (default) andPGSCHEMA_POSTGRES_VERSION=14go test ./cmd/apply(full package, includes the test that failed in CI)go vet ./...PANICand the full cause atLOG🤖 Generated with Claude Code