Skip to content

fix: retry embedded PostgreSQL launch when the probed port is taken - #666

Merged
tianzhou merged 2 commits into
mainfrom
fix/embedded-postgres-start-retry
Oct 27, 2026
Merged

tianzhou merged 2 commits into
mainfrom
fix/embedded-postgres-start-retry

Conversation

@tianzhou

Copy link
Copy Markdown
Contributor

Summary

The PG14 job of the v1.13.3 release run (38778778375) failed in TestApplyCommand_LockTimeoutRetryNotAppliedToConcurrentIndex with:

waiting for server to start.... stopped waiting
pg_ctl: could not start server

The other four version jobs passed, and the same workflow passed for v1.13.2 ten hours earlier, so this is a flaky launch, not a logic failure.

Cause

findAvailablePort reserves a port by listening on :0 and closing the socket; the embedded-postgres library probes the same port once more, then runs pg_ctl. Nothing holds the port in between, so another process (five test packages start embedded instances in parallel under go test ./...) can take it before the postmaster binds. The server ran with log_min_messages=PANIC, which hides the postmaster's own bind failure (LOG: could not bind ..., FATAL: no socket created for listening). That is why the CI output is silent between "waiting for server to start" and "stopped waiting": any post-config startup failure was swallowed at that level.

Fix (internal/postgres/embedded.go)

  • StartEmbeddedPostgres retries the launch up to 3 times, each on a fresh port and runtime directory.
  • The data directory of a failed attempt is removed instead of leaking under the temp dir.
  • The server runs at log_min_messages=LOG, so a failed launch reports its cause in the error. Output still goes to the library's temp log file (it is passed io.Discard as logger), so nothing new reaches the user's terminal.
  • findAvailablePort becomes a package variable so a test can inject a taken port.

Test

TestStartEmbeddedPostgres_RetriesWhenPortIsTaken hands the first attempt a port held by a listener and asserts the second attempt comes up on a different port with exactly one retry. A throwaway variant where every attempt gets the taken port fails after 3 attempts, so the test is not vacuous.

Test plan

  • go test ./internal/postgres on PG 18 (default) and PGSCHEMA_POSTGRES_VERSION=14
  • go test ./cmd/apply (full package, includes the test that failed in CI)
  • go vet ./...
  • Confirmed with a scratch program against embedded-postgres directly that a post-config startup failure prints nothing at PANIC and the full cause at LOG

🤖 Generated with Claude Code

The PG14 job of the v1.13.3 release run failed in
TestApplyCommand_LockTimeoutRetryNotAppliedToConcurrentIndex with
"pg_ctl: could not start server" and no further output. The port for
the embedded server is reserved by probing it and closing the probe
socket before pg_ctl binds it, so another process (such as a parallel
test package starting its own instance) can take it in between. With
log_min_messages=PANIC the postmaster's bind failure was never printed.

StartEmbeddedPostgres now retries the launch up to three times, each
on a fresh port and runtime directory, removes the data directory of a
failed attempt, and runs the server at log_min_messages=LOG so a failed
launch reports its cause in the error. The log still goes to the
library's temp file, not to the user's terminal.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 15:18
@greptile-apps

greptile-apps Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes affect embedded PostgreSQL startup used by plan generation and test setup.

The PR appears safe to merge, with a non-blocking request to test the retry limit.

Findings

  1. P2 Retry limit is untested ▶

Summary

Adds up to three embedded PostgreSQL launch attempts, each with a fresh port and runtime path.

  • Embedded PostgreSQL startup retries failed launches on fresh ports.

Reviews (1) · Last reviewed commit: "fix: retry embedded PostgreSQL launch wh..." · Reviewed by Greptile

Comment thread internal/postgres/embedded_test.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed-start cleanup can delete a still-running PostgreSQL server's data directory without first ensuring process termination.

1 open issue
What changed in this PR

This PR makes embedded PostgreSQL startup more resilient to port-allocation races during planning and parallel tests.

Changes:

  • Retry startup up to three times with fresh ports and runtime directories.
  • Remove failed-attempt directories and retain startup diagnostics.
  • Add test hooks and a regression test for retrying an occupied port.
File Description
internal/​postgres/​export_test.go Exposes internal helpers for testing.
internal/​postgres/​embedded.go Adds bounded startup retries, cleanup, and diagnostic logging.
internal/​postgres/​embedded_test.go Tests successful retry after an occupied-port failure.

Review effort: Balanced

Comment thread internal/postgres/embedded.go Resolved
When "pg_ctl start -w" gives up waiting, embedded-postgres returns an
error but the server keeps running, and its Stop refuses to act because
the instance was never marked started. Run "pg_ctl stop -m immediate"
against the attempt's data directory before removing it, so a retry
never deletes a running server's files.

Also cover the retry bound: a start that fails on every attempt returns
an error after three tries instead of retrying forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The localized changes have targeted regression coverage and no identified blocking issues; tests were not executed during this review.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@tianzhou
tianzhou merged commit 2a475ac into main Oct 27, 2026
2 checks passed
@tianzhou
tianzhou deleted the fix/embedded-postgres-start-retry branch October 27, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants