Skip to content

Gradual copy throttle recovery #471

Open

Description

@driv3r

Gradual copy throttle recovery

Context

Ghostferry currently stalls all copying under throttle and resumes all tables at full speed. The requested improvement is gradual recovery: resume one table with smaller batches, then increase concurrency after stability. Implement this through copy-batch admission while preserving the existing hard safety stop and independent binlog replay. During implementation, create one GitHub issue tracking the complete plan and deliver the work as a dependency-ordered stack of pull requests.

Findings

  • throttler.go:12-34: Throttler exposes a boolean; WaitForThrottle polls every 500 ms and releases all waiters when unthrottled or disabled.

  • throttler.go:73-164: LagThrottler combines operator pause with lag > MaxLag; its default update interval is 1 second. Missing/NULL lag samples retain the previous value.

  • data_iterator.go:62-165: a fixed Concurrency worker pool consumes whole tables; each worker calls cursor.Each and synchronous batch listeners until its table is complete.

  • binlog_writer.go:33-91: a bounded event channel feeds one writer, and BufferBinlogEvents sends synchronously into it. Making the writer wait on copy work would backpressure the streamer; the new recovery limiter must therefore govern copy admission only, leaving the writer's existing hard-throttle check unchanged.

  • cursor.go:98-155: the source transaction remains open across fetch and synchronous target-write listeners. Admission must occur before DB.Begin; release must occur after source rollback, including failure/empty-batch paths. Feeding tablesQueue more slowly cannot constrain already-running table cursors.

  • batch_writer.go:50-137: target retries and inline verification run while the source transaction remains open; copy pagination advances only after target commit. Do not insert scheduling waits inside this callback.

  • ferry.go:794-823: binlog writer, binlog streamer and data iterator run independently. RunStandaloneDataCopy also constructs a data iterator and runs during cutover without its own supporting-services loop.

  • row_batch.go:62-74: copy uses INSERT IGNORE, not an unconditional overwrite. Preserve this and source-share-lock protection when changing admission.

  • config.go:975-1005, cursor.go:66-73, control_server.go:370-392: live batch-size updates currently share a plain uint64; all runtime reads/writes must use atomic access before recovery composes with those updates. Per-table overrides remain the configured base.

  • Local find and literal searches found no existing semaphore, adaptive limiter or injectable clock abstraction. The small recovery controller below is new code rather than a second scheduler convention.

  • Publication discovery: origin is https://github.com/Shopify/ghostferry.git, the default branch is main, issues are enabled, and the current GitHub identity reports ADMIN. The clean investigate-throttling-improvements checkout and origin/main both pointed to 6f4289c7ca6ec1b36481aee7d70b23d5da928126 during planning. gh 2.103.0 is installed; an all-state issue search found no existing issue titled Gradual copy throttle recovery. Root/.github searches found no required stack tool or branch-naming convention; use ordinary Git branches and explicit GitHub PR bases.

  • .github/workflows/tests.yml:3-8 runs CI for PRs without a base-branch restriction, so dependent PR bases are supported. Its Go/Ruby matrix jobs use continue-on-error: true; an overall green workflow is not proof those tests passed. README.md:215-218 documents prerelease creation on pushes to main; this task publishes reviewable branches/PRs, not merges or releases.

Tracking issue and PR stack

Tracking issue: #471 — #471

Create the tracking issue before implementation

  • After implementation is approved, create one issue in Shopify/ghostferry titled Gradual copy throttle recovery. Its body is the complete approved contents of local://adaptive-copy-scheduling-plan.md, including the implementation specification, stack breakdown and verification—not a summary or a link to an inaccessible local file. Use gh issue create --repo Shopify/ghostferry --title "Gradual copy throttle recovery" --body-file -, supplying the Markdown on stdin.
  • Record the returned issue number/URL immediately in this canonical plan's tracking entry and use that issue for all three PRs. Do not invent an issue number. On an interrupted/ambiguous publication attempt, query the exact title and inspect the body before retrying; reuse the issue created for this plan rather than creating a duplicate. No labels, assignees or milestone are requested.
  • Keep this checklist in the issue; replace each PR label with its actual URL after publication. Check a PR item only after that PR is merged into main; evidence can be linked while review is pending. Keep the issue open at handoff—publishing the stack is not permission to merge it or close the issue.

PR 1 — Make shared throttle inputs race-free

  • Title: [1/3] Make throttle and batch-size access race-free.
  • Head: throttle-recovery/01-safe-inputs; base: main.
  • Implement only the existing-value synchronization portions of Approach steps 3 and 4: atomic batch-size store/load and HTTP config snapshot in config.go, cursor.go and control_server.go; protect the existing LagThrottler.lag reads/writes with a mutex in throttler.go. Preserve current throttle, polling, batch-size and recovery behavior. Do not add unused recovery configuration, sample metadata, controller methods or scheduling code to this PR.
  • Add TestConcurrentBatchSizeUpdates in the existing test/go/control_server_test.go (currently only its package declaration). Exercise the real config GET/POST handlers with httptest, concurrent updates alternating 104/200 and concurrent CursorConfig.GetBatchSize reads. Assert every observed base is one of the configured values, an explicit per-table override stays 80, and a final completed update to 104 is visible through both the HTTP snapshot and cursor getter. Coordinate goroutines with a barrier/WaitGroup; no sleeps. The race detector is part of this concurrency regression's acceptance.
  • Gate: MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestConcurrentBatchSizeUpdates|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs|TestDataIterator/TestBatchSizeUpdate' -count=1 -p=1. Also run a real-Ferry smoke using the fixture/lifecycle in Verification with concurrency 1, base 200, no initial pause and no new recovery config: hold its first 200-row callback, POST base 104, release, observe the next batch contains 104 rows, then drain and compare source/target rows. This verifies the live update path without waiting for PR 2.

PR 2 — Implement complete gradual copy recovery

  • Title: [2/3] Recover copy load gradually after throttling.
  • Head: throttle-recovery/02-gradual-recovery; base: throttle-recovery/01-safe-inputs.
  • Implement all remaining behavior in Approach steps 1–4 as one end-to-end feature: validated recovery config, FIFO permits and staged limits, cursor admission before source transactions, scaled batches, Ferry/standalone iterator lifecycle, and fresh lag observations. Reuse PR 1's synchronization; do not copy or reintroduce its changes. Production changes are in config.go, copy_throttle.go (new), data_iterator.go, cursor.go, ferry.go and throttler.go. Do not change binlog scheduling.
  • Include the root copy_throttle_test.go controller coverage, config/lag validation boundary tests in test/go/config_test.go and test/go/lag_throttler_test.go, and the real-MySQL recovery, source-lock, independent-replay and live-batch-size cases in test/go/data_iterator_test.go/test/go/race_conditions_integration_test.go. Core behavior and its regression tests land together; this PR must not depend on PR 3 for the recovery algorithm to work or for its primary safety tests.
  • Gate: Verification commands 1–5 and the full real-program smoke in both coordinate modes. Keep the existing interrupt/resume suite passing; the new recovery-specific subprocess case belongs to PR 3. Publish no intermediate unused API, partial controller or disabled scaffold.

PR 3 — Verify interruption and standalone cutover

  • Title: [3/3] Verify throttle recovery across interruption and cutover.
  • Head: throttle-recovery/03-resume-coverage; base: throttle-recovery/02-gradual-recovery.
  • Add the recovery-specific Ruby interrupt/resume regression and test/lib/go/throttle_recovery_ghostferry/main.go executable exactly as specified in Verification. Add TestCopyThrottleStandaloneCopyBypassesPause in test/go/data_iterator_test.go: with nonempty source tables, empty targets, SetPaused(true) and SetDisabled(true), run the actual RunStandaloneDataCopy without a Ferry.Run supporting-services loop; it must finish and preserve exact row contents. This catches a recovery sampler/bypass deadlock in the cutover-only path rather than merely checking a copied flag.
  • This PR contains regression coverage, not another scheduling change. If these cases expose a production defect, place its fix and directly associated tests in PR 2, then restack PR 3; do not hide the feature fix in the verification layer.
  • Gate: Verification command 6 in both coordinate modes; MYSQL_VERSION=8.0 go test ./test/go -run '^TestCopyThrottleStandaloneCopyBypassesPause$' -count=1 -p=1; and MYSQL_VERSION=8.0 go test ./sharding/test -run 'TestJoinedTablesTestSuite|TestPrimaryKeyTableTestSuite' -count=1 -p=1. Run the complete Verification command set at the stack tip before handoff, recording actual outcomes and the real-program smoke evidence on the issue.

Branch, publication and review procedure

  • Build the three layers bottom-up. After approval, fetch origin, create PR 1's branch from origin/main, commit its scoped changes, create PR 2's branch from PR 1's tip, then create PR 3's branch from PR 2's tip. Use the exact head/base names above. Preserve the existing investigation branch and unrelated working-tree changes; if the checkout becomes dirty with unrelated work, use a separate worktree at /tmp/ghostferry-throttle-recovery instead of resetting or stashing it. If that worktree path is occupied, use the lowest free numeric suffix. Reuse an existing stack branch only when its issue/PR linkage identifies this same task; otherwise suffix all three proposed branch names with the newly created issue number and preserve their parent relationships.
  • Initially use one scoped commit per PR, with the PR title minus its [n/3] prefix as the commit subject. Selectively stage only that layer's files/hunks. Do not build the whole feature on one branch and open three PRs with the same cumulative base.
  • Push each branch to origin and create a draft PR with gh pr create --repo Shopify/ghostferry --draft --base <base-from-above> --head <head-from-above> --title <exact-title-above> --body-file -. Each body contains Refs #<tracking-issue>, the previous PR dependency (or main for PR 1), the next PR when available, that layer's behavioral scope and the checks actually exercised. Update all issue/PR links after the three URLs are known; do not use Closes/Fixes on an intermediate PR.
  • Every PR diff is relative to its immediate parent, and every branch must build and pass its specified checks independently of children. Review changes to a parent require restacking its descendants onto the updated parent. Rewrite only branches created for this stack, publish rewrites with --force-with-lease, and never use bare --force or rewrite user-owned branches.
  • Do not merge, enable auto-merge, push to main, create release tags or close the issue. Record merge order as PR 1 → PR 2 → PR 3 for reviewers. When a parent is eventually merged, its child must be retargeted/restacked onto main so the child diff excludes already-landed commits; keep the next layer based on that updated child. This remains ordinary stacked PR maintenance, not a new tooling dependency.
  • Before handoff, confirm the issue contains the complete plan and links all three PRs, each PR has the intended base and only its layer's diff, and the issue links the verification evidence. Inspect actual Go/Ruby matrix results, not just aggregate CI status, because those jobs allow failures at workflow level. Return the issue URL and the ordered three PR URLs. Report unpublished artifacts or failed/unrun checks explicitly rather than marking the stack complete.

Approach

Implement gradual recovery at batch admission, not a new writer or table queue. Keep the existing fixed table-worker pool and concurrent binlog replay. The controller is shared by all cursors in one DataIterator.Run; it never holds source locks while waiting. Follow the stack ownership/order above: PR 1 completes the existing-value synchronization substeps of steps 3–4; PR 2 implements the remaining steps 1–4 together; PR 3 adds subprocess/cutover regression coverage. None requires changing write/stream scheduling.

1. Define bounded recovery and preserve hard-stop precedence

  • In config.go, add top-level DataIterationThrottleRecoveryConfig with exactly the exported fields Disabled bool and RecoveryInterval string, plus Validate() error. Add Config.DataIterationThrottleRecovery DataIterationThrottleRecoveryConfig next to DataIterationConcurrency, and DataIterator.ThrottleRecovery DataIterationThrottleRecoveryConfig. Empty interval normalizes to "5s"; malformed/nonpositive durations return DataIterationThrottleRecovery.RecoveryInterval must be a positive duration: %q with the supplied string. Recovery is enabled by default; Disabled: true retains immediate recovery for workload comparison/rollback. In Config.ValidateConfig, validate this after the existing zero-to-4 concurrency default; reject nonpositive effective concurrency with DataIterationConcurrency must be greater than zero. The controller constructor repeats the same validation for directly constructed iterators.
  • Cold starts without an observed throttle retain current maximum concurrency/full configured batches. On an observed hard throttle, stop new copy permits; in-flight batches finish normally. Once clear, allow one copy batch at a time, with effective rows max(1, configuredRows / 8) using integer division. A table may get repeated turns, but FIFO batch admission prevents one table starving other assigned tables.
  • After each healthy recovery interval, reduce the divisor 8 → 4 → 2 → 1; only then raise the concurrency limit by exactly one per interval until DataIterationConcurrency. Never raise both controls in one transition. This yields default batch sizes 25 → 50 → 100 → 200 at concurrency 1, then concurrency 2 → 3 → 4 for existing defaults.
  • A healthy interval starts when the first batch is admitted at that stage, requires at least one successfully completed nonempty batch from that stage and continuous observed !Throttled(). A batch failure resets the interval without increasing either control. No-work time cannot cause multiple catch-up increases. Another hard throttle resets recovery to one slot/divisor 8.
  • Every increase starts a new stage generation/window, with no carried-over success count or accumulated intervals. Failure resets affect recovery only; a transient read retry during an otherwise unthrottled cold-start run does not activate recovery. In-flight work is never preempted, including work admitted before a pause or disable transition.
  • Throttler.Disabled() takes precedence over hard pause and recovery: immediately restore configured limits at the next admission/poll, clear recovery history, and allow cutover/standalone copy to proceed. When re-enabled, use normal limits unless currently hard-throttled; do not replay an old recovery timer. DataIterationThrottleRecovery.Disabled disables only the new limiter, never the hard throttle.
  • Do not change Throttler, WaitForThrottle, the binlog writer's hard-stop call, manual pause API, lag threshold, or binlog coordinate semantics. This is recovery control, not a predictor of future overload.

2. Implement a shared, fair admission controller

  • New implementation path: copy_throttle.go; no equivalent limiter exists in the repository. Use an unexported copyThrottleController with a mutex, condition variable, monotonically increasing FIFO tickets, in-flight count, stage generation, concurrency limit, batch divisor and stability timestamps/counters. Return value-type permits rather than allocating a channel/closure per batch.

  • Concrete private API: newCopyThrottleController(throttler Throttler, maxConcurrency int, config DataIterationThrottleRecoveryConfig) (*copyThrottleController, error), acquire() (copyThrottlePermit, error), release(permit copyThrottlePermit, copied bool, err error), run(ctx context.Context), and observe(now time.Time). copyThrottlePermit carries the stage generation and divisor. All completions release capacity; only a successful nonempty batch counts as healthy progress. An empty terminal fetch is neutral; an error resets the stability window and advances its generation. Successful completions admitted under older generations do not count toward growth. Keep an internal now func() time.Time, defaulting to time.Now, so controller tests can advance time without sleeps or a new clock package.

  • Recheck hard throttle/disable on every acquisition and release, and from one reusable 500 ms ticker. Observe the signal once before starting workers. Wake waiters on releases/state transitions and when ticket advancement leaves spare capacity. Admission checks ticket order and inFlight < limit; lowering the limit never pretends existing permits were revoked. The ticker is the only timed waiter; no per-worker polling loops in the managed copy path. Retain the existing WaitForThrottle path when recovery is disabled or no controller is attached. Cancellation marks the controller closed and broadcasts; queued/future acquisitions return context.Canceled, never a permit. WithRetries already returns that error without retrying (utils.go:40-42).

  • Construct the controller as a local in DataIterator.Run before starting table workers; assign it to a new private Cursor.copyThrottle field after NewCursor. Start/stop/join its sampler inside that run, not Ferry.Run, so RunStandaloneDataCopy also works. Use a run-local cancellation context and deferred cancel/join for early returns; stop normally after all workers finish and before done listeners. Constructor errors go to the existing data-iterator fatal handler and return immediately. Nil throttler or disabled recovery needs no controller. A direct DataIterator with zero-value recovery config gets the same defaults as Ferry. Recovery state is not serialized; restart follows the same cold-start/observed-throttle rules.

  • Ferry.NewDataIterator copies the new config into the iterator; NewDataIteratorWithoutStateTracker inherits it. Do not attach the controller to IterativeVerifier's independent cursors or BinlogWriter.

  • The constructor rejects maxConcurrency <= 0; it does not silently normalize it. Only Config.ValidateConfig applies the existing zero-to-4 default before constructing an iterator. This avoids creating a four-slot limiter alongside a directly constructed zero-worker iterator.

3. Admit before source transactions and size each admitted batch

  • In Cursor.Each, acquire inside each source-read retry attempt, before DB.Begin or any source SELECT. Read the configured batch size after acquisition; apply that permit's divisor to the configured base. No shared maximum is overwritten by recovery.
  • Put the scaling formula in func (p copyThrottlePermit) batchSize(configuredRows uint64) uint64 and use it from Each; controller tests exercise that same function instead of duplicating the formula. Issued permit divisors are always one of 8, 4, 2 or 1. Keep CursorConfig.GetBatchSize returning the unscaled base for other existing consumers.
  • Split the query body into private fetch(db SqlPreparer, batchSize uint64) (*RowBatch, PaginationKey, error). Preserve exported Fetch(db SqlPreparer) (*RowBatch, PaginationKey, error) as its existing full-configured-size entry point; Each calls the private helper with its permit-specific size. LSP found only Cursor.Each calling Fetch in this repository. Preserve BuildSelect, column/fingerprint handling, row-lock SQL and pagination behavior.
  • Hold a permit through source fetch, synchronous listeners, target retries/commit and source rollback; release only after rollback. Release once on begin/fetch failure before a read retry sleeps, empty terminal fetch, invalid pagination, callback failure, or success. Mark success only for a nonempty batch whose callback completed. Never release/reacquire inside BatchWriter.WriteRowBatch; that would wait while holding source locks.
  • Preserve per-table override precedence in CursorConfig.GetBatchSize; scale whichever base it returns, including tiny bases down to one row. An operator's live batch-size reduction takes effect at the next admission, without resetting recovery or bypassing its divisor.
  • Make the existing live maximum race-free without changing its type/wire shape: atomic store in Config.Update, atomic load in CursorConfig.GetBatchSize, and encode a freshly constructed UpdatableConfig snapshot with an atomic load in ControlServer.HandleConfigGet. Startup-only assignments/defaults remain ordinary writes before workers start. Do not introduce a second mutable maximum or change /api/config fields.
  • Keep existing progress fields unchanged: TableProgress.BatchSize is the configured maximum and ActiveDataIterators counts started, incomplete tables, not current permits (ferry.go:1067-1095). Do not use either as proof of actual recovery concurrency; observe batch admission/transactions in verification. No new status schema or writer telemetry is required for this change.

4. Require fresh built-in lag evidence before increasing load

  • Protect LagThrottler.lag and new sample metadata with one mutex; Throttled reads under that mutex. Record a monotonically increasing sample sequence and observation time only after a successful non-NULL query result. Existing missing-row/NULL semantics still retain the last lag; existing SQL errors/retries/fatal handling remain unchanged. The freshness calculation requires a positive polling interval: after parsing in NewLagThrottler, reject interval <= 0 with invalid UpdateInterval: must be greater than zero rather than accepting a busy loop or impossible freshness window.
  • Add private LagThrottler.recoverySample() (sequence uint64, observedAt time.Time, interval time.Duration). The controller uses this optional internal capability when present. A stage increase also requires a sequence newer than the one at stage admission and a sample no older than twice the lag update interval. Missing/stale samples freeze growth at the current stage; they do not clear a hard stop or invent a new global stop. Generic custom throttlers and PauserThrottler use the existing boolean plus elapsed time/successful-copy evidence because their interface has no freshness contract.
  • Keep stability based on this external safety signal, not BinlogWriterLag: pausing the writer based on its own lag would create a feedback deadlock, and idle event timestamps are not an authoritative backlog measurement.

Critical files & anchors

  • sharding/sharding.go, ShardingFerry.Run:145-182: cutover disables throttling around binlog drain and standalone delta/primary-table copy; recovery must not delay this path.
  • testhelpers/test_ferry.go, TestFerry.Start:84-115: reuse copy listeners for barriers. Its misleadingly named binlog-apply listeners run on the streamer, not after target commit; verify replay by querying the target.
  • testhelpers/integration_test_case.go, CopyData/StartFerryAndDataWriter/Teardown: existing lifecycle and checksum verification; it drops gftest* and toggles source read-only, so only run against disposable databases.

Verification

Planning is read-only: no issue, branch, PR, migration, test suite or benchmark was created/executed. The reported binary-throttle problem is accepted; do not rerun a baseline merely to reconfirm it. Apply the per-layer gates in the stack section, then verify the complete stack before publishing the final evidence.

Prerequisites and commands

Run from the repository root after implementation. go.mod requires Go 1.27.2. Use disposable MySQL services from docker-compose_8.0.yml: source 29291, target 29292, root with empty password, ROW binlogs and GTID enabled. N1_PORT/N2_PORT override Go fixture ports. Ruby checks require the repository Gemfile dependencies already installed. Never point these commands at production or shared development databases.

Start the disposable fixture only after execution is approved: docker compose -f docker-compose_8.0.yml up -d. Wait for both docker compose -f docker-compose_8.0.yml exec -T mysql-1 mysqladmin ping -uroot and the equivalent mysql-2 command to succeed before tests. Set MYSQL_VERSION=8.0 for expected MySQL error handling in existing lag tests.

  1. Pure controller tests: go test -race . -run '^TestCopyThrottle' -count=1.
  2. Focused real-MySQL coverage: MYSQL_VERSION=8.0 go test ./test/go -run 'TestCopyThrottle|TestDataIterator|TestThrottler|TestNewThrottler|TestSelectUpdateBinlogCopy|TestUpdateBinlogSelectCopy|TestSelectCopyUpdateBinlog|TestOnlyDeleteRowWithMaxPaginationKey' -count=1 -p=1.
  3. Race-check the newly changed boundaries: MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestCopyThrottle|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs' -count=1 -p=1.
  4. Existing Go consumers: MYSQL_VERSION=8.0 go test . ./test/go ./copydb/test ./sharding/test -count=1 -p=1.
  5. Repeat command 2 with GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid MYSQL_VERSION=8.0; the default run uses file/position coordinates.
  6. Existing interrupt/resume executable coverage: MYSQL_VERSION=8.0 bundle exec rake test TEST=test/integration/interrupt_resume_test.rb; repeat with GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid.

Permanent behavior coverage

Add copy_throttle_test.go in package ghostferry for the private controller, matching existing root test conventions. Name tests with prefix TestCopyThrottle. Use a fake monotonic clock and channel barriers rather than wall-clock sleeps for progression/order assertions.

  • Four contenders, configured max 4/base 200, hard throttle observed then cleared: one admitted batch with 25 rows; no increase before the healthy interval; successive healthy stages admit limits/sizes (1,50), (1,100), (1,200), (2,200), (3,200), (4,200). Assert admission behavior, not just copied fields.
  • A new throttle while four batches are in flight allows those batches to finish but no new permit; clearing it cannot admit the first recovery batch until the outstanding count falls below one. Old-generation completions cannot jump stages. Repeated flapping always restarts at one/eighth-size.
  • No completed work, failed work, stale/unchanged lag samples, NULL/no rows and old-generation successes cannot produce growth. Fresh observations and successful work resume growth. Empty terminal fetch releases capacity without resetting healthy progress. Include RecoveryInterval/lag polling interval zero, negative and malformed input boundaries.
  • FIFO progress across assigned tables, tiny configured sizes (1 and 7 remain at least one), unequal per-table overrides, re-throttling during a fetch retry, and cleanup after begin/fetch/callback errors. Cancellation releases queued waiters with context.Canceled; it must not start SQL or leak the sampler.
  • SetDisabled(true) bypasses paused/recovering state without waiting for the stability interval. Re-enabling does not resurrect old timers. Disabled recovery leaves hard throttling intact. Empty/completed table sets and a single-table migration finish normally.

Extend test/go/data_iterator_test.go and test/go/race_conditions_integration_test.go with TestCopyThrottle... coverage using real queries and existing TestFerry/IntegrationTestCase lifecycle:

  • Pause four already-started table cursors and let previously admitted batches drain before clearing the pause. Hold the first resumed batch's listener and assert no second copy callback enters. Issue an UPDATE against a different waiting table through a separate source connection with innodb_lock_wait_timeout=1: it must complete, proving the blocked cursor did not acquire source row locks before its permit.
  • While that first copy callback is held, insert a source row beyond the captured maximum pagination key. Once the hard pause is clear, query the target until that row is committed: binlog replay must remain independent of copy recovery.
  • Update the configured base to 104 during first-stage recovery and observe a subsequent batch of 13 rows; an override of 80 still yields 10. Assert exact source/target equality after copy/drain, not just counts or callbacks.
  • Drive a real LagThrottler with the meta.lag_table fixture used in lag_throttler_test.go: above threshold stops both paths, fresh below-threshold samples allow staged copy recovery, deleting all lag rows does not invent new healthy samples. Keep these DB-mutating tests serial and join background goroutines before teardown.
  • Add test_interrupt_resume_during_throttle_recovery in test/integration/interrupt_resume_test.rb. Reuse its existing single-table seed and signal/state harness. Add the small executable test/lib/go/throttle_recovery_ghostferry/main.go using the same tf.Setup/tf.Run pattern as minimal_ghostferry/main.go: in RunCallbacks.BeforeInitialize, set concurrency 1, recovery interval "30s", and enable the existing control server on 127.0.0.1:18000. No existing integration executable enables this pause surface; reuse integrationferry rather than expanding its environment-variable config protocol. In Ruby, use Net::HTTP to POST /api/pause from the first BEFORE_ROW_COPY callback. From its first AFTER_ROW_COPY, start a joined thread that waits one second then POSTs /api/unpause; returning from that callback lets the permit release observe the pause. Pause again from the second BEFORE_ROW_COPY, then call term_and_wait_for_exit from the second AFTER_ROW_COPY so no third batch can enter. The interrupted target contains 225 rows (200 initial + 25 recovered); its max pagination key matches the state dump. Resume with the ordinary minimal executable and assert_test_table_is_identical. Run serially; join the unpause thread and propagate request failures. No recovery fields enter serialized state. Keep existing race-order and sharding standalone-copy cases passing.
    • The unpause thread is joined after the interrupted subprocess exits, not inside AFTER_ROW_COPY; the callback must return while the hard pause is still set so release observes it. Set the test executable's configured base explicitly to 200, rather than making the 225-row correctness assertion depend on an incidental default.

Real-program smoke, beyond tests

Create a throwaway Go executable at /tmp/ghostferry-recovery-smoke/main.go, run with MYSQL_VERSION=8.0 go run /tmp/ghostferry-recovery-smoke/main.go from the root, and capture its output. Use the real public Ferry, SQL connections and testhelpers.NewTestFerry/SeedInitialData, not mocked readers/writers. Fixture/setup contract:

  1. Recreate only gftest on disposable source/target. Seed source tables t1–t4 with 10,000 rows each and matching empty target tables. Configure concurrency 4, base 200, recovery interval "30s" so assertions fit within the first stage, and a PauserThrottler already paused. Enable the control server at 127.0.0.1:18000. Attach the first-copy listener barrier before Start; initialize/start/run the real ferry and wait until all four TargetPaginationKeys entries exist.
  2. Insert a sentinel source row with ID 10001 after those maxima are captured. POST /api/unpause. The first copy callback must report 25 rows. Hold it; within five seconds, the target must contain the sentinel while no second copy callback has entered. Query the target for evidence rather than treating streamer callbacks as commits.
  3. POST /api/config with {"DataIterationBatchSize":104} and GET it concurrently using Content-Type: application/json; the configured value must be 104. Release the first callback and observe a 13-row subsequent copy batch before the 30-second stage expires. Print actual table/batch-size/admission events. A missed deadline is a smoke failure, not a reason to relax the assertion.
  4. Let copying finish; call WaitUntilRowCopyIsComplete, FlushBinlogAndStopStreaming, join Run, then StopTargetVerifier. Use ChecksumTableVerifier.VerifyDuringCutover plus explicit ordered source/target row comparison for all four tables. The sentinel must occur exactly once. Repeat the executable with GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid.

The smoke must prove real reduced-batch admission, independent binlog progress, live batch-size composition and final data integrity; deterministic controller tests must prove the complete ramp without waiting through default five-second stages. Do not claim a throughput improvement from this scripted pause or from unit tests.

Assumptions & contingencies

  • User decision: preserve the hard lag stop for both copy and binlog work, including operator pause and cutover-disable semantics.
  • User decision: limit this change to gradual copy throttling. No interleaving assessment, arbitration prototype or write/stream scheduling change is required; retain only the binlog-progress checks that ensure copy recovery cannot stall replay beyond the existing hard stop.
  • Recommended rollout choice: enable recovery by default only after an observed throttle; retain current unthrottled startup speed. Set DataIterationThrottleRecovery.Disabled to true to opt out if workload evaluation rejects that choice. Five-second stages and the initial one-eighth batch size are conservative starting values, not measured optima; RecoveryInterval is the supported deployment tuning knob.
  • A custom throttler may provide only boolean health. The fixed fallback is already specified in step 4: preserve its contract and use elapsed healthy time plus completed work, without introducing a new load query or replacing it with LagThrottler.
  • No production throughput measurements were collected. If load-dependent throttling persists after recovery is deployed, retain the hard stop and report the measured bottleneck; do not weaken lag protection or claim the scheduling change solved a database-capacity limit.

Activity

  1. driv3r commented on Oct 30, 2026

    @driv3r
    ContributorAuthor

    Verification evidence (stack tip throttle-recovery/03-resume-coverage)

    Environment: macOS arm64, Go 1.27.2, and disposable MySQL 8.0.32 source and target from docker-compose_8.0.yml on ports 29291/29292 (ROW binlogs, GTID on).

    Stack: #472 → #473 → #474

    Verification commands

    # Command Result
    1 go test -race . -run '^TestCopyThrottle' -count=1 pass. Also passed with -count=30, and with -count=20 -cpu 1,2,8.
    2 MYSQL_VERSION=8.0 go test ./test/go -run 'TestCopyThrottle|TestDataIterator|TestThrottler|TestNewThrottler|TestSelectUpdateBinlogCopy|TestUpdateBinlogSelectCopy|TestSelectCopyUpdateBinlog|TestOnlyDeleteRowWithMaxPaginationKey' -count=1 -p=1 pass
    3 MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestCopyThrottle|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs' -count=1 -p=1 pass
    4 MYSQL_VERSION=8.0 go test . ./test/go ./copydb/test ./sharding/test -count=1 -p=1 pass
    5 command 2 with GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid pass at the tip. One earlier run on #473 failed in TestThrottlerMultipleServerIDs; see the existing flake below.
    6 MYSQL_VERSION=8.0 bundle exec rake test TEST=test/integration/interrupt_resume_test.rb, file/position and GTID 27/27 pass in both modes, including test_interrupt_resume_during_throttle_recovery. Some runs hung on an existing test; see below.
    #474 go test ./test/go -run '^TestCopyThrottleStandaloneCopyBypassesPause$' and go test ./sharding/test -run 'TestJoinedTablesTestSuite|TestPrimaryKeyTableTestSuite' pass
    #472 go test -race ./test/go -run 'TestConcurrentBatchSizeUpdates|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs|TestDataIterator/TestBatchSizeUpdate' pass. With the fix reverted, the new test reports 3 DATA RACEs.

    Mutation check: if admission moves after Begin + fetch, TestCopyThrottleResumesOneSmallBatchWithoutSourceLocks fails with Error 1205 (HY000): Lock wait timeout exceeded on the UPDATE of a waiting table.

    Existing flakes (not changed by this stack)

    • TestThrottlerThrottlesAndUnthrottles / TestThrottlerMultipleServerIDs: these rely on a 10 ms sleep against a 5 ms lag poll. In -count=100 loops, unmodified origin/main failed 3 of 200 runs, and [1/3] Make throttle and batch-size access race-free #472's tip also failed sometimes.
    • test_interrupt_resume_inline_verifier_will_verify_entries_in_reverify_store can hang the Ruby suite:
      • The test deletes MIN(id) + 1, which can be one of the 140 random holes in the seeded table. The DELETE then writes no binlog event, so AFTER_BINLOG_APPLY never fires.
      • Meanwhile the third AFTER_ROW_COPY handler runs sleep forever, so WEBrick shutdown blocks after Ghostferry's 30 s status timeout.
      • Captured hang: 3 copy batches, then no source binlog events for 30 s, then Client.Timeout exceeded while awaiting headers, then going to shutdown ..., and then the hang.
      • The suite hung in 3 of 11 local runs across [2/3] Recover copy load gradually after throttling #473 and the stack tip. One of those hangs was captured and traced to this test; the other two were not captured. It did not reproduce in 4 runs of origin/main, but the trigger is in the seeded data, not in the copy throttle.

    Real-program smoke, beyond tests

    The smoke is a throwaway /tmp/ghostferry-recovery-smoke/main.go, run with MYSQL_VERSION=8.0 go run /tmp/ghostferry-recovery-smoke/main.go from the repository root. It uses the public Ferry, real SQL connections, testhelpers.NewTestFerry-style config and SeedInitialData:

    • Setup: tables gftest.t1–t4 with 10,000 rows each, concurrency 4, base batch size 200, RecoveryInterval: "30s", a PauserThrottler paused before start, and the control server on 127.0.0.1:18000.
    • After the maxima are captured, it inserts sentinel t1.id=10001 and sends POST /api/unpause.
    • It holds the first batch, then runs POST /api/config {"DataIterationBatchSize":104} and GET concurrently.
    • It releases the batch, drains, and verifies with ChecksumTableVerifier.VerifyDuringCutover plus an ordered row comparison.

    File/position:

    [  0.00s] binlog coordinate mode: file_position
    [ 12.94s] seeded 4 tables with 10000 rows each
    [ 12.98s] captured max pagination key gftest.t1 = 10000
    [ 12.98s] captured max pagination key gftest.t2 = 10000
    [ 12.98s] captured max pagination key gftest.t3 = 10000
    [ 12.98s] captured max pagination key gftest.t4 = 10000
    [ 12.98s] inserted sentinel gftest.t1 id=10001 while paused
    [ 12.98s] POST /api/unpause -> 200
    [ 13.47s] copy batch #1 table=t4 rows=25 (held)
    [ 13.48s] sentinel replayed to target while copy batch #1 is held; copy callbacks entered: 1
    [ 13.48s] POST /api/config DataIterationBatchSize=104 -> 200; concurrent GET -> 200; GET after POST -> 104
    [ 13.49s] copy batch #2 table=t1 rows=13 after 0.02s of recovery
    [ 13.50s] copy batch #3 table=t3 rows=13
    [ 13.51s] copy batch #4 table=t2 rows=13
    [ 13.52s] copy batch #5 table=t4 rows=13
    [ 13.52s] copy batch #6 table=t1 rows=13
    [ 13.52s] copy batch #7 table=t3 rows=13
    [ 13.53s] copy batch #8 table=t2 rows=13
    [ 25.19s] row copy complete after 3079 copy batches
    [ 25.71s] ChecksumTableVerifier: DataCorrect=true 
    [ 25.71s] gftest.t1: source and target ordered rows identical (10001 rows)
    [ 25.72s] gftest.t2: source and target ordered rows identical (10000 rows)
    [ 25.73s] gftest.t3: source and target ordered rows identical (10000 rows)
    [ 25.74s] gftest.t4: source and target ordered rows identical (10000 rows)
    [ 25.74s] sentinel present exactly once on target
    [ 25.74s] SMOKE OK
    

    GTID (GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid):

    [  0.00s] binlog coordinate mode: gtid
    [ 13.13s] seeded 4 tables with 10000 rows each
    [ 13.16s] captured max pagination key gftest.t3 = 10000
    [ 13.16s] captured max pagination key gftest.t1 = 10000
    [ 13.16s] captured max pagination key gftest.t4 = 10000
    [ 13.16s] captured max pagination key gftest.t2 = 10000
    [ 13.16s] inserted sentinel gftest.t1 id=10001 while paused
    [ 13.16s] POST /api/unpause -> 200
    [ 13.66s] copy batch #1 table=t4 rows=25 (held)
    [ 13.67s] sentinel replayed to target while copy batch #1 is held; copy callbacks entered: 1
    [ 13.67s] POST /api/config DataIterationBatchSize=104 -> 200; concurrent GET -> 200; GET after POST -> 104
    [ 13.68s] copy batch #2 table=t3 rows=13 after 0.02s of recovery
    [ 13.69s] copy batch #3 table=t1 rows=13
    [ 13.69s] copy batch #4 table=t2 rows=13
    [ 13.69s] copy batch #5 table=t4 rows=13
    [ 13.70s] copy batch #6 table=t3 rows=13
    [ 13.70s] copy batch #7 table=t1 rows=13
    [ 13.70s] copy batch #8 table=t2 rows=13
    [ 34.47s] row copy complete after 3079 copy batches
    [ 34.98s] ChecksumTableVerifier: DataCorrect=true 
    [ 34.98s] gftest.t1: source and target ordered rows identical (10001 rows)
    [ 34.99s] gftest.t2: source and target ordered rows identical (10000 rows)
    [ 35.00s] gftest.t3: source and target ordered rows identical (10000 rows)
    [ 35.01s] gftest.t4: source and target ordered rows identical (10000 rows)
    [ 35.01s] sentinel present exactly once on target
    [ 35.01s] SMOKE OK
    

    The whole copy finished inside the first 30 s stage, so it ran as one 13-row batch at a time. The deterministic controller tests in #473 cover the full ramp. No throughput improvement is claimed from this scripted pause or from the unit tests.

    #472 live batch-size smoke (concurrency 1, base 200, no pause):

    batch 1: 200 rows (held)
    POST /api/config DataIterationBatchSize=104 -> 200
    GET /api/config -> {"DataIterationBatchSize":104}
    batch 2: 104 rows
    copied 1000 rows in 9 batches
    source/target ordered rows identical: 1000 rows
    SMOKE OK
    

    CI matrix

    The Go and Ruby jobs use continue-on-error, so the aggregate check proves nothing. The outcomes below come from each job's Running Golang tests / Running Ruby tests step.

    PR go-test (5.7 fp, 8.0 fp, 8.0 gtid, 8.4 fp, 8.4 gtid) ruby-test (5.7 fp, 8.0 fp, 8.0 gtid, 8.4 fp, 8.4 gtid, 8.4 zerolog) gh-285 (5.7, 8.0, 8.4)
    #472 all success all success; 5.7 hit the 15-minute job timeout on attempt 1 and passed on rerun all success
    #473 all success all success; 5.7 hit the 15-minute timeout on attempts 1 and 2 inside InterruptResumeTest and passed on attempt 3 all success
    #474 all success all success on the first attempt all success

    The CI logs confirm the new Go tests ran, for example PASS test/go.TestCopyThrottleResumesOneSmallBatchWithoutSourceLocks and --- PASS: TestCopyThrottleRecoversInStages.

    The cancelled jobs printed no captured test output, so the hanging test cannot be identified from CI. For comparison, ruby-test jobs on other branches and main were cancelled the same way in about 10% of the last ~35 runs per matrix entry (for example, 5.7: 4 of 35). These timeouts predate this stack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions