Repository navigation
Description
Gradual copy throttle recovery
Context
Ghostferry currently stalls all copying under throttle and resumes all tables at full speed. The requested improvement is gradual recovery: resume one table with smaller batches, then increase concurrency after stability. Implement this through copy-batch admission while preserving the existing hard safety stop and independent binlog replay. During implementation, create one GitHub issue tracking the complete plan and deliver the work as a dependency-ordered stack of pull requests.
Findings
-
throttler.go:12-34:Throttlerexposes a boolean;WaitForThrottlepolls every 500 ms and releases all waiters when unthrottled or disabled. -
throttler.go:73-164:LagThrottlercombines operator pause withlag > MaxLag; its default update interval is 1 second. Missing/NULL lag samples retain the previous value. -
data_iterator.go:62-165: a fixedConcurrencyworker pool consumes whole tables; each worker callscursor.Eachand synchronous batch listeners until its table is complete. -
binlog_writer.go:33-91: a bounded event channel feeds one writer, andBufferBinlogEventssends synchronously into it. Making the writer wait on copy work would backpressure the streamer; the new recovery limiter must therefore govern copy admission only, leaving the writer's existing hard-throttle check unchanged. -
cursor.go:98-155: the source transaction remains open across fetch and synchronous target-write listeners. Admission must occur beforeDB.Begin; release must occur after source rollback, including failure/empty-batch paths. FeedingtablesQueuemore slowly cannot constrain already-running table cursors. -
batch_writer.go:50-137: target retries and inline verification run while the source transaction remains open; copy pagination advances only after target commit. Do not insert scheduling waits inside this callback. -
ferry.go:794-823: binlog writer, binlog streamer and data iterator run independently.RunStandaloneDataCopyalso constructs a data iterator and runs during cutover without its own supporting-services loop. -
row_batch.go:62-74: copy usesINSERT IGNORE, not an unconditional overwrite. Preserve this and source-share-lock protection when changing admission. -
config.go:975-1005,cursor.go:66-73,control_server.go:370-392: live batch-size updates currently share a plainuint64; all runtime reads/writes must use atomic access before recovery composes with those updates. Per-table overrides remain the configured base. -
Local
findand literal searches found no existing semaphore, adaptive limiter or injectable clock abstraction. The small recovery controller below is new code rather than a second scheduler convention. -
Publication discovery:
originishttps://github.com/Shopify/ghostferry.git, the default branch ismain, issues are enabled, and the current GitHub identity reportsADMIN. The cleaninvestigate-throttling-improvementscheckout andorigin/mainboth pointed to6f4289c7ca6ec1b36481aee7d70b23d5da928126during planning.gh2.103.0 is installed; an all-state issue search found no existing issue titledGradual copy throttle recovery. Root/.githubsearches found no required stack tool or branch-naming convention; use ordinary Git branches and explicit GitHub PR bases. -
.github/workflows/tests.yml:3-8runs CI for PRs without a base-branch restriction, so dependent PR bases are supported. Its Go/Ruby matrix jobs usecontinue-on-error: true; an overall green workflow is not proof those tests passed.README.md:215-218documents prerelease creation on pushes tomain; this task publishes reviewable branches/PRs, not merges or releases.
Tracking issue and PR stack
Create the tracking issue before implementation
- After implementation is approved, create one issue in
Shopify/ghostferrytitledGradual copy throttle recovery. Its body is the complete approved contents oflocal://adaptive-copy-scheduling-plan.md, including the implementation specification, stack breakdown and verification—not a summary or a link to an inaccessible local file. Usegh issue create --repo Shopify/ghostferry --title "Gradual copy throttle recovery" --body-file -, supplying the Markdown on stdin. - Record the returned issue number/URL immediately in this canonical plan's tracking entry and use that issue for all three PRs. Do not invent an issue number. On an interrupted/ambiguous publication attempt, query the exact title and inspect the body before retrying; reuse the issue created for this plan rather than creating a duplicate. No labels, assignees or milestone are requested.
- Keep this checklist in the issue; replace each PR label with its actual URL after publication. Check a PR item only after that PR is merged into
main; evidence can be linked while review is pending. Keep the issue open at handoff—publishing the stack is not permission to merge it or close the issue.- PR 1 merged: race-free throttle and batch-size inputs — [1/3] Make throttle and batch-size access race-free #472
- PR 2 merged: complete gradual copy recovery — [2/3] Recover copy load gradually after throttling #473
- PR 3 merged: interruption/resume and standalone-cutover coverage — [3/3] Verify throttle recovery across interruption and cutover #474
- End-to-end verification evidence attached and all acceptance scenarios passing — Gradual copy throttle recovery #471 (comment) (local MySQL 8.0 + CI matrix; pre-existing Ruby-suite hangs noted there)
PR 1 — Make shared throttle inputs race-free
- Title:
[1/3] Make throttle and batch-size access race-free. - Head:
throttle-recovery/01-safe-inputs; base:main. - Implement only the existing-value synchronization portions of Approach steps 3 and 4: atomic batch-size store/load and HTTP config snapshot in
config.go,cursor.goandcontrol_server.go; protect the existingLagThrottler.lagreads/writes with a mutex inthrottler.go. Preserve current throttle, polling, batch-size and recovery behavior. Do not add unused recovery configuration, sample metadata, controller methods or scheduling code to this PR. - Add
TestConcurrentBatchSizeUpdatesin the existingtest/go/control_server_test.go(currently only its package declaration). Exercise the real config GET/POST handlers withhttptest, concurrent updates alternating 104/200 and concurrentCursorConfig.GetBatchSizereads. Assert every observed base is one of the configured values, an explicit per-table override stays 80, and a final completed update to 104 is visible through both the HTTP snapshot and cursor getter. Coordinate goroutines with a barrier/WaitGroup; no sleeps. The race detector is part of this concurrency regression's acceptance. - Gate:
MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestConcurrentBatchSizeUpdates|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs|TestDataIterator/TestBatchSizeUpdate' -count=1 -p=1. Also run a real-Ferry smoke using the fixture/lifecycle in Verification with concurrency 1, base 200, no initial pause and no new recovery config: hold its first 200-row callback, POST base 104, release, observe the next batch contains 104 rows, then drain and compare source/target rows. This verifies the live update path without waiting for PR 2.
PR 2 — Implement complete gradual copy recovery
- Title:
[2/3] Recover copy load gradually after throttling. - Head:
throttle-recovery/02-gradual-recovery; base:throttle-recovery/01-safe-inputs. - Implement all remaining behavior in Approach steps 1–4 as one end-to-end feature: validated recovery config, FIFO permits and staged limits, cursor admission before source transactions, scaled batches, Ferry/standalone iterator lifecycle, and fresh lag observations. Reuse PR 1's synchronization; do not copy or reintroduce its changes. Production changes are in
config.go,copy_throttle.go(new),data_iterator.go,cursor.go,ferry.goandthrottler.go. Do not change binlog scheduling. - Include the root
copy_throttle_test.gocontroller coverage, config/lag validation boundary tests intest/go/config_test.goandtest/go/lag_throttler_test.go, and the real-MySQL recovery, source-lock, independent-replay and live-batch-size cases intest/go/data_iterator_test.go/test/go/race_conditions_integration_test.go. Core behavior and its regression tests land together; this PR must not depend on PR 3 for the recovery algorithm to work or for its primary safety tests. - Gate: Verification commands 1–5 and the full real-program smoke in both coordinate modes. Keep the existing interrupt/resume suite passing; the new recovery-specific subprocess case belongs to PR 3. Publish no intermediate unused API, partial controller or disabled scaffold.
PR 3 — Verify interruption and standalone cutover
- Title:
[3/3] Verify throttle recovery across interruption and cutover. - Head:
throttle-recovery/03-resume-coverage; base:throttle-recovery/02-gradual-recovery. - Add the recovery-specific Ruby interrupt/resume regression and
test/lib/go/throttle_recovery_ghostferry/main.goexecutable exactly as specified in Verification. AddTestCopyThrottleStandaloneCopyBypassesPauseintest/go/data_iterator_test.go: with nonempty source tables, empty targets,SetPaused(true)andSetDisabled(true), run the actualRunStandaloneDataCopywithout aFerry.Runsupporting-services loop; it must finish and preserve exact row contents. This catches a recovery sampler/bypass deadlock in the cutover-only path rather than merely checking a copied flag. - This PR contains regression coverage, not another scheduling change. If these cases expose a production defect, place its fix and directly associated tests in PR 2, then restack PR 3; do not hide the feature fix in the verification layer.
- Gate: Verification command 6 in both coordinate modes;
MYSQL_VERSION=8.0 go test ./test/go -run '^TestCopyThrottleStandaloneCopyBypassesPause$' -count=1 -p=1; andMYSQL_VERSION=8.0 go test ./sharding/test -run 'TestJoinedTablesTestSuite|TestPrimaryKeyTableTestSuite' -count=1 -p=1. Run the complete Verification command set at the stack tip before handoff, recording actual outcomes and the real-program smoke evidence on the issue.
Branch, publication and review procedure
- Build the three layers bottom-up. After approval, fetch
origin, create PR 1's branch fromorigin/main, commit its scoped changes, create PR 2's branch from PR 1's tip, then create PR 3's branch from PR 2's tip. Use the exact head/base names above. Preserve the existing investigation branch and unrelated working-tree changes; if the checkout becomes dirty with unrelated work, use a separate worktree at/tmp/ghostferry-throttle-recoveryinstead of resetting or stashing it. If that worktree path is occupied, use the lowest free numeric suffix. Reuse an existing stack branch only when its issue/PR linkage identifies this same task; otherwise suffix all three proposed branch names with the newly created issue number and preserve their parent relationships. - Initially use one scoped commit per PR, with the PR title minus its
[n/3]prefix as the commit subject. Selectively stage only that layer's files/hunks. Do not build the whole feature on one branch and open three PRs with the same cumulative base. - Push each branch to
originand create a draft PR withgh pr create --repo Shopify/ghostferry --draft --base <base-from-above> --head <head-from-above> --title <exact-title-above> --body-file -. Each body containsRefs #<tracking-issue>, the previous PR dependency (ormainfor PR 1), the next PR when available, that layer's behavioral scope and the checks actually exercised. Update all issue/PR links after the three URLs are known; do not useCloses/Fixeson an intermediate PR. - Every PR diff is relative to its immediate parent, and every branch must build and pass its specified checks independently of children. Review changes to a parent require restacking its descendants onto the updated parent. Rewrite only branches created for this stack, publish rewrites with
--force-with-lease, and never use bare--forceor rewrite user-owned branches. - Do not merge, enable auto-merge, push to
main, create release tags or close the issue. Record merge order as PR 1 → PR 2 → PR 3 for reviewers. When a parent is eventually merged, its child must be retargeted/restacked ontomainso the child diff excludes already-landed commits; keep the next layer based on that updated child. This remains ordinary stacked PR maintenance, not a new tooling dependency. - Before handoff, confirm the issue contains the complete plan and links all three PRs, each PR has the intended base and only its layer's diff, and the issue links the verification evidence. Inspect actual Go/Ruby matrix results, not just aggregate CI status, because those jobs allow failures at workflow level. Return the issue URL and the ordered three PR URLs. Report unpublished artifacts or failed/unrun checks explicitly rather than marking the stack complete.
Approach
Implement gradual recovery at batch admission, not a new writer or table queue. Keep the existing fixed table-worker pool and concurrent binlog replay. The controller is shared by all cursors in one DataIterator.Run; it never holds source locks while waiting. Follow the stack ownership/order above: PR 1 completes the existing-value synchronization substeps of steps 3–4; PR 2 implements the remaining steps 1–4 together; PR 3 adds subprocess/cutover regression coverage. None requires changing write/stream scheduling.
1. Define bounded recovery and preserve hard-stop precedence
- In
config.go, add top-levelDataIterationThrottleRecoveryConfigwith exactly the exported fieldsDisabled boolandRecoveryInterval string, plusValidate() error. AddConfig.DataIterationThrottleRecovery DataIterationThrottleRecoveryConfignext toDataIterationConcurrency, andDataIterator.ThrottleRecovery DataIterationThrottleRecoveryConfig. Empty interval normalizes to"5s"; malformed/nonpositive durations returnDataIterationThrottleRecovery.RecoveryInterval must be a positive duration: %qwith the supplied string. Recovery is enabled by default;Disabled: trueretains immediate recovery for workload comparison/rollback. InConfig.ValidateConfig, validate this after the existing zero-to-4 concurrency default; reject nonpositive effective concurrency withDataIterationConcurrency must be greater than zero. The controller constructor repeats the same validation for directly constructed iterators. - Cold starts without an observed throttle retain current maximum concurrency/full configured batches. On an observed hard throttle, stop new copy permits; in-flight batches finish normally. Once clear, allow one copy batch at a time, with effective rows
max(1, configuredRows / 8)using integer division. A table may get repeated turns, but FIFO batch admission prevents one table starving other assigned tables. - After each healthy recovery interval, reduce the divisor
8 → 4 → 2 → 1; only then raise the concurrency limit by exactly one per interval untilDataIterationConcurrency. Never raise both controls in one transition. This yields default batch sizes25 → 50 → 100 → 200at concurrency 1, then concurrency2 → 3 → 4for existing defaults. - A healthy interval starts when the first batch is admitted at that stage, requires at least one successfully completed nonempty batch from that stage and continuous observed
!Throttled(). A batch failure resets the interval without increasing either control. No-work time cannot cause multiple catch-up increases. Another hard throttle resets recovery to one slot/divisor 8. - Every increase starts a new stage generation/window, with no carried-over success count or accumulated intervals. Failure resets affect recovery only; a transient read retry during an otherwise unthrottled cold-start run does not activate recovery. In-flight work is never preempted, including work admitted before a pause or disable transition.
Throttler.Disabled()takes precedence over hard pause and recovery: immediately restore configured limits at the next admission/poll, clear recovery history, and allow cutover/standalone copy to proceed. When re-enabled, use normal limits unless currently hard-throttled; do not replay an old recovery timer.DataIterationThrottleRecovery.Disableddisables only the new limiter, never the hard throttle.- Do not change
Throttler,WaitForThrottle, the binlog writer's hard-stop call, manual pause API, lag threshold, or binlog coordinate semantics. This is recovery control, not a predictor of future overload.
2. Implement a shared, fair admission controller
-
New implementation path:
copy_throttle.go; no equivalent limiter exists in the repository. Use an unexportedcopyThrottleControllerwith a mutex, condition variable, monotonically increasing FIFO tickets, in-flight count, stage generation, concurrency limit, batch divisor and stability timestamps/counters. Return value-type permits rather than allocating a channel/closure per batch. -
Concrete private API:
newCopyThrottleController(throttler Throttler, maxConcurrency int, config DataIterationThrottleRecoveryConfig) (*copyThrottleController, error),acquire() (copyThrottlePermit, error),release(permit copyThrottlePermit, copied bool, err error),run(ctx context.Context), andobserve(now time.Time).copyThrottlePermitcarries the stage generation and divisor. All completions release capacity; only a successful nonempty batch counts as healthy progress. An empty terminal fetch is neutral; an error resets the stability window and advances its generation. Successful completions admitted under older generations do not count toward growth. Keep an internalnow func() time.Time, defaulting totime.Now, so controller tests can advance time without sleeps or a new clock package. -
Recheck hard throttle/disable on every acquisition and release, and from one reusable 500 ms ticker. Observe the signal once before starting workers. Wake waiters on releases/state transitions and when ticket advancement leaves spare capacity. Admission checks ticket order and
inFlight < limit; lowering the limit never pretends existing permits were revoked. The ticker is the only timed waiter; no per-worker polling loops in the managed copy path. Retain the existingWaitForThrottlepath when recovery is disabled or no controller is attached. Cancellation marks the controller closed and broadcasts; queued/future acquisitions returncontext.Canceled, never a permit.WithRetriesalready returns that error without retrying (utils.go:40-42). -
Construct the controller as a local in
DataIterator.Runbefore starting table workers; assign it to a new privateCursor.copyThrottlefield afterNewCursor. Start/stop/join its sampler inside that run, notFerry.Run, soRunStandaloneDataCopyalso works. Use a run-local cancellation context and deferred cancel/join for early returns; stop normally after all workers finish and before done listeners. Constructor errors go to the existing data-iterator fatal handler and return immediately. Nil throttler or disabled recovery needs no controller. A directDataIteratorwith zero-value recovery config gets the same defaults as Ferry. Recovery state is not serialized; restart follows the same cold-start/observed-throttle rules. -
Ferry.NewDataIteratorcopies the new config into the iterator;NewDataIteratorWithoutStateTrackerinherits it. Do not attach the controller toIterativeVerifier's independent cursors orBinlogWriter. -
The constructor rejects
maxConcurrency <= 0; it does not silently normalize it. OnlyConfig.ValidateConfigapplies the existing zero-to-4 default before constructing an iterator. This avoids creating a four-slot limiter alongside a directly constructed zero-worker iterator.
3. Admit before source transactions and size each admitted batch
- In
Cursor.Each, acquire inside each source-read retry attempt, beforeDB.Beginor any source SELECT. Read the configured batch size after acquisition; apply that permit's divisor to the configured base. No shared maximum is overwritten by recovery. - Put the scaling formula in
func (p copyThrottlePermit) batchSize(configuredRows uint64) uint64and use it fromEach; controller tests exercise that same function instead of duplicating the formula. Issued permit divisors are always one of 8, 4, 2 or 1. KeepCursorConfig.GetBatchSizereturning the unscaled base for other existing consumers. - Split the query body into private
fetch(db SqlPreparer, batchSize uint64) (*RowBatch, PaginationKey, error). Preserve exportedFetch(db SqlPreparer) (*RowBatch, PaginationKey, error)as its existing full-configured-size entry point;Eachcalls the private helper with its permit-specific size. LSP found onlyCursor.EachcallingFetchin this repository. PreserveBuildSelect, column/fingerprint handling, row-lock SQL and pagination behavior. - Hold a permit through source fetch, synchronous listeners, target retries/commit and source rollback; release only after rollback. Release once on begin/fetch failure before a read retry sleeps, empty terminal fetch, invalid pagination, callback failure, or success. Mark success only for a nonempty batch whose callback completed. Never release/reacquire inside
BatchWriter.WriteRowBatch; that would wait while holding source locks. - Preserve per-table override precedence in
CursorConfig.GetBatchSize; scale whichever base it returns, including tiny bases down to one row. An operator's live batch-size reduction takes effect at the next admission, without resetting recovery or bypassing its divisor. - Make the existing live maximum race-free without changing its type/wire shape: atomic store in
Config.Update, atomic load inCursorConfig.GetBatchSize, and encode a freshly constructedUpdatableConfigsnapshot with an atomic load inControlServer.HandleConfigGet. Startup-only assignments/defaults remain ordinary writes before workers start. Do not introduce a second mutable maximum or change/api/configfields. - Keep existing progress fields unchanged:
TableProgress.BatchSizeis the configured maximum andActiveDataIteratorscounts started, incomplete tables, not current permits (ferry.go:1067-1095). Do not use either as proof of actual recovery concurrency; observe batch admission/transactions in verification. No new status schema or writer telemetry is required for this change.
4. Require fresh built-in lag evidence before increasing load
- Protect
LagThrottler.lagand new sample metadata with one mutex;Throttledreads under that mutex. Record a monotonically increasing sample sequence and observation time only after a successful non-NULL query result. Existing missing-row/NULL semantics still retain the last lag; existing SQL errors/retries/fatal handling remain unchanged. The freshness calculation requires a positive polling interval: after parsing inNewLagThrottler, rejectinterval <= 0withinvalid UpdateInterval: must be greater than zerorather than accepting a busy loop or impossible freshness window. - Add private
LagThrottler.recoverySample() (sequence uint64, observedAt time.Time, interval time.Duration). The controller uses this optional internal capability when present. A stage increase also requires a sequence newer than the one at stage admission and a sample no older than twice the lag update interval. Missing/stale samples freeze growth at the current stage; they do not clear a hard stop or invent a new global stop. Generic custom throttlers andPauserThrottleruse the existing boolean plus elapsed time/successful-copy evidence because their interface has no freshness contract. - Keep stability based on this external safety signal, not
BinlogWriterLag: pausing the writer based on its own lag would create a feedback deadlock, and idle event timestamps are not an authoritative backlog measurement.
Critical files & anchors
sharding/sharding.go,ShardingFerry.Run:145-182: cutover disables throttling around binlog drain and standalone delta/primary-table copy; recovery must not delay this path.testhelpers/test_ferry.go,TestFerry.Start:84-115: reuse copy listeners for barriers. Its misleadingly named binlog-apply listeners run on the streamer, not after target commit; verify replay by querying the target.testhelpers/integration_test_case.go,CopyData/StartFerryAndDataWriter/Teardown: existing lifecycle and checksum verification; it dropsgftest*and toggles source read-only, so only run against disposable databases.
Verification
Planning is read-only: no issue, branch, PR, migration, test suite or benchmark was created/executed. The reported binary-throttle problem is accepted; do not rerun a baseline merely to reconfirm it. Apply the per-layer gates in the stack section, then verify the complete stack before publishing the final evidence.
Prerequisites and commands
Run from the repository root after implementation. go.mod requires Go 1.27.2. Use disposable MySQL services from docker-compose_8.0.yml: source 29291, target 29292, root with empty password, ROW binlogs and GTID enabled. N1_PORT/N2_PORT override Go fixture ports. Ruby checks require the repository Gemfile dependencies already installed. Never point these commands at production or shared development databases.
Start the disposable fixture only after execution is approved: docker compose -f docker-compose_8.0.yml up -d. Wait for both docker compose -f docker-compose_8.0.yml exec -T mysql-1 mysqladmin ping -uroot and the equivalent mysql-2 command to succeed before tests. Set MYSQL_VERSION=8.0 for expected MySQL error handling in existing lag tests.
- Pure controller tests:
go test -race . -run '^TestCopyThrottle' -count=1. - Focused real-MySQL coverage:
MYSQL_VERSION=8.0 go test ./test/go -run 'TestCopyThrottle|TestDataIterator|TestThrottler|TestNewThrottler|TestSelectUpdateBinlogCopy|TestUpdateBinlogSelectCopy|TestSelectCopyUpdateBinlog|TestOnlyDeleteRowWithMaxPaginationKey' -count=1 -p=1. - Race-check the newly changed boundaries:
MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestCopyThrottle|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs' -count=1 -p=1. - Existing Go consumers:
MYSQL_VERSION=8.0 go test . ./test/go ./copydb/test ./sharding/test -count=1 -p=1. - Repeat command 2 with
GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid MYSQL_VERSION=8.0; the default run uses file/position coordinates. - Existing interrupt/resume executable coverage:
MYSQL_VERSION=8.0 bundle exec rake test TEST=test/integration/interrupt_resume_test.rb; repeat withGHOSTFERRY_BINLOG_COORDINATE_MODE=gtid.
Permanent behavior coverage
Add copy_throttle_test.go in package ghostferry for the private controller, matching existing root test conventions. Name tests with prefix TestCopyThrottle. Use a fake monotonic clock and channel barriers rather than wall-clock sleeps for progression/order assertions.
- Four contenders, configured max 4/base 200, hard throttle observed then cleared: one admitted batch with 25 rows; no increase before the healthy interval; successive healthy stages admit limits/sizes
(1,50),(1,100),(1,200),(2,200),(3,200),(4,200). Assert admission behavior, not just copied fields. - A new throttle while four batches are in flight allows those batches to finish but no new permit; clearing it cannot admit the first recovery batch until the outstanding count falls below one. Old-generation completions cannot jump stages. Repeated flapping always restarts at one/eighth-size.
- No completed work, failed work, stale/unchanged lag samples, NULL/no rows and old-generation successes cannot produce growth. Fresh observations and successful work resume growth. Empty terminal fetch releases capacity without resetting healthy progress. Include
RecoveryInterval/lag polling interval zero, negative and malformed input boundaries. - FIFO progress across assigned tables, tiny configured sizes (1 and 7 remain at least one), unequal per-table overrides, re-throttling during a fetch retry, and cleanup after begin/fetch/callback errors. Cancellation releases queued waiters with
context.Canceled; it must not start SQL or leak the sampler. SetDisabled(true)bypasses paused/recovering state without waiting for the stability interval. Re-enabling does not resurrect old timers. Disabled recovery leaves hard throttling intact. Empty/completed table sets and a single-table migration finish normally.
Extend test/go/data_iterator_test.go and test/go/race_conditions_integration_test.go with TestCopyThrottle... coverage using real queries and existing TestFerry/IntegrationTestCase lifecycle:
- Pause four already-started table cursors and let previously admitted batches drain before clearing the pause. Hold the first resumed batch's listener and assert no second copy callback enters. Issue an UPDATE against a different waiting table through a separate source connection with
innodb_lock_wait_timeout=1: it must complete, proving the blocked cursor did not acquire source row locks before its permit. - While that first copy callback is held, insert a source row beyond the captured maximum pagination key. Once the hard pause is clear, query the target until that row is committed: binlog replay must remain independent of copy recovery.
- Update the configured base to 104 during first-stage recovery and observe a subsequent batch of 13 rows; an override of 80 still yields 10. Assert exact source/target equality after copy/drain, not just counts or callbacks.
- Drive a real
LagThrottlerwith themeta.lag_tablefixture used inlag_throttler_test.go: above threshold stops both paths, fresh below-threshold samples allow staged copy recovery, deleting all lag rows does not invent new healthy samples. Keep these DB-mutating tests serial and join background goroutines before teardown. - Add
test_interrupt_resume_during_throttle_recoveryintest/integration/interrupt_resume_test.rb. Reuse its existing single-table seed and signal/state harness. Add the small executabletest/lib/go/throttle_recovery_ghostferry/main.gousing the sametf.Setup/tf.Runpattern asminimal_ghostferry/main.go: inRunCallbacks.BeforeInitialize, set concurrency 1, recovery interval"30s", and enable the existing control server on127.0.0.1:18000. No existing integration executable enables this pause surface; reuseintegrationferryrather than expanding its environment-variable config protocol. In Ruby, useNet::HTTPto POST/api/pausefrom the firstBEFORE_ROW_COPYcallback. From its firstAFTER_ROW_COPY, start a joined thread that waits one second then POSTs/api/unpause; returning from that callback lets the permit release observe the pause. Pause again from the secondBEFORE_ROW_COPY, then callterm_and_wait_for_exitfrom the secondAFTER_ROW_COPYso no third batch can enter. The interrupted target contains 225 rows (200 initial + 25 recovered); its max pagination key matches the state dump. Resume with the ordinary minimal executable andassert_test_table_is_identical. Run serially; join the unpause thread and propagate request failures. No recovery fields enter serialized state. Keep existing race-order and sharding standalone-copy cases passing.- The unpause thread is joined after the interrupted subprocess exits, not inside
AFTER_ROW_COPY; the callback must return while the hard pause is still set so release observes it. Set the test executable's configured base explicitly to 200, rather than making the 225-row correctness assertion depend on an incidental default.
- The unpause thread is joined after the interrupted subprocess exits, not inside
Real-program smoke, beyond tests
Create a throwaway Go executable at /tmp/ghostferry-recovery-smoke/main.go, run with MYSQL_VERSION=8.0 go run /tmp/ghostferry-recovery-smoke/main.go from the root, and capture its output. Use the real public Ferry, SQL connections and testhelpers.NewTestFerry/SeedInitialData, not mocked readers/writers. Fixture/setup contract:
- Recreate only
gfteston disposable source/target. Seed source tablest1–t4with 10,000 rows each and matching empty target tables. Configure concurrency 4, base 200, recovery interval"30s"so assertions fit within the first stage, and aPauserThrottleralready paused. Enable the control server at127.0.0.1:18000. Attach the first-copy listener barrier beforeStart; initialize/start/run the real ferry and wait until all fourTargetPaginationKeysentries exist. - Insert a sentinel source row with ID 10001 after those maxima are captured. POST
/api/unpause. The first copy callback must report 25 rows. Hold it; within five seconds, the target must contain the sentinel while no second copy callback has entered. Query the target for evidence rather than treating streamer callbacks as commits. - POST
/api/configwith{"DataIterationBatchSize":104}and GET it concurrently usingContent-Type: application/json; the configured value must be 104. Release the first callback and observe a 13-row subsequent copy batch before the 30-second stage expires. Print actual table/batch-size/admission events. A missed deadline is a smoke failure, not a reason to relax the assertion. - Let copying finish; call
WaitUntilRowCopyIsComplete,FlushBinlogAndStopStreaming, joinRun, thenStopTargetVerifier. UseChecksumTableVerifier.VerifyDuringCutoverplus explicit ordered source/target row comparison for all four tables. The sentinel must occur exactly once. Repeat the executable withGHOSTFERRY_BINLOG_COORDINATE_MODE=gtid.
The smoke must prove real reduced-batch admission, independent binlog progress, live batch-size composition and final data integrity; deterministic controller tests must prove the complete ramp without waiting through default five-second stages. Do not claim a throughput improvement from this scripted pause or from unit tests.
Assumptions & contingencies
- User decision: preserve the hard lag stop for both copy and binlog work, including operator pause and cutover-disable semantics.
- User decision: limit this change to gradual copy throttling. No interleaving assessment, arbitration prototype or write/stream scheduling change is required; retain only the binlog-progress checks that ensure copy recovery cannot stall replay beyond the existing hard stop.
- Recommended rollout choice: enable recovery by default only after an observed throttle; retain current unthrottled startup speed. Set
DataIterationThrottleRecovery.Disabledtotrueto opt out if workload evaluation rejects that choice. Five-second stages and the initial one-eighth batch size are conservative starting values, not measured optima;RecoveryIntervalis the supported deployment tuning knob. - A custom throttler may provide only boolean health. The fixed fallback is already specified in step 4: preserve its contract and use elapsed healthy time plus completed work, without introducing a new load query or replacing it with
LagThrottler. - No production throughput measurements were collected. If load-dependent throttling persists after recovery is deployed, retain the hard stop and report the measured bottleneck; do not weaken lag protection or claim the scheduling change solved a database-capacity limit.
Activity
driv3r commentedon Oct 30, 2026
Verification evidence (stack tip
throttle-recovery/03-resume-coverage)Environment: macOS arm64, Go 1.27.2, and disposable MySQL 8.0.32 source and target from
docker-compose_8.0.ymlon ports 29291/29292 (ROW binlogs, GTID on).Stack: #472 → #473 → #474
Verification commands
go test -race . -run '^TestCopyThrottle' -count=1-count=30, and with-count=20 -cpu 1,2,8.MYSQL_VERSION=8.0 go test ./test/go -run 'TestCopyThrottle|TestDataIterator|TestThrottler|TestNewThrottler|TestSelectUpdateBinlogCopy|TestUpdateBinlogSelectCopy|TestSelectCopyUpdateBinlog|TestOnlyDeleteRowWithMaxPaginationKey' -count=1 -p=1MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestCopyThrottle|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs' -count=1 -p=1MYSQL_VERSION=8.0 go test . ./test/go ./copydb/test ./sharding/test -count=1 -p=1GHOSTFERRY_BINLOG_COORDINATE_MODE=gtidTestThrottlerMultipleServerIDs; see the existing flake below.MYSQL_VERSION=8.0 bundle exec rake test TEST=test/integration/interrupt_resume_test.rb, file/position and GTIDtest_interrupt_resume_during_throttle_recovery. Some runs hung on an existing test; see below.go test ./test/go -run '^TestCopyThrottleStandaloneCopyBypassesPause$'andgo test ./sharding/test -run 'TestJoinedTablesTestSuite|TestPrimaryKeyTableTestSuite'go test -race ./test/go -run 'TestConcurrentBatchSizeUpdates|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs|TestDataIterator/TestBatchSizeUpdate'DATA RACEs.Mutation check: if admission moves after
Begin+ fetch,TestCopyThrottleResumesOneSmallBatchWithoutSourceLocksfails withError 1205 (HY000): Lock wait timeout exceededon the UPDATE of a waiting table.Existing flakes (not changed by this stack)
TestThrottlerThrottlesAndUnthrottles/TestThrottlerMultipleServerIDs: these rely on a 10 ms sleep against a 5 ms lag poll. In-count=100loops, unmodifiedorigin/mainfailed 3 of 200 runs, and [1/3] Make throttle and batch-size access race-free #472's tip also failed sometimes.test_interrupt_resume_inline_verifier_will_verify_entries_in_reverify_storecan hang the Ruby suite:MIN(id) + 1, which can be one of the 140 random holes in the seeded table. The DELETE then writes no binlog event, soAFTER_BINLOG_APPLYnever fires.AFTER_ROW_COPYhandler runssleepforever, so WEBrick shutdown blocks after Ghostferry's 30 s status timeout.Client.Timeout exceeded while awaiting headers, thengoing to shutdown ..., and then the hang.origin/main, but the trigger is in the seeded data, not in the copy throttle.Real-program smoke, beyond tests
The smoke is a throwaway
/tmp/ghostferry-recovery-smoke/main.go, run withMYSQL_VERSION=8.0 go run /tmp/ghostferry-recovery-smoke/main.gofrom the repository root. It uses the publicFerry, real SQL connections,testhelpers.NewTestFerry-style config andSeedInitialData:gftest.t1–t4with 10,000 rows each, concurrency 4, base batch size 200,RecoveryInterval: "30s", aPauserThrottlerpaused before start, and the control server on127.0.0.1:18000.t1.id=10001and sendsPOST /api/unpause.POST /api/config {"DataIterationBatchSize":104}andGETconcurrently.ChecksumTableVerifier.VerifyDuringCutoverplus an ordered row comparison.File/position:
GTID (
GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid):The whole copy finished inside the first 30 s stage, so it ran as one 13-row batch at a time. The deterministic controller tests in #473 cover the full ramp. No throughput improvement is claimed from this scripted pause or from the unit tests.
#472 live batch-size smoke (concurrency 1, base 200, no pause):
CI matrix
The Go and Ruby jobs use
continue-on-error, so the aggregate check proves nothing. The outcomes below come from each job'sRunning Golang tests/Running Ruby testsstep.InterruptResumeTestand passed on attempt 3The CI logs confirm the new Go tests ran, for example
PASS test/go.TestCopyThrottleResumesOneSmallBatchWithoutSourceLocksand--- PASS: TestCopyThrottleRecoversInStages.The cancelled jobs printed no captured test output, so the hanging test cannot be identified from CI. For comparison,
ruby-testjobs on other branches andmainwere cancelled the same way in about 10% of the last ~35 runs per matrix entry (for example, 5.7: 4 of 35). These timeouts predate this stack.