Skip to content

Copy rows at the minimum pagination key and reject negative keys - #465

Closed
driv3r wants to merge 3 commits into
mainfrom
fix-pagination-key-min-boundary
Closed

driv3r wants to merge 3 commits into
mainfrom
fix-pagination-key-min-boundary

Conversation

@driv3r

@driv3r driv3r commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #305.

Problem

The initial pagination key used to be the real value 0 (integers) or an empty binary/string value. Starting with WHERE pk > <min> skipped those rows, and the last < max loop skipped a table containing only the minimum key. Existing negative keys were also silently skipped. The InlineVerifier could not notice rows the iterator never produced.

Fix

  • An unstarted cursor has a nil checkpoint. Its first query has no lower bound; every real checkpoint uses strict >, including 0 and empty binary keys.
  • StateTracker.LastSuccessfulPaginationKey preserves that distinction: an absent checkpoint returns nil; an actual zero/empty checkpoint is retained on resume. The iterative verifier also starts without a lower bound.
  • DefaultBuildSelect and all three ShardedCopyFilter query shapes support nil. The earlier IsMin() interface addition is removed: key values no longer encode cursor state.
  • Startup validation reads the first signed integer pagination key selected by the same copy query/filter. Selected negative keys still abort. Excluded tenants' negative keys do not block a move, and an empty selected subset is skipped even when every row belonging to other tenants is negative.
  • The cursor also rejects a negative first row inserted between startup validation and the first batch, before it can wrap into an unsigned checkpoint.

Review regressions addressed

The initial version still chose >= after copying key 0. With batch size 1, its second query returned 0 again and failed the progress check. The new tests reproduced that failure for numeric and binary keys, including resuming at those checkpoints.

The initial negative-key check inspected the whole table. The new filtered-copy regression failed on an excluded tenant's negative key. Validation now uses the configured row filter rather than a table-wide pk < 0 query.

Coverage now includes:

  • Initial copying with batch size 1: integer 0, a table containing only 0, and an empty binary key followed by another key.
  • Resuming after a persisted zero/empty checkpoint without copying it again.
  • A selected positive subset alongside excluded negative keys; an empty selected subset in an all-negative table; a selected negative MEDIUMINT that must still fail.
  • A negative key inserted after the startup probe.
  • A complete inline-verified shard migration covering normal, primary-key and joined tables. This replaces the SQL-string-only minimum-key test.

API notes

  • CopyFilter.BuildSelect must accept a nil last key as no lower bound and return rows in ascending pagination-key order. Non-nil checkpoints are exclusive.
  • StateTracker.LastSuccessfulPaginationKey returns nil before any batch has been copied.
  • MaxPaginationKeys now takes *CursorConfig instead of a database connection so its validation can use the copy filter.
  • No change to the serialized representation of real pagination keys or to the PaginationKey interface.

Remaining limitation

This does not add general signed-key support. The existing binlog-only negative-key reverification gap is not addressed; the changes protect the initial copy and its resume boundaries.

Verification

On local MySQL 8.0.32:

  • The newly added review regressions failed before these corrections and pass afterward.
  • The root, test/go, copydb/test, and sharding/test suites passed during go test -count=1 -p=1 ./.... The overall command is blocked by an unchanged vet error at testhelpers/integration_test_case.go:124 (Fatalf called with a non-constant format string).
  • A standalone live migration with batch size 1 and InlineVerifier copied integer/binary minimum keys across normal, primary-key and joined tables, left an excluded-only negative-key table empty on the target, and returned DataCorrect: true.
  • Ruby iterate_by_column_test.rb, types_test.rb, iterative_verifier_test.rb, and interrupt_resume_test.rb: 41 tests, 0 failures.

@driv3r
driv3r force-pushed the fix-pagination-key-min-boundary branch from a98aa31 to 895cb05 Compare October 19, 2026 16:39
@driv3r
driv3r requested review from grodowski and mtaner October 19, 2026 16:44
The first batch selects pagination keys strictly greater than the
minimum (0 for integers, empty for binary keys), so rows keyed 0 or ''
are never copied, a table holding only key 0 is skipped, and rows with
negative keys are silently dropped.
Cursors starting from the minimum key (0, or empty for binary keys) now
select rows >= that key on the first batch, so those rows are copied and
a table holding only such a row is no longer skipped. DefaultBuildSelect
and ShardedCopyFilter use the new PaginationKey.IsMin().

Negative values in a signed integer pagination column cannot be reached
by integer pagination, so the data iterator now aborts instead of
silently skipping them.

Fixes #305.
@driv3r
driv3r force-pushed the fix-pagination-key-min-boundary branch from e4d9e18 to 16975d1 Compare October 19, 2026 18:19
@driv3r
driv3r requested a review from pawandubey October 22, 2026 08:17
Comment thread cursor.go

// The first query has no lower bound. Recheck its first row in case a
// negative key was inserted after startup validation.
if len(batchData) == 0 && !c.paginationKeyColumn.IsUnsigned &&

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bot found an edge case (quite unlikely?) where a negative pagination key can cause silent data loss here

Comment thread sharding/filter.go
return sq.Select(columns...).
From(quotedTable).
Join("("+selectPaginationKeys+") AS `batch` USING("+quotedPaginationKey+")", f.ShardingValue, lastPaginationKey.SQLValue()), nil
Join("("+selectPaginationKeys+") AS `batch` USING("+quotedPaginationKey+")", args...), nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add .OrderBy(quotedPaginationKey) after this .Join(...)?

selectPaginationKeys already sorts the keys, but this JOIN fetches the full rows for those keys and can return them in a different order. The cursor depends on the returned rows being sorted: it checks the first row for a negative key and uses the last row’s key as the next checkpoint.

I reproduced this on MySQL 8.0.32 with batch size 4. Startup sees keys 10, 20, 30, 40. If -1 is inserted before the first batch, this query returns [10, 20, 30, -1]. The negative-key check sees only 10 and passes. The checkpoint becomes -1, which wraps to the largest unsigned integer, so the cursor reports success without copying 40.

Adding ordering to this outer query makes it return [-1, 10, 20, 30], so the existing check rejects the negative key. Could we cover this with a regression test using a batch size greater than one?

Comment thread cursor.go

// The first query has no lower bound. Recheck its first row in case a
// negative key was inserted after startup validation.
if len(batchData) == 0 && !c.paginationKeyColumn.IsUnsigned &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this check if c.lastSuccessfulPaginationKey is not nil? - it means the following batches are not negative right?

And even though this is not a terminal error, this will still retry 5 times because the whole logic is wrapped in WithRetries

@driv3r

driv3r commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @grodowski and @mtaner for reviews! I think that's not the correct approach in the end, I think instead of validating for id>=0, we should add a signed PK support, and just be able to run with negative numbers as well. The current solution is like putting a lipstick on a pig, so lets skip it

@driv3r driv3r closed this Nov 3, 2026
@driv3r
driv3r deleted the fix-pagination-key-min-boundary branch November 3, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ghostferry misses data for PK values of <= 0

3 participants