Repository navigation
Conversation
a98aa31 to
895cb05
Compare
The first batch selects pagination keys strictly greater than the minimum (0 for integers, empty for binary keys), so rows keyed 0 or '' are never copied, a table holding only key 0 is skipped, and rows with negative keys are silently dropped.
Cursors starting from the minimum key (0, or empty for binary keys) now select rows >= that key on the first batch, so those rows are copied and a table holding only such a row is no longer skipped. DefaultBuildSelect and ShardedCopyFilter use the new PaginationKey.IsMin(). Negative values in a signed integer pagination column cannot be reached by integer pagination, so the data iterator now aborts instead of silently skipping them. Fixes #305.
e4d9e18 to
16975d1
Compare
|
|
||
| // The first query has no lower bound. Recheck its first row in case a | ||
| // negative key was inserted after startup validation. | ||
| if len(batchData) == 0 && !c.paginationKeyColumn.IsUnsigned && |
There was a problem hiding this comment.
My bot found an edge case (quite unlikely?) where a negative pagination key can cause silent data loss here
| return sq.Select(columns...). | ||
| From(quotedTable). | ||
| Join("("+selectPaginationKeys+") AS `batch` USING("+quotedPaginationKey+")", f.ShardingValue, lastPaginationKey.SQLValue()), nil | ||
| Join("("+selectPaginationKeys+") AS `batch` USING("+quotedPaginationKey+")", args...), nil |
There was a problem hiding this comment.
Can we add .OrderBy(quotedPaginationKey) after this .Join(...)?
selectPaginationKeys already sorts the keys, but this JOIN fetches the full rows for those keys and can return them in a different order. The cursor depends on the returned rows being sorted: it checks the first row for a negative key and uses the last row’s key as the next checkpoint.
I reproduced this on MySQL 8.0.32 with batch size 4. Startup sees keys 10, 20, 30, 40. If -1 is inserted before the first batch, this query returns [10, 20, 30, -1]. The negative-key check sees only 10 and passes. The checkpoint becomes -1, which wraps to the largest unsigned integer, so the cursor reports success without copying 40.
Adding ordering to this outer query makes it return [-1, 10, 20, 30], so the existing check rejects the negative key. Could we cover this with a regression test using a batch size greater than one?
|
|
||
| // The first query has no lower bound. Recheck its first row in case a | ||
| // negative key was inserted after startup validation. | ||
| if len(batchData) == 0 && !c.paginationKeyColumn.IsUnsigned && |
There was a problem hiding this comment.
do we need this check if c.lastSuccessfulPaginationKey is not nil? - it means the following batches are not negative right?
And even though this is not a terminal error, this will still retry 5 times because the whole logic is wrapped in WithRetries
|
Thanks @grodowski and @mtaner for reviews! I think that's not the correct approach in the end, I think instead of validating for id>=0, we should add a signed PK support, and just be able to run with negative numbers as well. The current solution is like putting a lipstick on a pig, so lets skip it |
Fixes #305.
Problem
The initial pagination key used to be the real value
0(integers) or an empty binary/string value. Starting withWHERE pk > <min>skipped those rows, and thelast < maxloop skipped a table containing only the minimum key. Existing negative keys were also silently skipped. The InlineVerifier could not notice rows the iterator never produced.Fix
>, including0and empty binary keys.StateTracker.LastSuccessfulPaginationKeypreserves that distinction: an absent checkpoint returns nil; an actual zero/empty checkpoint is retained on resume. The iterative verifier also starts without a lower bound.DefaultBuildSelectand all threeShardedCopyFilterquery shapes support nil. The earlierIsMin()interface addition is removed: key values no longer encode cursor state.Review regressions addressed
The initial version still chose
>=after copying key 0. With batch size 1, its second query returned 0 again and failed the progress check. The new tests reproduced that failure for numeric and binary keys, including resuming at those checkpoints.The initial negative-key check inspected the whole table. The new filtered-copy regression failed on an excluded tenant's negative key. Validation now uses the configured row filter rather than a table-wide
pk < 0query.Coverage now includes:
MEDIUMINTthat must still fail.API notes
CopyFilter.BuildSelectmust accept a nil last key as no lower bound and return rows in ascending pagination-key order. Non-nil checkpoints are exclusive.StateTracker.LastSuccessfulPaginationKeyreturns nil before any batch has been copied.MaxPaginationKeysnow takes*CursorConfiginstead of a database connection so its validation can use the copy filter.PaginationKeyinterface.Remaining limitation
This does not add general signed-key support. The existing binlog-only negative-key reverification gap is not addressed; the changes protect the initial copy and its resume boundaries.
Verification
On local MySQL 8.0.32:
test/go,copydb/test, andsharding/testsuites passed duringgo test -count=1 -p=1 ./.... The overall command is blocked by an unchanged vet error attesthelpers/integration_test_case.go:124(Fatalfcalled with a non-constant format string).DataCorrect: true.iterate_by_column_test.rb,types_test.rb,iterative_verifier_test.rb, andinterrupt_resume_test.rb: 41 tests, 0 failures.