Skip to content

Add DB::purgeTables() to drop all data in table section - #229

Merged
greensky00 merged 3 commits into
masterfrom
junahn_pr1
Nov 17, 2026
Merged

greensky00 merged 3 commits into
masterfrom
junahn_pr1

Conversation

@greensky00

Copy link
Copy Markdown
Contributor
  • New API that drops all table data by swapping every table for an empty one, without waiting for or blocking ongoing operations.

  • The log section is untouched; unflushed records remain and will be flushed to the new tables later.

  • Ongoing iterators and snapshots keep reading the old tables, which are deleted when their last reference is released.

  • Ongoing compaction, split, and merge check whether their source table has been removed under the manifest lock, and cancel themselves if so. This prevents purged data from being resurrected by their results.

  • Fixed compactL0() returning without clearing the compaction status when adding a new L0 table fails.

  • Added tests for basic purge and purge during each compaction type.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical ownership and locking bugs can cause use-after-free during compaction and migration, while incomplete purge enumeration can leave purged data readable.

5 open issues
What changed in this PR

Adds DB::purgeTables() to clear the table section while preserving unflushed log records, with safeguards intended to preserve existing readers and prevent compaction from restoring purged data.

Changes:

  • Adds the public API and replaces existing tables with empty L0 tables.
  • Adds purge checks to compaction, migration, split, merge, and repair paths, and fixes L0 compaction-status cleanup.
  • Adds tests for basic purge, iterator continuity, reopening, and purge during compaction.
File Description
tests/​jungle/​level_extension_test.cc Adds basic and concurrent-compaction purge tests.
src/​table_split.cc Adds purge guards for split, merge, and repair operations.
src/​table_mgr.h Declares the internal purge operation.
src/​table_mgr.cc Implements table replacement and old-table release.
src/​table_compaction.cc Adds cancellation checks and L0 error-path cleanup.
src/​jungle.cc Implements the public API with handle validation.
include/​libjungle/​jungle.h Declares and documents DB::purgeTables().

Review effort: Balanced

Comment thread src/table_compaction.cc Resolved
Comment thread src/table_compaction.cc Resolved
Comment thread src/table_compaction.cc Resolved
Comment thread src/table_mgr.cc
Comment on lines +489 to +490
for (size_t ii = 0; ii < num_levels; ++ii) {
mani->getTablesRange(ii, empty_key, empty_key, old_tables);
Comment thread src/table_mgr.cc
// Store manifest file first, and then release the old tables.
s = mani->store(true);

for (TableInfo*& t_info: old_tables) t_info->done();

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical durability and concurrency defects can delete still-referenced table files, resurrect purged records after a crash, and cause null dereferences or data races during compaction and teardown.

6 open issues
3 resolved since last review

Review effort: Balanced

Comment thread src/table_compaction.cc Resolved
Comment thread src/table_manifest.h Outdated
// Reset the file's back pointer BEFORE releasing the
// ownership, as the other owner can destroy the file
// right after that.
if (file->getTableInfo() == this) file->setTableInfo(nullptr);
Comment thread src/table_mgr.cc
Comment on lines +484 to +485
std::lock_guard<std::mutex> l_l0(L0Lock);
std::lock_guard<std::mutex> l_mani(mani->getLock());
Comment thread src/table_mgr.cc Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The purge regressions miss the compactL0 cancellation and cleanup path, and their sleep-based synchronization can fail nondeterministically without reliably exercising overlapping compaction.

5 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Synchronize purge tests with compaction using a barrier

tests/​jungle/​level_extension_test.cc:1452

The 200 ms sleep does not guarantee that the worker has selected its victim before the purge. If the worker is scheduled later, purgeTables() empties L1 first and victim selection returns TABLE_NOT_FOUND, failing the expected-result assertion even when the implementation is correct. Please use a callback/barrier to wait until the worker holds its source-table reference, then purge and release the worker, so these cases reliably exercise an overlapping compaction.

2 resolved since last review

Review effort: Balanced

Comment thread src/table_compaction.cc
Comment on lines +1073 to +1074
if ( origin_table->isRemoved() ||
(merge_table && merge_table->isRemoved()) ) {
* New API that drops all table data by swapping every table for an
empty one, without waiting for or blocking ongoing operations.

* The log section is untouched; unflushed records remain and will
be flushed to the new tables later.

* Ongoing iterators and snapshots keep reading the old tables, which
are deleted when their last reference is released.

* Ongoing compaction, split, and merge check whether their source
table has been removed under the manifest lock, and cancel
themselves if so. This prevents purged data from being resurrected
by their results.

* Fixed `compactL0()` returning without clearing the compaction
status when adding a new L0 table fails.

* Added tests for basic purge and purge during each compaction type.
* Initialize checkpoint 0 of newly compacted tables before adding
them to the manifest, as `purgeTables()` can free them right after.

* Read the new L0 table number in `compactL0()` while holding the
manifest lock, for the same reason.

* Add new tables and remove source tables in a single manifest lock
section in `compactInPlace()` and L0-only `compactL0()`.

* Start from the first table in `getTablesRange()` if the min key is
empty, so that tables sharing the empty min key are not skipped.

* Replace the migration flag with an owner count in `TableFile`, so
that a file shared by migrated tables is deleted only by its last
owner. This fixes use-after-free when old readers are still open.

* Added a test that removes migrated tables while a snapshot and an
iterator are open, with and without purge.
* Compaction, split, and merge now check whether their tables are
removed (e.g., migrated and then purged) right after locking them,
and cancel themselves if so. Otherwise, they could use a stale back
pointer of the shared file, which caused a null dereference in
in-place compaction.

* Make the back pointer of `TableFile` atomic, and clear it with
compare-and-swap, as multiple owners of a migrated file can release
it at the same time.

* If `purgeTables()` fails to store the manifest, keep the old table
files on disk, as the manifest file on disk may still refer to them.

* Added a debug callback before in-place compaction locks its victim,
and a test that migrates and purges the victim at that moment.
@greensky00
greensky00 merged commit 69fc9fb into master Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants