Skip to content

fix(core): retry lost custody declarations, bind handoff acks to incarnation, and key inbox by full tx identity (#1202) - #1239

Merged
RyanKung merged 3 commits into
developfrom
fix/1202-relay-custody-and-inbox
Nov 10, 2026
Merged

RyanKung merged 3 commits into
developfrom
fix/1202-relay-custody-and-inbox

Conversation

@RyanKung

@RyanKung RyanKung commented Nov 9, 2026

Copy link
Copy Markdown
Member

Summary

Closes #1202 (including consolidated #1207 and #1208).

  1. Retry lost relay custody declarations ([P1] Relay custody & inbox convergence: retry lost custody declarations, bind handoff cleanup acks to custody incarnation, and key inbox consumption by full transaction identity (#1202, #1207, #1208) #1202):
    • Changes PeerRing::relay_custody_declared to Mutex<BTreeMap<Did, uuid::Uuid>>, recording (peer, tx_id) before egress.send_payload in declare_opened_relay_custody.
    • Clears the declaration record (matching exact tx_id) whenever enqueue_detached yields Ok(None) or enqueued.await_admission resolves to Cancelled or Err(_), ensuring subsequent stabilization rounds retry lost declarations.
  2. Bind handoff cleanup acknowledgements to local custody incarnation ([P1] Bind handoff cleanup acknowledgements to the local custody incarnation #1207):
    • Adds per-slot StorageCustodyIncarnation tracking (generation + handoff_observed) on PeerRing and local custody incarnation metadata on PlacedEntry and SyncedEntryAck.
    • remove_storage_entry_confirmed_by deletes only when storage_custody_incarnation(key) == ack.custody_incarnation(), preventing delayed/ABA handoff ACKs from deleting re-acquired custody state.
  3. Key relay inbox consumption by complete signed transaction identity ([P2] Track relay inbox consumption by complete signed transaction identity #1208):
    • Introduces RelayTransactionId { stream: StreamKey, sequence: u64, digest: TransactionDigest } and updates RelayInFlight, InboxElement::relay_id, and SwarmInboxDelivery to track consumption by complete signed transaction identity rather than bare tx_id: Uuid.

Test Plan

  • cargo test -p rings-core --features dummy --lib -- dht::storage dht::stabilization message::handlers::storage swarm::application tests::default::test_inbox tests::default::test_dht_stateright (238 passed)
  • cargo check --tests -p rings-core

Copy link
Copy Markdown
Member Author

Reviewed commit: 9872ee70d6bd1eda274f9d77a263a9e50880a1b9.

Verdict: one confirmed P2 finding needs correction.

[P2] Bound custody-incarnation metadata across key eviction. The unconditional insertion in note_storage_custody_write creates metadata for every successfully written key, even without a handoff. No path removes these entries. Data eviction, expiry and acknowledged cleanup therefore leave historical keys in memory indefinitely. Sustained distinct incoming topic writes can grow this map without bound despite a bounded data store. Please reclaim inactive metadata while preserving ABA protection, for example by tracking bounded outstanding handoff capabilities with globally unique tokens.

Reproduction: a review-only regression, review_custody_metadata_stays_bounded_when_elements_are_evicted performs 100 valid join_storage_entry writes against MemStorage::bounded(1). It fails: storage contains 1 row, but the incarnation map contains 100 keys.

Other reviewed dimensions:

  • Protocol safety: declaration failure/cancellation clears only the matching transaction ID; stale watchers do not clear newer declarations.
  • Handoff and wire compatibility: cleanup shares storage_transition with writes. Both new incarnation fields use #[serde(skip)]; remote ACK matching restores the locally expected incarnation, leaving the wire shape unchanged.
  • Inbox identity: stream/sequence/digest identity separates distinct signed transactions sharing a UUID.

Validation: 24 existing tests passed: 11 in dht::storage::tests::test_handoff, 9 in message::handlers::storage::tests::test_sync_ack, and 4 in swarm::application::relay::tests. The new bounded-metadata regression fails as described. Broader inbox/model suites were inspected, not all executed; no complete workspace or browser matrix is claimed.

Integration: a trial merge with #1248 conflicts in crates/core/src/dht/chord/storage.rs. Resolve it while retaining both incarnation updates and the SlowLockGuard instrumentation, without weakening the storage-transition critical section, then rerun the combined handoff tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant