Skip to content

fix(transport,core,node): harden transition_gate re-entrancy, bounded mail/provider shutdown, and async lock hygiene (#1231) - #1248

Merged
RyanKung merged 2 commits into
developfrom
fix/1231-transition-gate-and-async-locks
Nov 10, 2026
Merged

RyanKung merged 2 commits into
developfrom
fix/1231-transition-gate-and-async-locks

Conversation

@RyanKung

Copy link
Copy Markdown
Member

Summary

Closes #1231.

  1. Prevent transition_gate re-entrancy self-deadlock (crates/node/src/extension/ext/registry.rs):
    • Adds poll-scoped thread-local active-runner tracking (ACTIVE_RUNNERS) across native and wasm32 so same-task re-entrant Scope::inject / Route::dispatch / Route::observe_link fails fast with Error::ExtensionError instead of deadlocking on transition_gate.
  2. Decouple JsShell network sends from transition_gate (extension/protocols/{outbox.rs,js.rs}):
    • Introduces a bounded per-peer FIFO OverlayOutbox (PeerQuota + per-peer drain workers on NodeTasks) so JsShell::run enqueues outbound payloads without awaiting network I/O under transition_gate.
  3. Bound Mail Recipient shutdown & Provider::close task draining (mail/recipient.rs, provider/mod.rs):
    • Races Recipient::poll against both StopTokens and DEFAULT_MAIL_POLL_TIMEOUT (30s), and bounds Provider::close phases with CLOSE_DRAIN_TIMEOUT (5s) + structured warnings so shutdown resolves within bounded time even when storage/network stalls.
  4. Scope long-held async locks & add observability (slow_lock.rs, registration.rs, measure.rs, dns-relay/src/recall/mod.rs):
    • Adds SlowLockGuard (500ms threshold + metrics) and narrows lock scopes in DhtRegistrationPublisher::publish_lease and PeriodicMeasure read paths.
  5. Move Notifier::wake waker invocation outside the mutex (crates/transport/src/notifier.rs).

Test Plan

  • cargo test -p rings-transport --lib notifier (7 passed)
  • cargo test -p rings-node --lib extension:: (73 passed)
  • cargo test -p rings-node --lib mail:: (18 passed) & test_close (7 passed)
  • cargo clippy -p rings-transport -p rings-core -p rings-dns-relay -p rings-node --lib --tests -- -D warnings
  • cargo check -p rings-node --lib --target wasm32-unknown-unknown --no-default-features --features browser

Copy link
Copy Markdown
Member Author

Reviewed commit 4716c88b57f006e55acf5b7ce5c6858ffb758e6d. Verdict: two P2 findings need correction.

  1. [P2] Separate rollback state from query-visible committed state. At measure.rs:943–947, before can come from an earlier admission's stale committed_evidence_override. Sequence: A's store commits but its future remains pending; cancel A; B's store fails; B rolls back to the state before A; retry A. A review-only deterministic fake-store regression produces EvidenceAdmission::Admitted on that retry, losing the already-persisted receipt's replay marker. This affects provisional service-receipt evidence, not core transaction replay. Snapshot the actual pre-B evidence separately from the query visibility override, preserving uncertain earlier admissions across subsequent failures.

  2. [P2] Report store-release timeout as an error. At provider/mod.rs:380–386, the timeout helper discards the release outcome and can return Ok(()). A cancelled storage caller can leave a blocking operation holding an Arc<FileStore> beyond the five-second deadline. Consequently, close reports success while the directory remains locked, contradicting its documented reopen guarantee. Return a typed timeout carrying the release witness instead of discarding it. This is confirmed structurally through the blocking-store ownership path; no real-filesystem stall reproduction was run.

The broader review covered poll-scoped reentrancy detection, bounded per-peer outboxes, cancellation accounting, stop-token races, lock ordering, and persistence serialization. Existing targeted extension/mail/evidence/processor-close tests passed 102/102; the additional cancellation regression failed as described above.

Integration: pairwise merge checks found textual conflicts with #1239 in crates/core/src/dht/chord/storage.rs and #1243 in crates/dns-relay/src/recall/mod.rs. Preserve both PRs' storage/persistence semantics when resolving them, then rerun the affected tests. Shutdown changes also require semantic integration with #1244.

…down, and async locks (#1231)

- Detect same-task re-entry into Runner::transition_gate via poll-scoped active-runner tracking (native + wasm32) so re-entrant Scope::inject fails fast instead of self-deadlocking
- Decouple JsShell outbound network sends from transition_gate via bounded per-peer FIFO OverlayOutbox spawned onto NodeTasks
- Race Recipient::poll and Recipient::run against StopToken and per-poll timeout, and bound Provider::close / Processor::close shutdown drain phases when storage or network stalls
- Add SlowLockGuard wait/hold observability and narrow DhtRegistrationPublisher::held and PeriodicMeasure evidence query lock scopes while preserving lock-order invariants
- Drain and invoke Notifier::wake wakers outside NotifierState mutex
@RyanKung
RyanKung force-pushed the fix/1231-transition-gate-and-async-locks branch from b7b9d81 to 190ec33 Compare November 10, 2026 10:03
@RyanKung
RyanKung merged commit e156b0f into develop Nov 10, 2026
19 of 20 checks passed
@RyanKung
RyanKung deleted the fix/1231-transition-gate-and-async-locks branch November 10, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant