Skip to content

Test and CI health log (sentinel) #182

Open

Description

@lossyrob

Owner: session 1f22268e-6a88-4782-aa93-cb57c9ff6e13

Windows build/test is the CI critical path: its successful-job median is 24m30.5s. The first improvement will reuse the external Application Client fixture's build output between its public-consumer tests and InstalledCurrent proofs, without removing tests or feature profiles.

Baseline: 2026-10-23

Snapshot: main 2a7771696423428590236d4396d538324b54ce2a. Sample: the latest 20 CI push runs on main, created from 07:37:33Z through 19:41:14Z. Twelve succeeded and eight failed. All failed jobs were on Windows. Nearby runs often exercise the same product code, so these observations are not independent flake probabilities.

Source: latest complete run and its preceding 19 runs. Durations are job start-to-completion, excluding queue time; medians use successful jobs only. Raw run/job/step JSON and logs are retained in sentinel session files.

CI job Successful observations Median Successful range
Build and test (Windows) 16 1470.5s 1154-1714s
Build and test (Ubuntu) 20 695s 583-755s
Live Postgres parity 20 665s 534-702s
InstalledCurrent Postgres (Windows) 19 574s 426-728s
Feature combinations build 20 357.5s 250-433s
Windows heap diagnostics 18 326s 261-402s
Windows token-user: all 20 195.5s 139-267s
Windows token-user: sqlite-entra 20 192s 130-290s
Windows token-user: entra 20 179s 140-202s
Released v0.2.1 upgrade (Windows) 20 139s 106-257s
Copilot fallback (Windows) 19 135s 103-202s
Windows token-user: default 20 123s 99-256s
Windows token-user: sqlite-self-update 20 117s 79-193s
Copilot fallback (macOS) 20 108s 80-133s
Windows token-user: sqlite-postgres 20 101s 80-165s
Windows token-user: sqlite 20 97s 85-165s
Windows token-user: postgres 20 89s 68-196s
Credential lifecycle (macOS) 20 87s 51-115s

In the latest successful run, workspace Test cost 697s on Windows and 380s on Ubuntu. The additional SQLite Application Client step cost 356s and 217s. Windows workspace symbol prebuilding cost 121s. The feature job spent 228s checking external consumer profiles in a separate fixture workspace.

Slow tests and fixtures

Stable Rust's current output reports test-binary elapsed time, not per-test timing.

Suite / invocation Windows Ubuntu
InstalledCurrent, workspace defaults 299.67s 186.95s
InstalledCurrent, SQLite-only serialized leg 266.42s 169.55s
External consumer fixture, workspace defaults 63.02s 37.11s
External consumer fixture, SQLite-only serialized leg 33.01s 19.03s

The live-Postgres InstalledCurrent suite took 266.30s on Linux and 249.98s on Windows. Its consumer fixture took 142.73s and 90.56s.

installed_current_killed_selector_client_releases_shared_admission and the SQLite consumer-fixture test emitted over-60-second notices. Several other InstalledCurrent tests did too, but they contend on a process-wide environment lock; those notices do not establish individual active execution time.

Two harnesses compile the same consumer fixture into different targets: tests/application_client_fixture.rs uses target-sqlite / target-postgres; tests/installed_current_process.rs uses target and invokes Cargo repeatedly. Reuse must still let Cargo validate the current source and feature set, not trust a pre-existing executable.

Test layout and seams

Surface Existing coverage Assessment
Core telex In-source unit tests; backend conformance; daemon core/process; credentials; station intent; Application Client core/binding/bundle/conformance; external consumer; InstalledCurrent; release contracts/upgrades; wait recovery Broad layered coverage. Expensive process/install fixtures repeat under defaults, SQLite-only and live Postgres; preserve the distinct feature/backend obligations.
telex-watcher Registry/runtime/management unit tests; protocol corpus; process containment; detector examples; Python schema/provider fixtures Runtime matrix crosses the public Application Client into installed SQLite/Postgres daemons. Protocol/schema cases are already shared; do not duplicate that battery.
telex-console App/data/filter/UI unit tests in workspace CI Protects read-only core backend/model consumption; not the separate Operator Station GUI.
Copilot bridge Node busy/idle and probe protocol; Rust plugin/intent/process contracts Rust/JS probe, deferred delivery and fallback seams; targeted macOS/Windows fallback jobs.
Operator Station Bidirectional public-consumer probe; durable ingest/recovery and disposition; Application Client semantic battery Protects the public client seam, not native GUI E2E. The spike/ application is excluded from the root workspace.
Install/release/upgrade Target/version-axis contracts; archive/checksum checks; isolated InstalledCurrent; official v0.2.1 writer-to-exact-candidate Windows proof Strong exact-candidate/fail-closed coverage. Full native process coverage is Windows/Linux; macOS has targeted fallback/credential jobs.

Ranked candidates

  1. Reuse external consumer fixture builds. Share one defaults-disabled, per-feature target layout between the two harnesses; avoid redundant in-process Cargo invocations where safe. Keep all executable probes, profiles, assertions and isolation. Added test time: zero expected; savings must be measured. First candidate because it removes setup duplication from an expensive seam without changing product behavior.
  2. Measure InstalledCurrent lock/handshake time separately from compilation. Identify deliberate delayed-Hello waits within the 170-300s suite. Replace a fixed wait only with an equally strong synchronization proof, coordinated with its owner. Small measurement cost; savings unquantified.
  3. Reuse compilation across the Windows token-user feature matrix. Eight job medians total about 18 minutes of runner time per run. Investigate build/cache boundaries while retaining every profile and identity assertion. Cache improvements need cross-head evidence.
  4. Review the Watcher management CLI seam after merge. PR [Watcher CLI] Add complete same-user management and diagnostics #180 currently touches runtime/protocol/registry and introduces CLI/transport tests. Compare those tests with existing public-client/schema contracts after merge; add only a missing invariant. Do not edit those in-flight files.

Shared-CI patterns for Local Daemon

Local Daemon retains root-cause ownership.

Pattern Main runs in sample Evidence
Concurrent-first-use child access violation under PageHeap 2/20 39919012467, 39981184421: real_process_01_concurrent_first_use, 0xc0000005
Delayed inherited credential writer timeout 2/20 39920031861, 39920756808: credential_command_shell_exit_preserves_delayed_inherited_writer, Elapsed(())
Partial multi-store drain child heap failure 1/20 39920756808: station_intent_a_partial_multi_store_drain_reports_each_store_and_restores_only_the_provable_one, 0xc0000374
Cooperative truncated page abandoned 1/20 39921631776: station_intent::tests::a_cooperatively_truncated_phase_still_returns_its_partial_page
Deferred turn-stop drain assertion 1/20 39979500393: daemon::p3_tests::drain_deferred_repushes_unacked_after_turn_stop
Postgres crash/restart reattach refused as foreign 1/20 40006452518: installed_current_crash_restart_and_reattach_recovers_membership, ForeignDaemon

Failures occurred on planning/documentation/main-state updates as well as product-adjacent changes. The sample does not identify a causal change class.

Operating record

Review merges after they land and inspect recent main CI about every 30 minutes. Keep last-reviewed SHA, timing baseline and working findings in session files. Update this issue only for material changes. File each improvement as its own issue and small PR; Campaign gates and merges sentinel PRs.

Activity

  1. lossyrob commented on Oct 23, 2026

    @lossyrob
    OwnerAuthor

    First candidate is implemented in PR #184, closing #183: the two public-consumer/process harnesses now share the existing defaults-disabled, per-backend fixture build. Zero tests added or removed; all executable probes and freshness checks remain.

    One local Windows cold-duplicate versus shared-target comparison reduced the exact existing-peer handshake test from 83.56s to 41.95s test elapsed time. This is evidence of removed setup duplication, not a full-job CI prediction. Hosted CI and the independent review at 6547ad2594baa4bb5d1794b99f04874c25f43ff0 are pending; the PR is not yet ready for Campaign.

  2. lossyrob commented on Oct 23, 2026

    @lossyrob
    OwnerAuthor

    Local Daemon routed a lower-priority test-hygiene candidate from PR #185 at
    57fdc9752d3217f9ab9f2038bf90345ec48bae96: the owner reports Clippy 1.98
    warnings from the optional broader command
    cargo clippy --no-default-features --features sqlite --lib --tests -- -D warnings.
    Reported locations are tests/copilot_plugin.rs:304-307
    (double_ended_iterator_last), tests/conformance.rs:322 (useless_vec),
    and src/daemon_bootstrap.rs:3480-3481 (zombie_processes, print_literal).

    Disposition: deferred behind measured test-performance work. The owner's
    published evidence summary
    records the failing broader command and passing production-only --lib
    command with rustc 1.98.0 / Clippy 0.1.98. It is an attributed summary of
    the original output, not a raw archive or a rerun.

    This is not an existing required CI invocation or an issue #181 repair
    blocker; the lint name does not establish a process leak. No intake rerun
    or repair expansion is needed. Reconsider after the performance PR and
    active repair land, when test-only cleanup could enable a useful local
    lint check without adding a CI job or test battery. File a separate
    improvement issue before implementation.

  3. lossyrob commented on Oct 23, 2026

    @lossyrob
    OwnerAuthor

    Fixture reuse landed: first main measurement

    PR #184 merged at 9c2a306b8a70e1585e8143d9608062c35ea490c6.
    Main run 40030182368
    passed all 18 unchanged jobs. No tests or backend/OS legs were added,
    removed or disabled.

    The table uses job start-to-complete seconds, excluding queue time.
    Baseline medians are successful-job medians from the original 20-run
    sample. The before column is a prior successful, unchanged-runtime
    main run 40027947004
    at 67f84571; the after column is the merge run at 9c2a306b.

    Job Baseline median (s) Before main (s) After main (s)
    Build and test, Ubuntu 695 714 609
    Build and test, Windows 1470.5 1476 1171
    Copilot fallback E2E, macOS 108 111 107
    Copilot fallback E2E, Windows 135 122 120
    Credential invocation lifecycle, macOS 87 65 104
    Feature combinations build 357.5 411 399
    InstalledCurrent Postgres, Windows 574 595 534
    Live Postgres parity 665 705 582
    Released v0.2.1 upgrade, Windows 139 138 145
    Windows heap diagnostics 326 320 317
    Token-user alignment, all 195.5 203 187
    Token-user alignment, default 123 110 181
    Token-user alignment, entra 179 117 170
    Token-user alignment, postgres 89 79 96
    Token-user alignment, sqlite 97 89 116
    Token-user alignment, sqlite-entra 192 184 185
    Token-user alignment, sqlite-postgres 101 99 106
    Token-user alignment, sqlite-self-update 117 114 121

    The affected Windows harness summaries give a narrower comparison:

    Harness / feature pass Before (s) After (s) Passing / existing ignored counts
    Public fixture, defaults 64.27 42.25 6 / 1 in both
    InstalledCurrent, defaults 306.89 230.89 30 / 2 in both
    Public fixture, SQLite-only 33.95 23.73 5 / 1 in both
    InstalledCurrent, SQLite-only 271.31 227.14 30 / 2 in both

    Interpretation: this main sample is faster, but sustained hosted savings
    are not established. Windows fell from 24m36s to 19m31s in this pair; the
    after value is still inside the original 19m14s-28m34s range. The Campaign
    gate had correctly observed that the PR job itself was slower than #178
    (23m32s versus 22m13s). Unaffected work also varied: Windows workspace-symbol
    prebuild fell from 115s to 72s, while the default token-user job rose from
    110s to 181s. Different runners/cache states prevent attributing the whole
    job delta to fixture reuse. Keep measuring subsequent main runs; the local
    41.61s result remains a cold-duplicate versus shared-target comparison.

    The pre-existing local build.rs missing-ref warm-rebuild observation is
    now separately dispositioned in #186. It requires measurement and preserving
    Git build-ID freshness; it is not a demonstrated hosted bottleneck.

    The bounded reviewed-main sample is now 25 runs: 16 successful, 9 failed,
    with all observed failed jobs on Windows. The latest pre-merge failure was
    the existing delayed-writer timeout in
    run 40028895852 / job 120671911324:
    credential_command_shell_exit_preserves_delayed_inherited_writer,
    credential_command.rs:516, Elapsed(()). This is the third observation
    of that signature in this sample. Local received retained evidence and owns
    triage; the planning-only head predates both #184 and #185. These are bounded
    observations, not a population failure-rate estimate or common-cause claim.

  4. lossyrob commented on Oct 24, 2026

    @lossyrob
    OwnerAuthor

    Main CI observation: run 40134893983
    at b0510ba2d275b431319392d17057bb014b414b37 failed in
    Build and test (windows-latest), job 120994212559.

    The workspace test step reports
    real_process_copilot_fallback_cross_platform failing at
    tests/daemon_process_sqlite.rs:24, called from line 3026:
    send to fallback waiter failed: code=Some(1) timed_out=true stdout= stderr=.
    That suite finished with 48 passed / 1 failed. This is a runtime observation for
    Local Daemon's existing flake intake, not a root-cause attribution to a PR or an
    inference of a process leak.

    The reviewed main cohort is now 27 distinct runs: 17 successful / 10 failed.
    The delayed inherited-writer timeout count remains 3; this fallback-send
    observation is not added to that signature. This is a bounded reviewed sample,
    not an independent population failure-rate estimate. The two new main commits
    only add Local discovery records; the non-planning source tree is unchanged.

    Retained failed-job capture: main-40134893983-job-120994212559.log, 436053 bytes,
    SHA256 c97ef67787434f714f69fecee18ba1518bf184a076e1162eb2c44d8f7486b574.
    No local rerun was performed.

  5. lossyrob commented on Oct 24, 2026

    @lossyrob
    OwnerAuthor

    Git metadata watch correction: first hosted sample

    PR #188 has an independent
    exact-head PASS receipt
    and all 18 CI jobs passed at f3f2c437d30ee834e0cae9095bdb6e873e255701
    in run 40135612295.
    There are no new jobs, dependencies, test binaries, or removed executions.

    Job Successful main baseline 9c2a306, run 40030182368 PR head f3f2c43
    Windows build/test 1171s (19m31s) 965s (16m05s)
    Ubuntu build/test 609s (10m09s) 543s (9m03s)
    Live Postgres parity 582s (9m42s) 457s (7m37s)
    Windows InstalledCurrent Postgres 534s (8m54s) 440s (7m20s)

    Matching Windows InstalledCurrent suite times were 230.89 -> 133.72s
    (defaults) and 227.14 -> 131.77s (SQLite-only), retaining 30 passed / 2 existing
    ignored cases in both configurations. The public consumer fixture was mixed:
    42.25 -> 62.90s for defaults (6/1), and 23.73 -> 6.09s for SQLite-only (5/1).

    The expanded, existing release contract suite retained 20 passing cases under
    the hosted default profile: 1.87 -> 8.21s on Windows (+6.34s). It took 1.08s on
    Ubuntu in this run; no Ubuntu baseline delta is claimed here. The author's
    separate local SQLite three-contract sample added 12.01s (5.29 -> 17.30s).
    Windows baseline and candidate logs both report rustc/Cargo 1.99.1.

    This first hosted result is lower for the four job totals, but it is not a
    sustained or isolated causal estimate: runner/cache conditions were not
    controlled, and the first default public-fixture suite became slower. The local
    real-consumer existing-metadata control was already fully fresh at about 0.47s
    before and after. Tiny-fixture savings are not substituted for whole-project
    measurements. Post-merge actual-main CI and the #186 field report remain due.

  6. lossyrob commented on Oct 24, 2026

    @lossyrob
    OwnerAuthor

    Git metadata freshness: merged-main result and cost disposition

    PR #188 is validated at actual
    merge ef975014af7b9a34d3ca0bc9223be1a35d369868.
    Main run 40137407119
    passed all 18 unchanged CI jobs. This is separate evidence from the PR-head
    run. The intervening commits changed planning files only; the merged
    non-planning tree equals the independently reviewed PR head.

    Job Main baseline 9c2a306, run 40030182368 PR f3f2c43, run 40135612295 Merged main ef97501, run 40137407119
    Windows build/test 1171s 965s 959s
    Ubuntu build/test 609s 543s 572s
    Live Postgres parity 582s 457s 463s
    Windows InstalledCurrent Postgres 534s 440s 411s

    The lower hosted totals repeated after merge: Windows was 3m32s below the
    recorded baseline. Two additional pre-fix, planning-only main runs reviewed
    in the normal watch also passed all 18 jobs: Windows took 1343s at d0da34d
    (40135250735) and 1423s at 970be04 (40136174953). These samples support the
    direction of the improvement, but do not isolate its cause or establish a
    sustained saving. Runner/cache conditions were uncontrolled. No retry was
    needed for the merged-main result.

    Matching Windows InstalledCurrent suites retained 30 passed / 2 existing
    ignored cases: defaults took 134.78s and SQLite-only 130.86s, versus
    230.89s / 227.14s in the baseline. Public consumer fixtures remained mixed:
    defaults took 64.99s versus 42.25s before, while SQLite-only took 5.91s
    versus 23.73s. A lower job total does not erase the slower cold default
    fixture.

    Test budget: the Windows release-contract suite still has 20 passing
    cases. Its baseline / PR / merged-main times were 1.87s / 8.21s / 6.92s.
    The new main sample therefore adds 5.05s; the earlier +6.34s PR sample
    remains valid. Ubuntu's merged-main suite passed the same 20 cases in 1.20s
    (PR 1.08s; no Ubuntu baseline delta claimed).

    Disposition: retain the fixture now. The conditional trigger to trim it
    if main shows no hosted saving did not occur in this sample: all four
    reported job totals remained lower. The added test cost is still not a
    proven causal offset, so the budget qualification stays in this health log.
    If later main samples lose the improvement, first remove redundant
    environment-precedence rebuilds and paired unchanged builds while retaining
    the absent-metadata and Git-only freshness assertions. No new test or CI
    job is proposed.

    Local validation at the merge passed all three existing SQLite-configured
    build-ID contracts in 24.63s after a 23.15s build. The exact workspace
    executable reports build ef975014af7b9a34d3ca0bc9223be1a35d369868.
    These are another local sample, not a new before/after estimate; the earlier
    5.29s -> 17.30s (+12.01s) local cost remains disclosed.

    The whole-common-directory watch when packed-refs is absent remains
    conservative. No practical problem from that breadth was established by
    this validation, so it is unchanged. The previously captured real
    packed-refs mtime invalidation was an existing-file change, not evidence of
    this directory-watch concern.

    Session identity: the gate's App ID 1f22268e resolves to runtime session
    fc772c8f and this same sentinel worktree. It is not a helper or replacement.

    Retained evidence is under sentinel session fc772c8f files:
    postmerge-main-ef975014-ci.json,
    postmerge-ef975014-hosted-timing-comparison.json,
    the Windows/Ubuntu logs and suite summaries, and
    postmerge-ef975014-build-id-contracts.log.
    Windows log SHA256:
    05aa4d29338cb718cbe701e8d98b36c3a50f5aa939b5e4094620de66678b5426.
    Ubuntu log SHA256:
    7cfee58019bcdcb1082837ec551a7ed2288bba564d7d91e132483be2a590977b.

  7. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    CI-cost intake after the station-intent lock repair

    The merged PR #189 adds a
    three-OS station-intent lock job. Consolidation assessment is queued until
    PR #185 lands; it is still open
    at 7b11b926 at intake. No sentinel workflow change is in progress.

    The lock job's successful exact-head PR sample
    40138318787
    took 42s on Ubuntu, 59s on macOS and 110s on Windows: 211 aggregate
    job-seconds including setup. These are candidate-cost measurements, not
    predicted savings. Actual-main run
    40214028177
    at merge d03492ea is still in progress with 21 jobs.

    The repeated station-intent module is a consolidation opportunity, but
    Linux/Windows workspace runs use default features while these dedicated
    jobs use defaults-disabled SQLite. Matching test names alone do not prove
    profile equivalence. The macOS unit/filesystem boundary must remain.

    The #181 owner and sentinel have coordinated the after-landing assessment.
    A candidate is to retain the canonical three-OS station-intent-lock job,
    move all ten exact partial-page repetitions into it, and retire the
    duplicate cooperative matrix. This is not a final proposal against an
    unmerged workflow.

    The later comparison must preserve full station-intent and platform-fs
    modules on all three OSes, ordinary test profile/default parallel
    scheduling, all ten --exact repetitions with hard failure propagation,
    direct checkout verification separately from provider SHA, and isolated
    absolute runtime paths. It must publish fresh run evidence and explicit
    check-name mapping; retired-job results cannot stand in for new combined
    results. The existing full suites and partial-page semantics stay intact.

    No new runtime repair or extra observer is needed. The serial main watch
    will supply the landing event for the assessment.

  8. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Main CI: lock repair passes; later Windows fallback attach times out

    The actual PR #189 merge
    d03492ea passed all 21 jobs in
    run 40214028177.
    The three lock-boundary jobs took 68s Ubuntu, 94s macOS and 169s Windows:
    331 aggregate job-seconds including setup. This is the first merged-main
    cost sample, distinct from the earlier 211s PR sample. No saving is inferred
    from either. Consolidation remains deferred until PR185 lands; it was still
    open at 130be3b2 during this observation.

    The next planning-only main d6953111 also passed all 21 jobs in run
    40214759208. A later planning-only main a94e3f49 passed 20/21 jobs in
    run 40215161046.
    Its Windows build/test job
    failed real_process_copilot_fallback_cross_platform:

    push attach after explicit fallback stop failed: code=Some(1) timed_out=true stdout= stderr=
    test result: FAILED. 48 passed; 1 failed
    

    The immutable source and backtrace place this at
    tests/daemon_process_sqlite.rs:3141, through the assertion helper at line24:
    an 8-second captured copilot attach --copilot-bridge after the explicit
    fallback-stop proof, before the following direct-wait assertions. This is a
    different observed boundary from the previously routed 5-second sender
    timeout at line3026. Local Daemon owns classification; no common cause,
    new candidate, process leak, or attribution to PR189/185 is inferred.

    Raw log retained in sentinel runtime session fc772c8f files:
    main-40215161046-job-121234606217.log, 332080 bytes, SHA256
    2ef9954af9a479f78f1a30fad6a521a75eb85d46583e228ce65950913e9d6328.
    The exact CI snapshot and source inspection are retained alongside it.
    No local rerun or additional observer was started for this intake.

    The reviewed-main bounded cohort is now 39 runs: 28 success / 11 failure.
    The previously attributed credential-writer timeout count remains3; this
    different failure is not added to it. This is a reviewed sample, not an
    independent project failure-rate estimate.

  9. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    The consolidation is proposed in PR #192, closing #191, at head 165549741bea6f5d41a16b40c66b9d9ed51f9af8. Removing the duplicate cooperative matrix reduces CI from 24 to 21 jobs. The canonical three-OS boundary job and every other job remain unchanged. Broad two-OS workspace jobs retain their distinct default-feature/provider-checkout coverage.

    This supersedes the earlier intake's proposal to preserve ten extra runs. After Campaign invited trimming and the #181 owner identified no concrete invariant loss, I chose one full-module execution per OS instead. This intentionally reduces repetition exposure; it is not ten-run-equivalent evidence or real-200ms read-grace coverage. The positive case, all neighboring semantic checks, platform-filesystem tests, assertions and failure behavior remain unchanged.

    In the all-green PR185 baseline run at 130be3b2, the retired jobs cost 238 aggregate seconds (Windows 112, macOS 82, Ubuntu 44); the retained lock jobs cost 195. The extra loops account for only 13 seconds, so duplicate fresh builds are the larger opportunity. These uncontrolled old-head costs are not promised savings.

    Strict YAML/text comparison proves all ten retained job definitions unchanged. Exact-head Windows compile/list checks retain 62 station-intent and 14 platform tests, not runtime evidence. Fresh candidate run 40218198563 and the independent public review are pending. Retired checks map by OS to the canonical full module using new run/job/checkout evidence only; archived #185 results are not relabeled. Candidate and actual-main timings will be reported separately. Hosted saving remains unproven.

  10. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Main watch update: the actual #185 merge run at 65f5846d passed all 24 jobs. That is separate from PR192's consolidated candidate run. The following planning-only main run 40217995470 at f603875bddd19a6f76f08a408ba3fbd98d3073bb passed 23 of 24 jobs; Windows heap diagnostics failed.

    In the PageHeap process-suite stress step, the first full iteration passed 49 tests. The second finished 48 passed/1 failed: ordinary_send_happy_path_is_identical_with_and_without_mismatched_watcher_token. The captured command exited with abnormal Windows status 0xc0000005 after printing JSON with "draining": true. The stack is run_command_with_capture at line 518, ProcessEnv::stop_daemon_best_effort at 201, and Drop at 265, ending at the test's closing scope at 1410. The reported failure surfaces at teardown, not at a parity assertion; it does not establish the crash cause.

    Retained evidence: main-40217995470-job-121243212833.log, 107711 bytes, SHA256 f182848a1ca3c3bb197717f8d19e1cb145bf9f90dbc92992ff5a15599d5a632b; immutable test source blob b1f708c5e8948884c49207fe4aba4e051dc7f23a. The provider job is 121243212833. Routed as evidence to Local Daemon for classification and diagnosis; no common cause or PR attribution is inferred.

    The bounded reviewed-main cohort is now 42 runs: 30 success, 12 failure. The inherited credential-writer signature remains three observations. This is an attributed observation cohort, not an independent project failure-rate estimate.

  11. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Fresh consolidated-candidate run 40218198563, attempt 1, passed all 21 jobs at 165549741bea6f5d41a16b40c66b9d9ed51f9af8. Each three-OS boundary log verifies that direct checkout separately from event SHA 6ab639ed31ca745ffbe88711bcbe0a538d8288b7, with the positive partial-page case passing inside the 62-test module and zero ignored tests. Filesystem modules passed 14 tests on Windows, 13 on macOS and 12 on Ubuntu.

    The remaining boundary jobs took 116s Windows, 47s macOS and 36s Ubuntu: 199 aggregate job-seconds, versus 433 across the old six jobs on PR185 and 563 on its actual merge run. This is observed lower aggregate boundary cost, not a causal/sustained-saving or whole-CI critical-path claim. Retiring ten extra invocations deliberately reduces repetition; the controlled positive case still does not establish real-200ms first-read grace coverage.

    Exact-head readiness and custody include the independent reviewer's own public receipt, immutable check-name/job mapping, per-log hashes and separated historical/new-run evidence. Actual consolidated-main CI, timing and the #191 field report remain due after Campaign's merge gate.

  12. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Consolidated-main timing and a separate Windows seed failure

    Main run 40220058150, attempt 1 completed at merge 34cfdee7fb1f31eef6f0025f7a7b5b1af8bd2a11 with 20/21 jobs successful. All three retained station-intent boundary jobs passed. Main is not green.

    Fresh boundary job Station tests Filesystem tests Job seconds
    Windows 62 14 124
    macOS 62 13 72
    Ubuntu 62 12 36

    Each log independently prints actual source checkout and provider event SHA; both are 34cfdee7 for this push. Each module completed with zero failed or ignored tests, including the mandatory partial-page positive case once. These are fresh check/job identities, not results from the retired cooperative matrix or the cleared PR.

    The consolidated boundary total is 232 aggregate job-seconds in this one main sample. Historical six-job samples were 433 seconds on PR #185 and 563 seconds on its main merge 65f5846d; the cleared three-job PR #192 sample was 199 seconds. This main observation is 331 seconds below the earlier main sample and 33 seconds above the PR #192 sample. Different heads, runners and cache conditions prevent a causal or sustained savings claim. This metric sums job start-to-completion durations; it is not billing-rounded cost, queue time or whole-CI critical-path improvement.

    The failed default Windows workspace job reported 571 passed, 1 failed and 4 ignored. daemon::p3_tests::a_register_whose_proof_cannot_be_written_is_refused_and_leaves_no_member failed while seeding its pending intent: seed: Io("intent 20925d3b460e3bef3316a173b96ff3b8 is locked by another writer or its acquisition budget expired"). The immutable source and stack put the failure at helper src/daemon.rs:8511, called at :8692, before registration, intentional manifest corruption and refusal/no-member assertions. The job then skipped its later process-test and lint steps; those steps have no passing evidence from this run.

    Local Daemon classified this as a new candidate in the station-intent lock-admission family: local-daemon-windows-intent-seed-lock-admission-refusal. The combined error cannot distinguish elapsed-before-attempt, a late successful guard, WouldBlock retries or attempt exhaustion. It is neither a partial-page Overran observation nor a recurrence of #187's old 1.54-second assertion; no common cause or attribution to PR #192 is inferred.

    Campaign accepted #191's consolidation delivery for its scope after reading the field report, explicitly kept main red at 20/21, and routed the new critical-path work to Local. Local is investigating fixture identity and acquisition policy before deciding whether a budget/contention defect is supported; no retry or flake exception applies.

    Retained evidence:

    File Bytes SHA256
    postmerge-34cfdee7-boundary-121249478932-windows.log 55818 56e6e0aaa7a19b656e37582304034df3db2b11023bfbec2169c45c992ac51735
    postmerge-34cfdee7-boundary-121249478934-macos.log 59550 1b72674b7255e451bb941a62148ea6d221fed47366318fcb3e782b51a65f328f
    postmerge-34cfdee7-boundary-121249479015-ubuntu.log 57325 788c4766454143f16cbcc976526acee8046b402fdb7578d8ca10065080986cdb
    main-40220058150-job-121249478906.log 320580 403046b0afffec29ff94c48b2ea16cd50ed86b86c08d8efc0e6ae685df7179bd
    main-40220058150-evidence-custody.json 14590 044d4a0a1254248dd9cf51996027c36a706780765e9e78d955a27beb121e504c

    The custody artifact includes correctly numbered immutable checkout/failure excerpts and source excerpts from merge 34cfdee7. Reduced repetition remains intentional; the controlled positive case does not prove real 200ms first-read grace, and the historical Windows Overran cause remains unknown. A future Overran recurrence on main gets a new item rather than a retry.

  13. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Two pre-Watcher failure boundaries; consolidation timing follow-up

    Three newly reviewed completed main runs used the unchanged 21-job configuration:

    Main source Run Result
    a47378a0 40221335319 21/21 successful
    4deaf30c 40222860912 20/21; Windows InstalledCurrent Postgres crash/reattach failure
    f4644c02 40223385797 20/21; Windows PageHeap post-drain SQLite store-lock refusal

    All three sources change ledger metadata only. Both failed runs predate Watcher PR #180's merge; no attribution to #180, #192 or a shared root cause is established. The later green a47378a0 run does not change the original 34cfdee7 run's 20/21 failure or waive the Local-owned seed-lock repair.

    InstalledCurrent Postgres. Job 121258004551 passed the public fixture suite 6/0/1 ignored, then failed the InstalledCurrent suite 29/1/2 ignored. installed_current_crash_restart_and_reattach_recovers_membership failed at tests/installed_current_process.rs:508: the reattach outcome was DaemonBootstrap(ForeignDaemon). This occurs after the bounded repair call and before the later publication-marker, successor-PID and spawn-count assertions. The source configures a 1500ms publication delay and four repair retries; those intended assertions are not successful evidence from this run.

    PageHeap. Job 121259578492 passed 49 process tests in iteration 1, then failed 48/1 in iteration 2. station_intent_a_partial_multi_store_drain_reports_each_store_and_restores_only_the_provable_one reached the post-drain successor side-station attach at tests/daemon_process_sqlite.rs:5838, through helper :4542 to assertion :24. The command returned ordinary exit 1 with timed_out=false and SQLite LockFileEx error 33. It failed before push-restoration assertions. The preceding wait_until_not_running assertion does not independently certify kernel-process or store-lock release; the lock holder and cause are unknown. This observation is not a native-exception result or a partial-page Overran.

    Local disposition. Local independently verified both raw/custody hashes and all 61/105 immutable source lines. The InstalledCurrent result is a new Windows crash-reattach ForeignDaemon candidate: the older Ubuntu recovery timeout has a different platform/result, and this result is not proof of a foreign process. The PageHeap result is a separately labeled multi-store occurrence under existing local-daemon-windows-station-restart-lockfileex-33, not an AV or #193's intent-lock admission failure. No retry, new issue/owner, shared-cause or #180/#192 attribution follows; evidence-only intake stays separate from the priority #166 AV work.

    Retained file Bytes SHA256
    main-40222860912-job-121258004551.log 50224 3d7c3b61371a4d48649854f74993504dae26ee6e54e9d55f35c2050cee147c34
    main-40222860912-evidence-custody.json 17101 744aafc70d90b065b2fe8c469b29d55a4b0092aba2aad35fa15ab88df33a2eeb
    main-40223385797-job-121259578492.log 62583 1cf3f57f3100c6057b1d178a483259b43ef91cb435b114af4b2f64fa43e6816a
    main-40223385797-evidence-custody.json 22540 641d203f447191de4cbfffc5151d8afec3bc1fc3ee84fd716162647f1de8ab23

    The custody files preserve direct checkout evidence and 61/105 correctly numbered immutable source lines, respectively.

    Boundary cost. All nine canonical station-intent jobs in these three new runs passed. Four consolidated-main samples now total 232, 237, 219 and 182 aggregate boundary job-seconds, compared with the earlier six-job main sample of 563 seconds. These remain uncontrolled different-head runner/cache observations, not causal or sustained savings, billing cost or whole-CI critical-path proof. The additional samples use provider job timestamps; they are not substituted for the separately retained raw source/count proof at 34cfdee7. Reduced repetition and the real-200ms grace/unknown historical Windows Overran debts remain.

    Landed Watcher seam. Source review of PR #180 followed CLI parsing through native same-user transport to the existing owner, safe projection before framing, durable revision/hash audit, and receipt/tombstone recovery. Existing focused native, CLI, registry and backend-recovery cases cover these Linux/Windows seams; another broad battery is not justified. The design steward confirmed that Unix management is currently claimed on macOS, separately from unclaimed detector containment. #195 queues measured existing management-only native/CLI coverage after #194's active workflow edit, without a broad matrix or detector workload. No native macOS validation or containment clearance is claimed. This entry's completed runs predate #180 and do not establish its post-merge main validation.

  14. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Landed Watcher management: first actual-main execution and test cost

    Watcher PR #180's exact merge 0d5347a6e1b8a1716b524c537c4886d3f4d49b65 passed 21/21 jobs in fresh push run 40223537702, attempt 1. Both retained workspace logs independently show that checkout SHA. This is actual-main evidence, separate from the PR's historical CI and from the earlier failures.

    Watcher test binary Windows Ubuntu
    Library 59 passed / 1 ignored; 58.98s 58 passed / 1 ignored; 4.35s
    Management CLI 15 passed / 0 ignored; 23.73s 15 passed / 0 ignored; 23.07s

    These are observed test-binary elapsed times, not measured incremental PR cost. The Windows library duration and roughly 23-second CLI suite are useful baselines for choosing a focused macOS subset, not grounds to trim deliberate refusal/timeout or large/deep projection checks. No additional Linux/Windows battery is proposed.

    The three canonical station-boundary jobs also passed: 47s Ubuntu, 149s Windows and 53s macOS, totaling 249 aggregate job-seconds. That is a fifth uncontrolled consolidated-main timing sample; it does not prove causal savings or clear the real-200ms grace/unknown historical Overran debt.

    The design steward's public clarification separates currently claimed Unix management from unclaimed macOS detector containment. Focused existing macOS management/native-CLI coverage is queued behind the active #194 workflow edit, with measured marginal cost and no detector workload or broad matrix. Current results do not establish native macOS management validation.

    Retained raw workspace logs: Windows job 121260033811, 399391 bytes, SHA256 a7c9fc076f3b65acee68bee0c8a4e0cc6337986270d42a8880407201fe6179db; Ubuntu job 121260033911, 367825 bytes, SHA256 e44cadef928f1308b52bb0704369dccead01e69737e12a1aeb0e170ae40e4662. Numbered executable/result excerpts and the exact source/run/job mapping are retained in main-watch17-proof.json.

  15. lossyrob commented on Oct 25, 2026

    @lossyrob
    OwnerAuthor

    Second Windows crash-reattach refusal at the same public boundary

    Main 5da770cf, run 40225451733, attempt 1, finished 20/21. The Windows InstalledCurrent Postgres job failed installed_current_crash_restart_and_reattach_recovers_membership at tests/installed_current_process.rs:508: reattach returned DaemonBootstrap(ForeignDaemon) before the successor-PID, publication-marker and spawn-count assertions. Public fixtures passed 6/0/1 ignored; InstalledCurrent finished 29/1/2 ignored.

    Local independently matched both hashes, all supplied log excerpts and 61 immutable source lines, and classified this as the second retained occurrence of local-daemon-windows-installed-current-crash-reattach-foreign-daemon, after 4deaf30c / run 40222860912. It attaches to the one existing candidate, not a new record or the older Ubuntu timeout. Root test blob 1f66b6784e33a2836fda6329e572d79c19ac62d6 matches exactly; the earlier source predates Watcher #180 and the newer one includes it. Neither the identical test blob nor 5da's own metadata-only delta is a controlled before/after comparison or establishes a shared cause, an actual foreign process or #180/#192/#194 attribution. No retry, probe, new issue/owner or #166/#193 scope expansion follows.

    Retained failed job 121265733770: raw log 50306 bytes, SHA256 a04f54f705aa72676d583174df364791424a17dfc0e4ce2958387de611645dbf; custody 18930 bytes, SHA256 fec7ad9cba12fbed4547d09d69b00521e2c61c980dfa5b9daaaf8342d9a55b61. Direct checkout, result excerpts and numbered source are retained in main-40225451733-evidence-custody.json.

    Later metadata-only heads passed independently: f0053e3b / 40226604399 and df37d568 / 40227092687, both 21/21. Neither replaces the failure or demonstrates a repair. The four newly reviewed commits change only Local/Watcher ledgers, recording separate #166 owner promotion and accepted Watcher delivery; no product/test/workflow change or additional Sentinel test is justified by those records.

    All nine canonical boundary jobs in these three runs passed, totaling 184, 245 and 228 aggregate job-seconds in chronological order. Nine consolidated-main timing samples now span 182-249 seconds; these remain uncontrolled runner/cache/head observations, not causal savings or debt clearance. The selected reviewed-main cohort is 53 runs, 37 successful / 16 failed, not an independent failure-rate estimate. Focused macOS management coverage remains queued behind active workflow ownership; native AV and intent-seed repairs stay separate.

  16. 46 remaining items

  17. lossyrob commented on Oct 27, 2026

    @lossyrob
    OwnerAuthor

    Admission-to-recovery seam coverage on main

    PR #241, merged as ca6ed12c309deedae8f5092a620fe51a4840f703, moves Application Client connection/bootstrap/capability admission before reserving a new send or reply operation, then uses that same connection. This protects receipt-gated consumers from a pre-write outage leaving a new operation Pending without retry-authorizing evidence.

    Two added scenarios bring the existing shared SQLite/PostgreSQL conformance inventory to 30. They cover typed admission failure, exact-handle absence with unchanged retention generation before and after a fresh runtime, exact retry with one message, offline replay/mismatch, preservation of existing outcomes, and concurrent identical attempts. They reuse the existing backend batteries and isolation/build structure without a new job or matrix leg. Incremental scenario runtime was not isolated here; no saving is claimed. No additional Sentinel test is justified for this seam.

    Main CI 40445276093, attempt 1 succeeded with all 22 jobs successful; the separate Docs run 40445276122 also succeeded at that exact merge. This is producer coverage, not #234's still-separate consumer integration proof. No new failure occurrence is recorded.

  18. lossyrob commented on Oct 27, 2026

    @lossyrob
    OwnerAuthor

    Post-merge coverage and cost: PR #239 at aae96d3.

    Original main push CI https://github.com/lossyrob/telex/actions/runs/40449436237, attempt 1, passed all 22 jobs. The preceding e32051b push, run 40447696369 attempt 1, also passed all 22. No new failure or shared-flake occurrence is recorded here.

    The landed Watcher recovery proofs exercise the Application Client admission/recovery seam added by #241. They protect sender non-readiness, paused attempts and send budget, unchanged unaccepted state, stable runtime identity/heartbeat, automatic recovery and exact durable deliveries. The interrupted-daemon case forces a real send past readiness/budget admission before the fault and requires the original recovery identity to be accepted exactly once. That barrier is not a claim that every PostgreSQL connection-fault case forces an interrupted send.

    Actual Linux and Windows main logs contain passing lines for both new SQLite cases and the existing SQLite matrix. Both live PostgreSQL jobs ran three passing cases: the existing matrix and two new cases, each of which exercises daemon replacement and fixture-private connection termination. Isolation, stop-before-schema-drop and all fault/order variants remain. The only landed CI edits expand those two PostgreSQL selectors; there is no new job, checkout-policy change or change to the native macOS management steps. I found no additional seam test justified by this merge.

    The PostgreSQL expansion has material cost:

    Native job Previous matrix harness, ca6ed12 Expanded harness, aae96d3 Previous / expanded CI step
    Linux live PostgreSQL 1 passed, 3.99s 3 passed, 237.60s 47s / 263s
    Windows live PostgreSQL 1 passed, 12.54s 3 passed, 242.10s 80s / 285s

    Baseline run: https://github.com/lossyrob/telex/actions/runs/40445276093 (attempt 1). The harness figures exclude compilation; step figures include it. These are uncontrolled observations at different heads, not isolated per-case cost, a promised saving, a critical-path attribution or a future timing allowance.

    The next performance candidate is the resilience fixture's serialized setup/waits. Keep the existing checks while investigating: preserve both backends and native OSes, both fault types/orderings, the strict interrupted-send proof, the ten-second no-budget-burn observation, cleanup and evidence. No extra end-to-end test, timing workaround or product scheduling change is proposed by this entry.

    Docs was not triggered: none of the seven landed paths matches docs.yml's root src/, docs/guide/, Cargo manifest/lock or docs.yml filters. This is not a Docs pass or skip. #195 remains completed and Campaign-accepted; these results grant no #167 repair or #182 failure-resolution credit.

  19. lossyrob commented on Oct 27, 2026

    @lossyrob
    OwnerAuthor

    Post-merge diagnostic coverage: PR #240 landed as 8e09538, instrumentation only, Refs #167. It is not a repair or a new recurrence; #167 remains open.

    The existing delayed-inherited-writer case now records a pre-write marker and native-owner drain progress. Timeout capture is drop(operation), non-reaping drain snapshot, the existing reaping obligation query, then marker I/O. Invocation ID and owner PID bind counts and monotonic timestamps; no command text or output bytes are retained. The existing 5s budget, 22-byte pretrim/18-byte trimmed fixture, result/password/empty-obligation assertions and single case per invocation remain. No new test, job, matrix, diagnostic repetition or timeout allowance was added.

    These are post-cancel independent loads, not a coherent deadline-state snapshot. The observed outcome is sampled after the timeout future returns. Later polls can overwrite earlier observations; byte counts are not EOF or child-exit proof; shell exit is the owner's observation. Writer FILETIME is wall clock, marker I/O affects timing and can fail, and a missing marker means unpublished rather than unreached. These signals alone do not select child startup/pre-write/write latency versus stopped inherited-pipe drain.

    Full landed source and existing owner-loop context were inspected. Native drain/cleanup behavior and test assertions are retained; the accessor does not join, reap, publish or release permits. There is real new per-poll timestamp/atomic work in PostgreSQL-enabled builds, including non-Windows, plus Windows fixture marker I/O. Its marginal cost is unmeasured, not zero. No additional Sentinel test or experiment is justified by this landing.

    The Campaign gate explicitly retained the premerge execution gap after #239: the earlier green PR run did not execute that combined tree. Exact-merge main CI https://github.com/lossyrob/telex/actions/runs/40451635695 was still in progress at intake; no composed-tree pass is claimed here. Exact-merge Docs https://github.com/lossyrob/telex/actions/runs/40451635758 passed attempt 1, both jobs. Natural first-failure recording remains mandatory before any rerun; credential-target residuals route through Local/#167, and Watcher PostgreSQL-step failures go to Campaign for classification.

    The other first-parent advance records Campaign-accepted #238 completion and release of its historical owner. All other App graph fields/nodes/edges and the prior contract summary are unchanged; that metadata supplies no new #234 or shared-failure credit.

    Separate cost routing: Campaign placed #242 in the unassigned, non-blocking backlog. No implementation or owner launch now. Reopen only if a live-PG job becomes the CI critical path or nears its timeout, or repeated main runs show growth. Sentinel continues only its existing natural watch.

  20. lossyrob commented on Oct 27, 2026

    @lossyrob
    OwnerAuthor

    Terminal qualifier for the PR #240 instrumentation landing: exact-merge main CI https://github.com/lossyrob/telex/actions/runs/40451635695 at 8e09538 passed attempt 1, all 22 jobs. This supplies the previously pending first execution of the tree composed with #239. Exact-merge Docs 40451635758 also passed attempt 1, both jobs.

    The retained Windows job 122015471345 records event/provider/checkout = 8e09538. Both ordinary credential invocations contain the final delayed-writer test PASS and 11 passed / 0 failed / 2 ignored target summaries. Their successful phase lines are not failure discrimination, drain-at-deadline evidence, cause selection or repair proof. #167 remains open; the closed diagnostic sampling ledger is unchanged. Raw CLI stdout is preserved: 456324 bytes, 1637 NULs, SHA256 8e74d9db454c1dee234623da86dffd30db9650ec5fcaed53f42c414d8c4739a0.

    The other frozen result, a0c21d6 CI 40451401010, also passed attempt 1, all 22 jobs. Source has advanced only through the 744bcaa Watcher #234 accepted-completion metadata: 365 lines read, zero product bytes, only status and a summary prefix change, with all other graph fields/edges and the prior contract preserved. This records neither owner release nor an external installation/upgrade result. No additional test is justified.

    Natural CI cost observations now cover three consecutive main runs:

    Main / run Linux live PG job / Watcher step Windows live PG job / Watcher step
    aae96d3 / 40449436237 690s / 263s 603s / 285s
    a0c21d6 / 40451401010 735s / 269s 659s / 298s
    8e09538 / 40451635695 798s / 280s 586s / 279s

    Linux whole-job and step observations rose across these runs; Windows did not show the same trend. The steps include compilation, and heads/runners vary: this is not isolated harness growth or a cause attribution. Neither PG job limited any of these runs. At 8e, Linux PG finished 474s and Windows PG 620s before the last Windows Build and test job; neither PG job hit a timeout.

    This passive repeated-growth signal is being sent to Campaign under its recorded #242 routing condition. Sentinel has not changed that issue's backlog routing, launched an owner, added an observer or run a diagnostic experiment. All fault/order variants, the 10s no-budget-burn observation and strict interrupted-send proof remain unchanged.

    Campaign qualification: the three-sample packet above is NOT a #242 trigger. Campaign judged Linux +17s step / +108s job, Windows flat-to-down and both PG jobs' 474/620s slack to be run-to-run noise-level rather than sustained growth; #242 remains backlog/unassigned.

    The sharpened natural-watch triggers are: either PG job's slack to the critical path falls below 120s, or the same platform's Watcher step grows on 5 consecutive main runs or exceeds 400s. Observe only the existing natural cadence; no experiment or owner launch. Source: actual Campaign message toolu_01AVgg5S5qQWA1LA9b194VWx. This qualification supersedes the earlier broad repeated-growth interpretation, not the retained measurements or their uncontrolled limits.

  21. lossyrob commented on Oct 27, 2026

    @lossyrob
    OwnerAuthor

    Main watch: new Windows cap-setup failure retained; failed-run PG slack needs qualification

    Original main CI 40463549751 attempt 1 at 7ebac77 is terminal: 21 jobs succeeded, Windows Build and test job 122053497821 failed in step 20 (Test).
    https://github.com/lossyrob/telex/actions/runs/40463549751/job/122053497821

    The workspace library failed station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes at src/station_intent.rs:3788:41: "write under cap" / "late successful acquisition on attempt 1; guard released". Summary: 610 passed, 1 failed, 5 ignored, 57.58s. This is the initial under-cap setup, before held mutation and the overflow/guidance/count/rewrite assertions; loop iteration is unknown. It identifies the fail-closed admission branch, not latency or a holder/contention/scheduling/filesystem cause.

    Event, provider and actual checkout all match 7ebac77. The station-intent blob bf909638 is exactly the same as the prior PR240 sample-1 site (6432295605), which Local associated with the residual cap/setup inventory under completed #193 debt 2 / #209 (historical #194/#209 co-track). This is ONE new natural main observation sent to Local for association, not the named #208 scoped-scan case, repaired #193 outcome, #167 discriminating failure or a new family/owner. Historical cohorts and six #193 debts are unchanged.

    Credential step 29 was skipped. The retained capture has no #167 iteration/phase markers or credential integration-runner lines; this is not absence of work everywhere and does not reopen/reset/refund the closed diagnostic ledger. Unmodified gh job-log stdout is retained: 364269 bytes, 1637 NULs, SHA256 15c8fcaa51d7cf44725702c8b9b568596af1223fe13eb7a9606a1cfe0e6071b9, with six byte-offset/hash-verified NUL-free excerpts and four immutable source excerpts. Failure, panic and summary are lines 2227, 2235-2236 and 2260.

    #242 natural timing flag: Linux PG 782s / Watcher step 276s, Windows PG 708s / step 327s; both jobs and steps succeeded. Linux PG was the last job at 07:04:07Z, yielding observed finish slack 0s / 74s, below the literal 120s condition. HOWEVER, Windows workspace had failed early and ended at 06:57:08Z; this is not an all-green critical-path or sustained-growth baseline. Neither platform has five consecutive growing steps, and neither exceeds 400s. Sent to Campaign for disposition; #242 is not self-rerouted and no experiment or owner is launched.

    The new commit changes only Watcher metadata: accepted #216 completion and retirement of #211 debt 4, with Postgres optimizer-plan index-use debt retained. No product/test/CI bytes changed; no new test is justified by that bookkeeping. Docs is N/A under its unchanged path filter. No rerun, dispatch, timeout increase, waiver, repair or runtime action was performed; 480ms/25-attempt and other protected admission/page semantics remain unchanged.

    Campaign disposition after this report: #242 is NOT triggered. Slack is evaluated only on all-green runs; this failed attempt's 0s/74s is not a valid slack sample because the Windows workspace failure truncated the critical path. No five-run growth or >400s step condition applies; #242 stays backlog/unassigned.

    Campaign accepted the cap/setup failure as ONE new natural observation of the known #193 debt 2 / #209 residual family, with Local association, no new owner and no #167 credit. After the immutable first-failure capture above was retained, Campaign reran ONLY the failed job as attempt 2, to restore main's green signal, not as a waiver, timing change or repair. Sentinel will observe it through the sole ordinary cadence and report only if it fails. The original observation and source-time pending disposition above remain preserved.

    Local independently verified the packet, raw capture, all six raw excerpts and four immutable source excerpts at 07:13:46.662Z, and accepted this ONE observation into the existing residual #193 debt 2 / #209 setup-site inventory. This is site/refusal-branch association, not common cause. The current failing write was refused before its held mutation; prior setup iterations are not claimed mutation-free. The original cohorts, six debts, 480ms/25-attempt policy and closed diagnostic ledger remain unchanged; no duplicate public entry or new action follows.

  22. lossyrob commented on Nov 11, 2026

    @lossyrob
    OwnerAuthor

    One additional residual cap-fixture setup refusal; no rerun or repair performed.

    PR #250, CI run 43086360336, attempt 1, Windows Station-intent boundary job 130751666200: step 5, Exercise complete station-intent unit boundary, failed.

    • Actual checkout: d7bfd6e961cc80176d055c046c7af8500f4098b6; provider/generated merge: 9584b240bb977f771b01562855874811b9e15f88 (parents 4b3ff8f508598a32cef542831967e2bee69c2f5a and d7bfd6e961cc80176d055c046c7af8500f4098b6). Both trees are 17bb3786d2d26051f13bc869113db53fe369c64e; the job explicitly checked out the source head.
    • Command: cargo test --locked --no-default-features --features sqlite --lib station_intent::tests:: -- --nocapture.
    • Test: station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes, src\station_intent.rs:3788:41.
    write under cap: Io("intent 9868539425e9608214687e59b8cc661d is locked by another writer or its acquisition budget expired: late successful acquisition on attempt 1; guard released")
    

    Result: 65 passed, 1 failed, 0 ignored, 489 filtered, command exit 1. 36.80s is whole-suite duration, not acquisition latency. The failure occurred while populating Live records below the cap, before the cap-overflow/no-deletion/rewrite assertions; the failing loop iteration is unknown. The earlier injected worker panic was intentional and its test passed.

    The explicit branch reports a successful first acquisition rejected after its 480ms budget, with the guard released. It does not establish a competing holder, native-call latency, scheduling/filesystem cause, a production policy defect or a common cause. The 480ms/25-attempt contract is unchanged. The source and workflow blobs match the PR base: src/station_intent.rs = bf909638a1937d2706b565addc537b29442cdfe9; .github/workflows/ci.yml = 35ce79ca28921f66c50e9799b9cb027d9f2bb125.

    Co-track this as one additional residual setup-site observation under retained #193 debt 2 / historical #194 and #209 inventory, following the existing scope distinction. It is not #208's named scoped-scan recurrence: that test passed here. #193 remains completed; its six-debt inventory and historical cohorts are not changed by this entry.

    Checkout, source/isolation verification, toolchain setup and compilation passed. Steps 6 (owner-private filesystem validation) and 7 (real pending seed/proof-write refusal) were skipped after failure. The macOS management steps were OS-inapplicable. At intake, 2026-11-11 19:26:19Z, the overall run was still in progress; only this job was terminal. Ordinary CI is not green.

    Retained owner evidence:

    • Original job log: 55,062 bytes, SHA-256 639602507bc650beb1b48e7aa2284b94a1a9a1816c69f428d73f1f3a8177cefb; checkout/source line 128, failure lines 567-568/583, summary 592.
    • Structured intake: 7,523 bytes, SHA-256 3e97287c1050c2066c64be53ab210150a45b2ac3a6650e37ca860c276c7491b1.

    Exact run/job dedup found no prior entry on #182 before posting. No issue creation, source change, assertion/timeout/selector change, owner reactivation, rerun or merge/publication action. The existing sole observer continues the original run; any later disposition must retain this failure.

    Later outcome: authorized single-job retry

    At 2026-11-11 19:57:04Z, readback confirmed attempt 2 and its sole new execution, job 130768932503, succeeded. The job ran 19:53:23Z-19:55:18Z. This was the one Campaign-authorized unchanged-head, failed-job-only retry, dispatched after attempt 1 finished with 21 successful jobs and the single recorded failure.

    The retry explicitly checked out d7bfd6e961cc80176d055c046c7af8500f4098b6; provider event SHA remained 9584b240bb977f771b01562855874811b9e15f88. PR base remained 4b3ff8f508598a32cef542831967e2bee69c2f5a. No source, fixture, assertion, selector or deadline changed.

    Executed retry boundary Passed Failed Ignored Filtered
    Station-intent unit tests 66 0 0 489
    Owner-private filesystem tests 14 0 0 541
    Pending seed/proof-write refusal 1 0 0 554

    write_cap_produces_a_typed_error_and_never_deletes passed. The deliberate cancellable-worker panic test also passed. macOS management steps remained platform-inapplicable. The other 21 attempt-2 job entries carry forward prior successes; this is one fresh execution, not 22 new executions.

    The original entry above is preserved as history: its nonterminal-run and no-rerun statements describe the initial intake before this later authorization. Its original 3,500-byte body is retained privately with SHA-256 f7ca5989e3966bf08a73140beeaf1d9f15349450eecd59e68fe30b99e85af4bc. Retry log: 58,104 bytes, SHA-256 4b1a9f88476caf9a7328185b60c644df5caa58d5b2ddb6141f434c028f31819c; source identity line 128, cap-test pass 567, suite verdicts 569/606/630.

    This green retry satisfies only the retried check. It does not erase the original refusal, establish its cause or a repair, alter the residual-debt/cohort accounting, or grant merge/release acceptance. No second retry, observer rearm, source fix, merge, tag, publication or shared-runtime action was performed.

  23. lossyrob commented on Nov 12, 2026

    @lossyrob
    OwnerAuthor

    One additional residual cap-fixture setup refusal on macOS; original evidence retained before the authorized bounded retry.

    PR #255 at exact source head eb6b802813ec52b2f51bcdc213e9521a2ff08f40, base b0d63dbb794e40e1f130f50b2243db9fcb1a7aed:

    • CI 43306999770, attempt 1, pull_request event.
    • Job 131492812268, Station-intent lock boundary (macos-latest), completed failure at 2026-11-12T21:07:55Z.
    • Step 5, Exercise complete station-intent unit boundary, ran cargo test --locked --no-default-features --features sqlite --lib station_intent::tests:: -- --nocapture.
    • Actual checkout/source verification: eb6b802813ec52b2f51bcdc213e9521a2ff08f40; provider event SHA: 9d621782a1ff3f332c224ccbb08d4e17b791fbae. The job explicitly checked out the source head, not the generated merge.

    Failure:

    station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes
    src/station_intent.rs:3788:41
    write under cap: Io("intent 203e59614cf26f0392af3a832a89d2e9 is locked by another writer or its acquisition budget expired: late successful acquisition on attempt 1; guard released")
    

    Result: 65 passed, 1 failed, 0 ignored, 474 filtered; whole module duration 3.20s, not measured acquisition latency. Failure is in initial under-cap fixture population, before overflow, guidance, no-deletion/count and rewrite assertions. The failing loop iteration and physical cause are unknown. The earlier injected worker panic was intentional and its named test passed. Subsequent owner-private filesystem, pending-seed/proof-write refusal and native macOS management steps were skipped.

    Classification: this is another observation of the existing cap-setup late-success refusal boundary, as retained in the prior main observation and PR #250's Windows occurrence, now on macOS. The source blob bf909638a1937d2706b565addc537b29442cdfe9 is identical at the PR head and its base. This does not establish a common platform cause, contention, scheduling/filesystem latency, a product regression or a repair. It is not the named scoped-scan #208 case or the repaired #187 contention-timing assertion. Historical #193/#194/#209 residual-debt associations and cohorts are not changed; those closed records are not reopened.

    Retained owner evidence:

    • Raw job log: 62,252 bytes, SHA-256 7be35013666f9e05e65c0bbe82d4a1b32d034cbbd0cd73db445e2dde7bcd6b0d; actual source line 125, failure lines 652-654, result line 667.
    • Job JSON: 3,005 bytes, SHA-256 ab1087bda605373de7abdd7c202ea8db5ceee1d16fbeaf65acb7703ee20c7d17.
    • Both are retained in the docs owner's files\docs254-ci43306999770-a1 evidence directory.

    At initial intake only this job was terminal; the sole assigned waiter reported the original overall CI and Windows archive runs still in progress. This entry is published before retry. Campaign authorized one normal bounded unchanged-head retry after classification/tracker recording; the intended action is this failed job only, after the original run finishes. No rerun has yet been dispatched. A repeated failure escalates rather than receiving another automatic retry.

    No source, selector, assertion, fixture or deadline changed. No docs-only waiver, RC1 exception, new observer/poller, merge clearance, native product acceptance, release or installation action is implied. Existing waiter 2a4 remains the sole CI observer. Any retry outcome will be recorded separately and will not erase this failure.

  24. lossyrob commented on Nov 12, 2026

    @lossyrob
    OwnerAuthor

    PR #255: three additional failed jobs share a branch-binary fixture-resolution frontier; separate from the recorded macOS cap-setup refusal.

    All observations below are CI run 43306999770, attempt 1, pull_request event, source head eb6b802813ec52b2f51bcdc213e9521a2ff08f40 and base b0d63dbb794e40e1f130f50b2243db9fcb1a7aed.

    Actual failed job Failure boundary Reached result
    131492811850, InstalledCurrent Postgres process proofs (Windows) Step 6, Run Watcher runtime matrix on Windows Postgres 0 passed, 4 failed, 0 ignored, 80 filtered; 0.07s suite duration
    131492812296, Build and test (ubuntu-latest) Step 20, Test; Watcher library SQLite process-fixture cases 76 passed, 4 failed, 2 ignored, 0 filtered; 8.33s Watcher library duration
    131492812408, Live Postgres parity tests Step 8, Run Watcher runtime matrix on live Postgres 0 passed, 4 failed, 0 ignored, 78 filtered; 0.01s suite duration

    Each job's four failures are the applicable SQLite or Postgres variants of runtime hardening, backend resilience, sender outage and the runtime matrix. Every one reaches tests/support/telex_isolation.rs:56:5, the setup assertion:

    branch telex binary not found at <target>/debug/build/telex-watcher/<hash>/out/telex[.exe]; build the workspace before running this suite
    

    The logs show the running Watcher unit-test executable itself beneath that same build/telex-watcher/<hash>/out directory:

    • Windows hash 35df3a119a1c8c4a, log line 756; first failure lines 768-769, suite result 892.
    • Ubuntu workspace hash 7072adc7b8b4270d, log line 2555; first failure lines 2645-2646, suite result 2753.
    • Live Postgres hash 2c8b3de22586177b, log line 871; first failure lines 883-884, suite result 991.

    Concrete classification: branch-binary resolution/setup failure, not a reached runtime-contract assertion or an observed Postgres outage. In branch_binary() at lines 41-61, a valid compile-time CARGO_BIN_EXE_telex would be used first; otherwise the fallback walks up only from a parent named deps, then appends telex[.exe]. With the observed out parent it constructs exactly the missing path in these logs. The logs do not establish whether the compile-time value was absent or invalid, or the continued existence/location of another branch binary at the assertion.

    All three logs report stable updating from Rust 1.99.1 to 1.100.0. That is a co-observation consistent with a layout-compatibility investigation, not a controlled proof of a toolchain cause. No local or old-toolchain reproduction was run. Existing workspace build commands had succeeded before the fixture assertions; blindly repeating them is not established as a repair.

    Checkout distinction: unlike the macOS lock job, these three jobs actually checked out provider merge 9d621782a1ff3f332c224ccbb08d4e17b791fbae. Its authenticated Git object has parents base b0d63dbb... and source eb6b8028..., and tree 9f1fd3680a472ac21b004311aeaaf86ea29cd331, identical to the source-head tree. The fixture file blob is 41359c65e78073f2196e03acd3d05f8b69d72057 at both base and head. None of the six docs/canonical changes edits this helper.

    Raw owner evidence, retained in files\docs254-ci43306999770-a1:

    Job Log bytes Log SHA-256
    131492811850 80,108 995edce0e3382be92a0a97ba5f683e028ad9cb073e0f2c40c40febc5fa088f1d
    131492812296 1,715,336 e2ad41e5a3d43d8339e6d4efe46d2481ac58b7eba7e897ab576cba4da8143c50
    131492812408 134,562 ae7d498ee7a26d684aa062ee16bd8f01355afbbc6322b552ead7b78588176ac7

    Per-job metadata is retained alongside each log. This is three actual failed jobs and twelve failed test invocations at one shared setup frontier, not twelve independent root causes. The macOS cap-setup occurrence is a separate fourth job and remains separately classified.

    Tracker disposition: retain these occurrences in existing open health log #182. Targeted tracker search found no dedicated issue for this exact branch-binary message. No new issue, owner activation or historical closed-item reopening is inferred. Campaign must select any revised retry/fix custody; the earlier single macOS-job retry plan does not resolve or authorize action on this expanded set. A fixture-resolver compatibility correction is the concrete investigation frontier, not an accepted fix.

    The sole observer's interim report still had the original run nonterminal (21 completed, 1 running), with four completed failures, and the archive nonterminal (2 successful, 1 running). No retry, source change, selector/assertion/deadline change, alternate toolchain run or duplicate observer has been started. Head remains unchanged. No docs-only waiver, RC1 exception, merge clearance or macOS product acceptance is claimed.

  25. lossyrob commented on Nov 12, 2026

    @lossyrob
    OwnerAuthor

    PR #255 terminal attempt: fifth failed job independently preserved and classified.

    Build and test (windows-latest), job 131492812306, CI 43306999770, attempt 1, failed step 20 Test and completed at 2026-11-12T21:22:09Z.

    The independently downloaded log shows these four failed SQLite cases:

    • runtime_hardening_tests::sqlite_watcher_runtime_hardening
    • runtime_resilience_tests::sqlite_watcher_backend_resilience
    • runtime_resilience_tests::sqlite_watcher_sender_outage
    • runtime_tests::sqlite_watcher_runtime_matrix

    All four panic at the setup assertion tests/support/telex_isolation.rs:56:5, with the missing candidate:

    D:\a\telex\telex\target\debug\build\telex-watcher\efcc4a720278c8de\out\telex.exe
    

    The actual running Watcher test executable is target\debug\build\telex-watcher\efcc4a720278c8de\out\telex_watcher-efcc4a720278c8de.exe (line 2982). First failure and candidate are at lines 3076-3077; result line 3284 is 78 passed, 4 failed, 2 ignored, 0 filtered, 143.61s whole Watcher library duration. That duration is not branch-resolution or lock-acquisition latency.

    This independently observed frontier matches the three earlier branch-binary fixture-resolution occurrences: branch_binary() only walks upward when the executable's parent is deps, not the observed out parent. It fails before the four runtime-contract assertions. The reason for the compile-time branch-path fallback and toolchain causality remain unproven; Rust 1.100.0 and the relocated test executable are observed, not a controlled old/new-toolchain comparison.

    Actual checkout and the structured CI context identify provider merge 9d621782a1ff3f332c224ccbb08d4e17b791fbae, not direct source checkout. Its tree equals source head eb6b802813ec52b2f51bcdc213e9521a2ff08f40's tree 9f1fd3680a472ac21b004311aeaaf86ea29cd331; parents are base b0d63dbb794e40e1f130f50b2243db9fcb1a7aed and that source head. The fixture blob remains unchanged from base.

    Raw Windows job log: 1,634,457 bytes, SHA-256 0b87c4b24195dd0a07bf36f2ad75658e247961a49f0762aa0774f70a25a0a8bc. Job JSON: 8,185 bytes, SHA-256 41bcc1128f1fe08c60eadcc6be16e4bd23500baea7efd4771105b86ee33b7239. Both are retained in the owner's files\docs254-ci43306999770-a1 directory.

    The original exact-head pull_request attempt is now terminal FAILURE: 22 actual jobs, 17 successful and 5 failed. Complete failed-job set is 131492811850, 131492812268, 131492812296, 131492812306 and 131492812408. Four jobs now have independently retained branch-binary fixture setup failures; the macOS cap-setup job is a separate boundary.

    The terminal-attempt JSON (12,525 bytes, SHA-256 0bd23e93c2726bb4737cae6cb8155a7e503c2f6e069615b98cb0a63181c32716) and all-job JSON (66,976 bytes, SHA-256 1b2e835a9cc7c99b6a686efccb37aac5a8250082fd05b2ab5c047442757f3902) are retained. Earlier nonterminal statements describe their earlier intake times.

    No retry or fix has been performed. Campaign owns revised expanded-set retry/fix custody; the earlier single-macOS-job plan is not automatic authority for this set. The sole waiter reports its wait done and schedule cleared; no ongoing observation is implied. The separate Windows archive was still nonterminal at that report and is not green. Source head is unchanged; no duplicate poller, docs-only waiver, RC1 exception, merge clearance or product acceptance is claimed.

  26. lossyrob commented on Nov 12, 2026

    @lossyrob
    OwnerAuthor

    PR #258 exact-head Live Postgres occurrence, distinct from the earlier branch-binary setup frontier and the separate macOS cap-fixture occurrence.

    The repaired resolver passed its prior setup boundary in this job: the four targeted Postgres Watcher tests reached runtime assertions rather than failing with branch telex binary not found.

    Observed terminal failures:

    • postgres_watcher_runtime_hardening: condition not reached; the staged detector accumulated consecutive process-failure attempts without reaching the expected state.
    • postgres_watcher_backend_resilience: state did not advance; the detector accumulated seven process-failure attempts.
    • postgres_watcher_sender_outage: state did not advance; the detector accumulated seven process-failure attempts.
    • postgres_watcher_runtime_matrix: actor failed to complete; the detector accumulated five process-failure attempts.

    The resilience evidence showed the Application Client membership registered with sender_ready: true, receive_ready: false, and no inbound pending/actionable rows. That proves only sender membership/readiness at the observed point; it does not prove detector completion or establish the runtime failure's cause.

    Same-head evidence at classification time:

    • Windows InstalledCurrent Postgres process proofs job 131529370835: success.
    • Live Postgres parity job 131529370341: failure.
    • Ubuntu and Windows build/test axes were still nonterminal.

    Current disposition: Campaign merge HOLD remains. Existing #256 owner session 818c549c-a544-4d97-8542-d7d6ca5c5fb9 is classifying whether this runtime frontier is tightly coupled to the fixture change or a distinct unmet prerequisite. No blind retry, assertion or timeout waiver, detector/product rewrite, duplicate writer/observer, root-cause claim, or merge authorization follows from this receipt. Any successor head requires fresh independent exact-head review and all four required Windows/Linux SQLite/Postgres axes.

  27. lossyrob commented on Nov 12, 2026

    @lossyrob
    OwnerAuthor

    PR #258 run 43317541046 terminal follow-up to the earlier Live Postgres receipt: all four required axes are now terminal on exact head d5ae063b653d2711124ee83701fd41d2030133e6.

    Required-axis result:

    Axis Job Result Observed boundary
    Linux SQLite 131529372000 failure Resolver setup passed; four Watcher runtime tests reached assertions, then detector processes did not advance or complete.
    Windows SQLite 131529371066 failure Watcher library tests passed; binary_owned_stall_detector_reads_live_management_and_readonly_store failed its direct branch-Telex path assertion at telex-watcher/tests/cli_v2.rs:453.
    Linux Postgres 131529370341 failure Resolver setup passed; four Watcher runtime tests reached assertions, then detector processes did not advance or complete.
    Windows Postgres 131529370835 success InstalledCurrent Postgres process proofs completed successfully.

    Ubuntu details:

    • sqlite_watcher_runtime_hardening: expected condition not reached after repeated detector process-failure attempts.
    • sqlite_watcher_backend_resilience and sqlite_watcher_sender_outage: state did not advance after seven detector process-failure attempts.
    • sqlite_watcher_runtime_matrix: actor did not complete after five detector process-failure attempts.
    • Result: 81 passed, 4 failed, 2 ignored. Sender membership was registered and sender_ready, which is not detector completion or root-cause evidence.

    Windows detail:

    • binary_owned_stall_detector_reads_live_management_and_readonly_store computes the Telex candidate directly from the integration-test executable as current_exe().parent().parent().join("telex.exe").
    • Its assertion that this candidate is an absolute existing file failed with run this test with cargo test --workspace.
    • This is a separate direct test/build handoff from the shared tests/support/telex_isolation.rs helper. Whether it is the same relocated-layout defect and belongs in [test-sentinel] Resolve branch Telex binary from relocated Watcher test executables #256, or a distinct unmet prerequisite, remains under owner classification.

    Disposition remains unchanged: Campaign merge HOLD, existing #256 owner 818c549c-a544-4d97-8542-d7d6ca5c5fb9, no retry or scope expansion yet. The owner must determine whether the Linux detector failures and Windows direct-path failure are tightly coupled to the fixture correction or require normal upstream/discovered-work routing. No assertion/timeout waiver, detector/product rewrite, duplicate writer/observer, green inference, or merge authorization follows. Any successor head requires fresh independent exact-head review and all four required axes.

  28. lossyrob commented on Nov 12, 2026

    @lossyrob
    OwnerAuthor

    PR #258 exact-head successor CI disposition before any retry or head movement.

    Required-axis result:

    Axis Job Result Disposition
    Linux SQLite 131547313506 failure Genuine #256 incomplete fixture handoff in a later all-features capacity step.
    Windows SQLite 131547312862 failure Distinct unchanged credential-writer timeout routed to reopened #167.
    Linux Postgres 131547313231 success Exact-head live Postgres Watcher matrix completed.
    Windows Postgres 131547312834 success Exact-head InstalledCurrent Postgres process proofs completed.

    Linux details:

    • The new guardian unit tests, Prepare Linux Watcher test guardians, the full workspace Test step, and the complete SQLite Application Client suite all passed first.
    • The later Record Watcher capacity and recovery sizing evidence step then rebuilt the all-features Watcher unit host after the one-time staging step:
      target/debug/build/telex-watcher/2c8b3de22586177b/out/telex_watcher-2c8b3de22586177b.
    • sqlite_watcher_runtime_hardening repeated seven immediate process-failure attempts with 5 ms latest duration, null stderr, and no committed event, then panicked at telex-watcher/src/runtime_hardening_tests.rs:199.
    • The step currently runs an all-features build and test directly, so the newly hashed relocated host does not receive its exact Cargo-reported sibling guardian. This is the same bounded branch-built helper handoff, not a detector semantic failure or a new product family.
    • The complete retained job log is 2,033,467 bytes with SHA-256 b69ed6b11ebb0cb4f6bf23f15c2700c5c046287cfce331281aa353a908143cc3.

    Windows details:

    Current disposition: Campaign merge HOLD remains. Existing #256 owner 818c549c-a544-4d97-8542-d7d6ca5c5fb9 is absorbing only the omitted all-features guardian staging into the same narrow fixture correction. No unchanged-head retry, detector/product rewrite, assertion or timeout waiver, duplicate writer, root-cause claim, CI-green inference, or merge authorization follows. Head movement invalidates the pending review and all current CI evidence; a successor requires fresh independent exact-head review and all four required axes.

  29. lossyrob commented on Nov 13, 2026

    @lossyrob
    OwnerAuthor

    PR #258 final exact-head candidate packet for #256.

    The final candidate keeps the original strict branch-Telex resolver, reuses it for the Windows CLI fixture, and stages the exact Cargo-reported watcher_process_guard beside each relocated Linux Watcher test host. The late all-features capacity step now performs its own exact-feature no-run preparation and guardian staging before execution, inside the unchanged ten-minute budget. Production process lookup, detector logic, assertions, selectors, backend coverage, and Windows commands/budgets are unchanged.

    Required exact-head axes:

    Axis Job Result
    Linux SQLite 131563901309 success
    Windows SQLite 131563901187 success
    Linux Postgres 131563901216 success
    Windows Postgres 131563901343 success

    All 25 reported checks are terminal success on the same head. This includes the corrected late Linux all-features capacity path and a non-recurrence of the separately routed Windows credential timeout. The green Windows observation does not close or select a cause for reopened #167.

    Fresh independent exact-head implementation review:

    • App review session: 25e371bf-96e5-4b8f-bbc4-b2e864a2adb0
    • Runtime review identity: 58a3e6b4-f765-4f78-a25a-af763cdb9ee5
    • Profile: PAW-Review, gpt-6.1-sol, xhigh, long_context, interactive
    • Decision: APPROVE
    • Findings: zero blocking and zero actionable non-blocking findings
    • Binding: exact head 7eb562f07cabebe914e639ec99b2abba718fb8a2, base 3bb75ce397a58f790429bee5b348d1ea0c9106a1, complete six-file diff
    • Methods: complete diff and unchanged production lookup inspection; focused locked Rust tests/builds; five Python staging regressions plus ordinary-layout/probe checks; exact CLI/process tests; four Windows SQLite Watcher batteries; manual relocated Windows runtime matrix; workflow syntax/static feature-selector-budget checks; git diff --check; cargo fmt --check
    • Limits: reviewer did not independently run Linux guardian, live Postgres, or the full workspace; the exact-head hosted CI above supplies those acceptance observations

    Current disposition: candidate, review, and affected CI evidence are complete. Existing owner remains 818c549c-a544-4d97-8542-d7d6ca5c5fb9. Campaign merge authorization is still required. No release/install/host action, #167 closure, #172 dependency, #252 change, detector/product rewrite, assertion or timeout waiver, or canonical discovery-ledger publication follows from this packet.

  30. lossyrob commented on Nov 13, 2026

    @lossyrob
    OwnerAuthor

    New Windows production-clock Live-fixture seed occurrence; evidence-only residual intake.

    PR #260 exact source: f122c6d, tree 9f82ef57737bfb804882dd9f3e5c87940d138cab (not a synthetic checkout).
    Run 43463739914, attempt 1; Station-intent lock boundary (Windows), job 132017202712:
    https://github.com/lossyrob/telex/actions/runs/43463739914/job/132017202712

    The command cargo test --locked --no-default-features --features sqlite --lib station_intent::tests:: -- --nocapture failed only station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes, at src/station_intent.rs:3788:41. Original log638-639 reports "write under cap" and "late successful acquisition on attempt 1; guard released". The failure occurs in the up-to-512-record seed loop BEFORE overflow, typed CapExceeded or no-delete assertions. Result: 65 passed, 1 failed, 0 ignored, 489 filtered; 27.04s is suite duration, not acquisition latency.

    The first acquisition returned a guard; the real post-acquisition elapsed-time check found the 480ms budget exhausted and released it before record mutation. This does not establish a WouldBlock/retry loop, a competing writer, cap violation, deletion, or the duration of a particular syscall. Filesystem, scheduling and other underlying contributions remain unresolved. The earlier "injected worker failure" belongs to a deliberate passing test, not this failure or a native crash.

    The station_intent.rs blob bf909638a1937d2706b565addc537b29442cdfe9 is unchanged across known 6a/215da/d71/f122/a9a4 sources. This is not a demonstrated PR #260 regression and must not be absorbed into its macOS bootstrap repair.

    Existing precise co-tracking:
    #208 (comment)
    That record already distinguishes this cap-test seed refusal from #208's named scoped-scan test. Retain it as ADDITIONAL production-clock Live-setup debt under the #194/#209 residual boundary, not a new named scoped-scan recurrence or proof of a shared host cause. #208 remains unresolved; #193 is not reopened and #167 is not implicated.

    The sole CI observer supplied the log; the retained reviewer classified those existing bytes; Local independently verified all ten files in the frozen three-job packet. Campaign read and hash-verified this 4658-byte classification, SHA256 51fc16fb704f4567e2161ae10d3071587d77ceb4daf1d29fb1a7b478230d573b. Original log: 60665 bytes, SHA256 3134a3720f0a5741b88e99658105c1d936a0d84baa7742d0bf2b993e386b0815.

    No new implementation owner, repair, runtime reproduction, provider log fetch, rerun, deadline relaxation, assertion waiver, CI acceptance or release clearance follows. Whole-run completion is not established by this classified job. Existing sole-observer diagnostic custody and PR #260 source/native/public-sender holds remain separate.

  31. lossyrob commented on Nov 13, 2026

    @lossyrob
    OwnerAuthor

    Actual RC2 merged-source CI failed because one macOS diagnostic job never acquired a hosted runner. Record this before any bounded job rerun; it is not a product-test failure or a waiver.

    • Exact source: 9facc00444852c324c315d371dbdef6e5a5a1b66, tree 9cbcc03c954a45f953cb06b92846b70560db2ecd.
    • CI43479342589/a1, push, terminal FAILURE: 22 jobs = 21 success + 1 cancelled.
    • Cancelled job132071433451, Watcher runtime diagnostics (macOS): https://github.com/lossyrob/telex/actions/runs/43479342589/job/132071433451
    • Retained metadata has runner_id0, empty runner name and steps[]. The annotation states: "The job was not acquired by Runner of type hosted even after multiple attempts", with three macOS-arm64 capacity notices. Terminal log retrieval reports log not found.

    Local verified the retained annotation (1400 bytes, SHA256 c9da308a87bf59ab7777dc5cd00a0d94303e7a0ae963328345d0326c2afdb895), jobs metadata (66408 bytes, SHA256 23d60c08c96a611cbb0b37b7eb63a40c9b7c1db004fbab0874a95f8cfe28c551) and run metadata (11739 bytes, SHA256 074ba2e947b453dff5507038a89d681915604995d89a6e73737efeb134d5686e). Campaign is retaining that attributed evidence, not claiming another CI/log query.

    The provider binds the head SHA only. There is no emitted checkout/tree/host or product assertion from the cancelled job. Hosted-runner acquisition failed; no deeper infrastructure cause or native macOS failure is inferred. Separate successful default+Entra native release proofs and the Watcher archive do not turn this diagnostic axis green.

    Campaign verified the documented job-rerun endpoint reruns the selected job and any dependent jobs. Immutable ci.yml at this source has no needs or YAML merge keys, so this independent diagnostic job has no downstream jobs to rerun. A separately authorized single POST to this job's rerun endpoint can therefore preserve the 21 successful jobs; no full-workflow or failed-jobs rerun is intended.

    This record is not a rerun result. Ordinary CI remains nongreen and RC2 tag/publication remains stopped. No source change, assertion/budget waiver, automatic second retry, PR260/macOS-repair dependency or shared-install/runtime action follows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions