Repository navigation
Description
Owner: session 1f22268e-6a88-4782-aa93-cb57c9ff6e13
Windows build/test is the CI critical path: its successful-job median is 24m30.5s. The first improvement will reuse the external Application Client fixture's build output between its public-consumer tests and InstalledCurrent proofs, without removing tests or feature profiles.
Baseline: 2026-10-23
Snapshot: main 2a7771696423428590236d4396d538324b54ce2a. Sample: the latest 20 CI push runs on main, created from 07:37:33Z through 19:41:14Z. Twelve succeeded and eight failed. All failed jobs were on Windows. Nearby runs often exercise the same product code, so these observations are not independent flake probabilities.
Source: latest complete run and its preceding 19 runs. Durations are job start-to-completion, excluding queue time; medians use successful jobs only. Raw run/job/step JSON and logs are retained in sentinel session files.
| CI job | Successful observations | Median | Successful range |
|---|---|---|---|
| Build and test (Windows) | 16 | 1470.5s | 1154-1714s |
| Build and test (Ubuntu) | 20 | 695s | 583-755s |
| Live Postgres parity | 20 | 665s | 534-702s |
| InstalledCurrent Postgres (Windows) | 19 | 574s | 426-728s |
| Feature combinations build | 20 | 357.5s | 250-433s |
| Windows heap diagnostics | 18 | 326s | 261-402s |
| Windows token-user: all | 20 | 195.5s | 139-267s |
| Windows token-user: sqlite-entra | 20 | 192s | 130-290s |
| Windows token-user: entra | 20 | 179s | 140-202s |
| Released v0.2.1 upgrade (Windows) | 20 | 139s | 106-257s |
| Copilot fallback (Windows) | 19 | 135s | 103-202s |
| Windows token-user: default | 20 | 123s | 99-256s |
| Windows token-user: sqlite-self-update | 20 | 117s | 79-193s |
| Copilot fallback (macOS) | 20 | 108s | 80-133s |
| Windows token-user: sqlite-postgres | 20 | 101s | 80-165s |
| Windows token-user: sqlite | 20 | 97s | 85-165s |
| Windows token-user: postgres | 20 | 89s | 68-196s |
| Credential lifecycle (macOS) | 20 | 87s | 51-115s |
In the latest successful run, workspace Test cost 697s on Windows and 380s on Ubuntu. The additional SQLite Application Client step cost 356s and 217s. Windows workspace symbol prebuilding cost 121s. The feature job spent 228s checking external consumer profiles in a separate fixture workspace.
Slow tests and fixtures
Stable Rust's current output reports test-binary elapsed time, not per-test timing.
| Suite / invocation | Windows | Ubuntu |
|---|---|---|
| InstalledCurrent, workspace defaults | 299.67s | 186.95s |
| InstalledCurrent, SQLite-only serialized leg | 266.42s | 169.55s |
| External consumer fixture, workspace defaults | 63.02s | 37.11s |
| External consumer fixture, SQLite-only serialized leg | 33.01s | 19.03s |
The live-Postgres InstalledCurrent suite took 266.30s on Linux and 249.98s on Windows. Its consumer fixture took 142.73s and 90.56s.
installed_current_killed_selector_client_releases_shared_admission and the SQLite consumer-fixture test emitted over-60-second notices. Several other InstalledCurrent tests did too, but they contend on a process-wide environment lock; those notices do not establish individual active execution time.
Two harnesses compile the same consumer fixture into different targets: tests/application_client_fixture.rs uses target-sqlite / target-postgres; tests/installed_current_process.rs uses target and invokes Cargo repeatedly. Reuse must still let Cargo validate the current source and feature set, not trust a pre-existing executable.
Test layout and seams
| Surface | Existing coverage | Assessment |
|---|---|---|
Core telex |
In-source unit tests; backend conformance; daemon core/process; credentials; station intent; Application Client core/binding/bundle/conformance; external consumer; InstalledCurrent; release contracts/upgrades; wait recovery | Broad layered coverage. Expensive process/install fixtures repeat under defaults, SQLite-only and live Postgres; preserve the distinct feature/backend obligations. |
telex-watcher |
Registry/runtime/management unit tests; protocol corpus; process containment; detector examples; Python schema/provider fixtures | Runtime matrix crosses the public Application Client into installed SQLite/Postgres daemons. Protocol/schema cases are already shared; do not duplicate that battery. |
telex-console |
App/data/filter/UI unit tests in workspace CI | Protects read-only core backend/model consumption; not the separate Operator Station GUI. |
| Copilot bridge | Node busy/idle and probe protocol; Rust plugin/intent/process contracts | Rust/JS probe, deferred delivery and fallback seams; targeted macOS/Windows fallback jobs. |
| Operator Station | Bidirectional public-consumer probe; durable ingest/recovery and disposition; Application Client semantic battery | Protects the public client seam, not native GUI E2E. The spike/ application is excluded from the root workspace. |
| Install/release/upgrade | Target/version-axis contracts; archive/checksum checks; isolated InstalledCurrent; official v0.2.1 writer-to-exact-candidate Windows proof | Strong exact-candidate/fail-closed coverage. Full native process coverage is Windows/Linux; macOS has targeted fallback/credential jobs. |
Ranked candidates
- Reuse external consumer fixture builds. Share one defaults-disabled, per-feature target layout between the two harnesses; avoid redundant in-process Cargo invocations where safe. Keep all executable probes, profiles, assertions and isolation. Added test time: zero expected; savings must be measured. First candidate because it removes setup duplication from an expensive seam without changing product behavior.
- Measure InstalledCurrent lock/handshake time separately from compilation. Identify deliberate delayed-Hello waits within the 170-300s suite. Replace a fixed wait only with an equally strong synchronization proof, coordinated with its owner. Small measurement cost; savings unquantified.
- Reuse compilation across the Windows token-user feature matrix. Eight job medians total about 18 minutes of runner time per run. Investigate build/cache boundaries while retaining every profile and identity assertion. Cache improvements need cross-head evidence.
- Review the Watcher management CLI seam after merge. PR [Watcher CLI] Add complete same-user management and diagnostics #180 currently touches runtime/protocol/registry and introduces CLI/transport tests. Compare those tests with existing public-client/schema contracts after merge; add only a missing invariant. Do not edit those in-flight files.
Shared-CI patterns for Local Daemon
Local Daemon retains root-cause ownership.
| Pattern | Main runs in sample | Evidence |
|---|---|---|
| Concurrent-first-use child access violation under PageHeap | 2/20 | 39919012467, 39981184421: real_process_01_concurrent_first_use, 0xc0000005 |
| Delayed inherited credential writer timeout | 2/20 | 39920031861, 39920756808: credential_command_shell_exit_preserves_delayed_inherited_writer, Elapsed(()) |
| Partial multi-store drain child heap failure | 1/20 | 39920756808: station_intent_a_partial_multi_store_drain_reports_each_store_and_restores_only_the_provable_one, 0xc0000374 |
| Cooperative truncated page abandoned | 1/20 | 39921631776: station_intent::tests::a_cooperatively_truncated_phase_still_returns_its_partial_page |
| Deferred turn-stop drain assertion | 1/20 | 39979500393: daemon::p3_tests::drain_deferred_repushes_unacked_after_turn_stop |
| Postgres crash/restart reattach refused as foreign | 1/20 | 40006452518: installed_current_crash_restart_and_reattach_recovers_membership, ForeignDaemon |
Failures occurred on planning/documentation/main-state updates as well as product-adjacent changes. The sample does not identify a causal change class.
Operating record
Review merges after they land and inspect recent main CI about every 30 minutes. Keep last-reviewed SHA, timing baseline and working findings in session files. Update this issue only for material changes. File each improvement as its own issue and small PR; Campaign gates and merges sentinel PRs.
Activity
lossyrob commentedon Oct 23, 2026
First candidate is implemented in PR #184, closing #183: the two public-consumer/process harnesses now share the existing defaults-disabled, per-backend fixture build. Zero tests added or removed; all executable probes and freshness checks remain.
One local Windows cold-duplicate versus shared-target comparison reduced the exact existing-peer handshake test from 83.56s to 41.95s test elapsed time. This is evidence of removed setup duplication, not a full-job CI prediction. Hosted CI and the independent review at
6547ad2594baa4bb5d1794b99f04874c25f43ff0are pending; the PR is not yet ready for Campaign.lossyrob commentedon Oct 23, 2026
Local Daemon routed a lower-priority test-hygiene candidate from PR #185 at
57fdc9752d3217f9ab9f2038bf90345ec48bae96: the owner reports Clippy 1.98warnings from the optional broader command
cargo clippy --no-default-features --features sqlite --lib --tests -- -D warnings.Reported locations are
tests/copilot_plugin.rs:304-307(
double_ended_iterator_last),tests/conformance.rs:322(useless_vec),and
src/daemon_bootstrap.rs:3480-3481(zombie_processes,print_literal).Disposition: deferred behind measured test-performance work. The owner's
published evidence summary
records the failing broader command and passing production-only
--libcommand with rustc 1.98.0 / Clippy 0.1.98. It is an attributed summary of
the original output, not a raw archive or a rerun.
This is not an existing required CI invocation or an issue #181 repair
blocker; the lint name does not establish a process leak. No intake rerun
or repair expansion is needed. Reconsider after the performance PR and
active repair land, when test-only cleanup could enable a useful local
lint check without adding a CI job or test battery. File a separate
improvement issue before implementation.
lossyrob commentedon Oct 23, 2026
Fixture reuse landed: first main measurement
PR #184 merged at
9c2a306b8a70e1585e8143d9608062c35ea490c6.Main run 40030182368
passed all 18 unchanged jobs. No tests or backend/OS legs were added,
removed or disabled.
The table uses job start-to-complete seconds, excluding queue time.
Baseline medians are successful-job medians from the original 20-run
sample. The before column is a prior successful, unchanged-runtime
main run 40027947004
at
67f84571; the after column is the merge run at9c2a306b.The affected Windows harness summaries give a narrower comparison:
Interpretation: this main sample is faster, but sustained hosted savings
are not established. Windows fell from 24m36s to 19m31s in this pair; the
after value is still inside the original 19m14s-28m34s range. The Campaign
gate had correctly observed that the PR job itself was slower than #178
(23m32s versus 22m13s). Unaffected work also varied: Windows workspace-symbol
prebuild fell from 115s to 72s, while the default token-user job rose from
110s to 181s. Different runners/cache states prevent attributing the whole
job delta to fixture reuse. Keep measuring subsequent main runs; the local
41.61s result remains a cold-duplicate versus shared-target comparison.
The pre-existing local
build.rsmissing-ref warm-rebuild observation isnow separately dispositioned in #186. It requires measurement and preserving
Git build-ID freshness; it is not a demonstrated hosted bottleneck.
The bounded reviewed-main sample is now 25 runs: 16 successful, 9 failed,
with all observed failed jobs on Windows. The latest pre-merge failure was
the existing delayed-writer timeout in
run 40028895852 / job 120671911324:
credential_command_shell_exit_preserves_delayed_inherited_writer,credential_command.rs:516,Elapsed(()). This is the third observationof that signature in this sample. Local received retained evidence and owns
triage; the planning-only head predates both #184 and #185. These are bounded
observations, not a population failure-rate estimate or common-cause claim.
lossyrob commentedon Oct 24, 2026
Main CI observation: run 40134893983
at
b0510ba2d275b431319392d17057bb014b414b37failed inBuild and test (windows-latest), job 120994212559.
The workspace test step reports
real_process_copilot_fallback_cross_platformfailing attests/daemon_process_sqlite.rs:24, called from line 3026:send to fallback waiter failed: code=Some(1) timed_out=true stdout= stderr=.That suite finished with 48 passed / 1 failed. This is a runtime observation for
Local Daemon's existing flake intake, not a root-cause attribution to a PR or an
inference of a process leak.
The reviewed main cohort is now 27 distinct runs: 17 successful / 10 failed.
The delayed inherited-writer timeout count remains 3; this fallback-send
observation is not added to that signature. This is a bounded reviewed sample,
not an independent population failure-rate estimate. The two new main commits
only add Local discovery records; the non-planning source tree is unchanged.
Retained failed-job capture:
main-40134893983-job-120994212559.log, 436053 bytes,SHA256
c97ef67787434f714f69fecee18ba1518bf184a076e1162eb2c44d8f7486b574.No local rerun was performed.
lossyrob commentedon Oct 24, 2026
Git metadata watch correction: first hosted sample
PR #188 has an independent
exact-head PASS receipt
and all 18 CI jobs passed at
f3f2c437d30ee834e0cae9095bdb6e873e255701in run 40135612295.
There are no new jobs, dependencies, test binaries, or removed executions.
Matching Windows InstalledCurrent suite times were 230.89 -> 133.72s
(defaults) and 227.14 -> 131.77s (SQLite-only), retaining 30 passed / 2 existing
ignored cases in both configurations. The public consumer fixture was mixed:
42.25 -> 62.90s for defaults (6/1), and 23.73 -> 6.09s for SQLite-only (5/1).
The expanded, existing release contract suite retained 20 passing cases under
the hosted default profile: 1.87 -> 8.21s on Windows (+6.34s). It took 1.08s on
Ubuntu in this run; no Ubuntu baseline delta is claimed here. The author's
separate local SQLite three-contract sample added 12.01s (5.29 -> 17.30s).
Windows baseline and candidate logs both report rustc/Cargo 1.99.1.
This first hosted result is lower for the four job totals, but it is not a
sustained or isolated causal estimate: runner/cache conditions were not
controlled, and the first default public-fixture suite became slower. The local
real-consumer existing-metadata control was already fully fresh at about 0.47s
before and after. Tiny-fixture savings are not substituted for whole-project
measurements. Post-merge actual-main CI and the #186 field report remain due.
lossyrob commentedon Oct 24, 2026
Git metadata freshness: merged-main result and cost disposition
PR #188 is validated at actual
merge
ef975014af7b9a34d3ca0bc9223be1a35d369868.Main run 40137407119
passed all 18 unchanged CI jobs. This is separate evidence from the PR-head
run. The intervening commits changed planning files only; the merged
non-planning tree equals the independently reviewed PR head.
The lower hosted totals repeated after merge: Windows was 3m32s below the
recorded baseline. Two additional pre-fix, planning-only main runs reviewed
in the normal watch also passed all 18 jobs: Windows took 1343s at d0da34d
(40135250735) and 1423s at 970be04 (40136174953). These samples support the
direction of the improvement, but do not isolate its cause or establish a
sustained saving. Runner/cache conditions were uncontrolled. No retry was
needed for the merged-main result.
Matching Windows InstalledCurrent suites retained 30 passed / 2 existing
ignored cases: defaults took 134.78s and SQLite-only 130.86s, versus
230.89s / 227.14s in the baseline. Public consumer fixtures remained mixed:
defaults took 64.99s versus 42.25s before, while SQLite-only took 5.91s
versus 23.73s. A lower job total does not erase the slower cold default
fixture.
Test budget: the Windows release-contract suite still has 20 passing
cases. Its baseline / PR / merged-main times were 1.87s / 8.21s / 6.92s.
The new main sample therefore adds 5.05s; the earlier +6.34s PR sample
remains valid. Ubuntu's merged-main suite passed the same 20 cases in 1.20s
(PR 1.08s; no Ubuntu baseline delta claimed).
Disposition: retain the fixture now. The conditional trigger to trim it
if main shows no hosted saving did not occur in this sample: all four
reported job totals remained lower. The added test cost is still not a
proven causal offset, so the budget qualification stays in this health log.
If later main samples lose the improvement, first remove redundant
environment-precedence rebuilds and paired unchanged builds while retaining
the absent-metadata and Git-only freshness assertions. No new test or CI
job is proposed.
Local validation at the merge passed all three existing SQLite-configured
build-ID contracts in 24.63s after a 23.15s build. The exact workspace
executable reports build
ef975014af7b9a34d3ca0bc9223be1a35d369868.These are another local sample, not a new before/after estimate; the earlier
5.29s -> 17.30s (+12.01s) local cost remains disclosed.
The whole-common-directory watch when packed-refs is absent remains
conservative. No practical problem from that breadth was established by
this validation, so it is unchanged. The previously captured real
packed-refs mtime invalidation was an existing-file change, not evidence of
this directory-watch concern.
Session identity: the gate's App ID
1f22268eresolves to runtime sessionfc772c8fand this same sentinel worktree. It is not a helper or replacement.Retained evidence is under sentinel session
fc772c8ffiles:postmerge-main-ef975014-ci.json,postmerge-ef975014-hosted-timing-comparison.json,the Windows/Ubuntu logs and suite summaries, and
postmerge-ef975014-build-id-contracts.log.Windows log SHA256:
05aa4d29338cb718cbe701e8d98b36c3a50f5aa939b5e4094620de66678b5426.Ubuntu log SHA256:
7cfee58019bcdcb1082837ec551a7ed2288bba564d7d91e132483be2a590977b.lossyrob commentedon Oct 25, 2026
CI-cost intake after the station-intent lock repair
The merged PR #189 adds a
three-OS station-intent lock job. Consolidation assessment is queued until
PR #185 lands; it is still open
at
7b11b926at intake. No sentinel workflow change is in progress.The lock job's successful exact-head PR sample
40138318787
took 42s on Ubuntu, 59s on macOS and 110s on Windows: 211 aggregate
job-seconds including setup. These are candidate-cost measurements, not
predicted savings. Actual-main run
40214028177
at merge
d03492eais still in progress with 21 jobs.The repeated station-intent module is a consolidation opportunity, but
Linux/Windows workspace runs use default features while these dedicated
jobs use defaults-disabled SQLite. Matching test names alone do not prove
profile equivalence. The macOS unit/filesystem boundary must remain.
The #181 owner and sentinel have coordinated the after-landing assessment.
A candidate is to retain the canonical three-OS station-intent-lock job,
move all ten exact partial-page repetitions into it, and retire the
duplicate cooperative matrix. This is not a final proposal against an
unmerged workflow.
The later comparison must preserve full station-intent and platform-fs
modules on all three OSes, ordinary test profile/default parallel
scheduling, all ten
--exactrepetitions with hard failure propagation,direct checkout verification separately from provider SHA, and isolated
absolute runtime paths. It must publish fresh run evidence and explicit
check-name mapping; retired-job results cannot stand in for new combined
results. The existing full suites and partial-page semantics stay intact.
No new runtime repair or extra observer is needed. The serial main watch
will supply the landing event for the assessment.
lossyrob commentedon Oct 25, 2026
Main CI: lock repair passes; later Windows fallback attach times out
The actual PR #189 merge
d03492eapassed all 21 jobs inrun 40214028177.
The three lock-boundary jobs took 68s Ubuntu, 94s macOS and 169s Windows:
331 aggregate job-seconds including setup. This is the first merged-main
cost sample, distinct from the earlier 211s PR sample. No saving is inferred
from either. Consolidation remains deferred until PR185 lands; it was still
open at
130be3b2during this observation.The next planning-only main
d6953111also passed all 21 jobs in run40214759208. A later planning-only main
a94e3f49passed 20/21 jobs inrun 40215161046.
Its Windows build/test job
failed
real_process_copilot_fallback_cross_platform:The immutable source and backtrace place this at
tests/daemon_process_sqlite.rs:3141, through the assertion helper at line24:an 8-second captured
copilot attach --copilot-bridgeafter the explicitfallback-stop proof, before the following direct-wait assertions. This is a
different observed boundary from the previously routed 5-second sender
timeout at line3026. Local Daemon owns classification; no common cause,
new candidate, process leak, or attribution to PR189/185 is inferred.
Raw log retained in sentinel runtime session
fc772c8ffiles:main-40215161046-job-121234606217.log, 332080 bytes, SHA2562ef9954af9a479f78f1a30fad6a521a75eb85d46583e228ce65950913e9d6328.The exact CI snapshot and source inspection are retained alongside it.
No local rerun or additional observer was started for this intake.
The reviewed-main bounded cohort is now 39 runs: 28 success / 11 failure.
The previously attributed credential-writer timeout count remains3; this
different failure is not added to it. This is a reviewed sample, not an
independent project failure-rate estimate.
lossyrob commentedon Oct 25, 2026
The consolidation is proposed in PR #192, closing #191, at head
165549741bea6f5d41a16b40c66b9d9ed51f9af8. Removing the duplicate cooperative matrix reduces CI from 24 to 21 jobs. The canonical three-OS boundary job and every other job remain unchanged. Broad two-OS workspace jobs retain their distinct default-feature/provider-checkout coverage.This supersedes the earlier intake's proposal to preserve ten extra runs. After Campaign invited trimming and the #181 owner identified no concrete invariant loss, I chose one full-module execution per OS instead. This intentionally reduces repetition exposure; it is not ten-run-equivalent evidence or real-200ms read-grace coverage. The positive case, all neighboring semantic checks, platform-filesystem tests, assertions and failure behavior remain unchanged.
In the all-green PR185 baseline run at
130be3b2, the retired jobs cost 238 aggregate seconds (Windows 112, macOS 82, Ubuntu 44); the retained lock jobs cost 195. The extra loops account for only 13 seconds, so duplicate fresh builds are the larger opportunity. These uncontrolled old-head costs are not promised savings.Strict YAML/text comparison proves all ten retained job definitions unchanged. Exact-head Windows compile/list checks retain 62 station-intent and 14 platform tests, not runtime evidence. Fresh candidate run 40218198563 and the independent public review are pending. Retired checks map by OS to the canonical full module using new run/job/checkout evidence only; archived #185 results are not relabeled. Candidate and actual-main timings will be reported separately. Hosted saving remains unproven.
lossyrob commentedon Oct 25, 2026
Main watch update: the actual #185 merge run at
65f5846dpassed all 24 jobs. That is separate from PR192's consolidated candidate run. The following planning-only main run 40217995470 atf603875bddd19a6f76f08a408ba3fbd98d3073bbpassed 23 of 24 jobs; Windows heap diagnostics failed.In the PageHeap process-suite stress step, the first full iteration passed 49 tests. The second finished 48 passed/1 failed:
ordinary_send_happy_path_is_identical_with_and_without_mismatched_watcher_token. The captured command exited with abnormal Windows status0xc0000005after printing JSON with"draining": true. The stack isrun_command_with_captureat line 518,ProcessEnv::stop_daemon_best_effortat 201, andDropat 265, ending at the test's closing scope at 1410. The reported failure surfaces at teardown, not at a parity assertion; it does not establish the crash cause.Retained evidence:
main-40217995470-job-121243212833.log, 107711 bytes, SHA256f182848a1ca3c3bb197717f8d19e1cb145bf9f90dbc92992ff5a15599d5a632b; immutable test source blobb1f708c5e8948884c49207fe4aba4e051dc7f23a. The provider job is 121243212833. Routed as evidence to Local Daemon for classification and diagnosis; no common cause or PR attribution is inferred.The bounded reviewed-main cohort is now 42 runs: 30 success, 12 failure. The inherited credential-writer signature remains three observations. This is an attributed observation cohort, not an independent project failure-rate estimate.
lossyrob commentedon Oct 25, 2026
Fresh consolidated-candidate run 40218198563, attempt 1, passed all 21 jobs at
165549741bea6f5d41a16b40c66b9d9ed51f9af8. Each three-OS boundary log verifies that direct checkout separately from event SHA6ab639ed31ca745ffbe88711bcbe0a538d8288b7, with the positive partial-page case passing inside the 62-test module and zero ignored tests. Filesystem modules passed 14 tests on Windows, 13 on macOS and 12 on Ubuntu.The remaining boundary jobs took 116s Windows, 47s macOS and 36s Ubuntu: 199 aggregate job-seconds, versus 433 across the old six jobs on PR185 and 563 on its actual merge run. This is observed lower aggregate boundary cost, not a causal/sustained-saving or whole-CI critical-path claim. Retiring ten extra invocations deliberately reduces repetition; the controlled positive case still does not establish real-200ms first-read grace coverage.
Exact-head readiness and custody include the independent reviewer's own public receipt, immutable check-name/job mapping, per-log hashes and separated historical/new-run evidence. Actual consolidated-main CI, timing and the #191 field report remain due after Campaign's merge gate.
lossyrob commentedon Oct 25, 2026
Consolidated-main timing and a separate Windows seed failure
Main run 40220058150, attempt 1 completed at merge
34cfdee7fb1f31eef6f0025f7a7b5b1af8bd2a11with 20/21 jobs successful. All three retained station-intent boundary jobs passed. Main is not green.Each log independently prints actual source checkout and provider event SHA; both are
34cfdee7for this push. Each module completed with zero failed or ignored tests, including the mandatory partial-page positive case once. These are fresh check/job identities, not results from the retired cooperative matrix or the cleared PR.The consolidated boundary total is 232 aggregate job-seconds in this one main sample. Historical six-job samples were 433 seconds on PR #185 and 563 seconds on its main merge
65f5846d; the cleared three-job PR #192 sample was 199 seconds. This main observation is 331 seconds below the earlier main sample and 33 seconds above the PR #192 sample. Different heads, runners and cache conditions prevent a causal or sustained savings claim. This metric sums job start-to-completion durations; it is not billing-rounded cost, queue time or whole-CI critical-path improvement.The failed default Windows workspace job reported 571 passed, 1 failed and 4 ignored.
daemon::p3_tests::a_register_whose_proof_cannot_be_written_is_refused_and_leaves_no_memberfailed while seeding its pending intent:seed: Io("intent 20925d3b460e3bef3316a173b96ff3b8 is locked by another writer or its acquisition budget expired"). The immutable source and stack put the failure at helpersrc/daemon.rs:8511, called at:8692, before registration, intentional manifest corruption and refusal/no-member assertions. The job then skipped its later process-test and lint steps; those steps have no passing evidence from this run.Local Daemon classified this as a new candidate in the station-intent lock-admission family:
local-daemon-windows-intent-seed-lock-admission-refusal. The combined error cannot distinguish elapsed-before-attempt, a late successful guard, WouldBlock retries or attempt exhaustion. It is neither a partial-pageOverranobservation nor a recurrence of #187's old 1.54-second assertion; no common cause or attribution to PR #192 is inferred.Campaign accepted #191's consolidation delivery for its scope after reading the field report, explicitly kept main red at 20/21, and routed the new critical-path work to Local. Local is investigating fixture identity and acquisition policy before deciding whether a budget/contention defect is supported; no retry or flake exception applies.
Retained evidence:
postmerge-34cfdee7-boundary-121249478932-windows.log56e6e0aaa7a19b656e37582304034df3db2b11023bfbec2169c45c992ac51735postmerge-34cfdee7-boundary-121249478934-macos.log1b72674b7255e451bb941a62148ea6d221fed47366318fcb3e782b51a65f328fpostmerge-34cfdee7-boundary-121249479015-ubuntu.log788c4766454143f16cbcc976526acee8046b402fdb7578d8ca10065080986cdbmain-40220058150-job-121249478906.log403046b0afffec29ff94c48b2ea16cd50ed86b86c08d8efc0e6ae685df7179bdmain-40220058150-evidence-custody.json044d4a0a1254248dd9cf51996027c36a706780765e9e78d955a27beb121e504cThe custody artifact includes correctly numbered immutable checkout/failure excerpts and source excerpts from merge
34cfdee7. Reduced repetition remains intentional; the controlled positive case does not prove real 200ms first-read grace, and the historical WindowsOverrancause remains unknown. A futureOverranrecurrence on main gets a new item rather than a retry.lossyrob commentedon Oct 25, 2026
Two pre-Watcher failure boundaries; consolidation timing follow-up
Three newly reviewed completed main runs used the unchanged 21-job configuration:
a47378a04deaf30cf4644c02All three sources change ledger metadata only. Both failed runs predate Watcher PR #180's merge; no attribution to #180, #192 or a shared root cause is established. The later green
a47378a0run does not change the original34cfdee7run's 20/21 failure or waive the Local-owned seed-lock repair.InstalledCurrent Postgres. Job 121258004551 passed the public fixture suite 6/0/1 ignored, then failed the InstalledCurrent suite 29/1/2 ignored.
installed_current_crash_restart_and_reattach_recovers_membershipfailed attests/installed_current_process.rs:508: the reattach outcome wasDaemonBootstrap(ForeignDaemon). This occurs after the bounded repair call and before the later publication-marker, successor-PID and spawn-count assertions. The source configures a 1500ms publication delay and four repair retries; those intended assertions are not successful evidence from this run.PageHeap. Job 121259578492 passed 49 process tests in iteration 1, then failed 48/1 in iteration 2.
station_intent_a_partial_multi_store_drain_reports_each_store_and_restores_only_the_provable_onereached the post-drain successor side-station attach attests/daemon_process_sqlite.rs:5838, through helper:4542to assertion:24. The command returned ordinary exit 1 withtimed_out=falseand SQLiteLockFileExerror 33. It failed before push-restoration assertions. The precedingwait_until_not_runningassertion does not independently certify kernel-process or store-lock release; the lock holder and cause are unknown. This observation is not a native-exception result or a partial-pageOverran.Local disposition. Local independently verified both raw/custody hashes and all 61/105 immutable source lines. The InstalledCurrent result is a new Windows crash-reattach
ForeignDaemoncandidate: the older Ubuntu recovery timeout has a different platform/result, and this result is not proof of a foreign process. The PageHeap result is a separately labeled multi-store occurrence under existinglocal-daemon-windows-station-restart-lockfileex-33, not an AV or #193's intent-lock admission failure. No retry, new issue/owner, shared-cause or #180/#192 attribution follows; evidence-only intake stays separate from the priority #166 AV work.main-40222860912-job-121258004551.log3d7c3b61371a4d48649854f74993504dae26ee6e54e9d55f35c2050cee147c34main-40222860912-evidence-custody.json744aafc70d90b065b2fe8c469b29d55a4b0092aba2aad35fa15ab88df33a2eebmain-40223385797-job-121259578492.log1cf3f57f3100c6057b1d178a483259b43ef91cb435b114af4b2f64fa43e6816amain-40223385797-evidence-custody.json641d203f447191de4cbfffc5151d8afec3bc1fc3ee84fd716162647f1de8ab23The custody files preserve direct checkout evidence and 61/105 correctly numbered immutable source lines, respectively.
Boundary cost. All nine canonical station-intent jobs in these three new runs passed. Four consolidated-main samples now total 232, 237, 219 and 182 aggregate boundary job-seconds, compared with the earlier six-job main sample of 563 seconds. These remain uncontrolled different-head runner/cache observations, not causal or sustained savings, billing cost or whole-CI critical-path proof. The additional samples use provider job timestamps; they are not substituted for the separately retained raw source/count proof at
34cfdee7. Reduced repetition and the real-200ms grace/unknown historical WindowsOverrandebts remain.Landed Watcher seam. Source review of PR #180 followed CLI parsing through native same-user transport to the existing owner, safe projection before framing, durable revision/hash audit, and receipt/tombstone recovery. Existing focused native, CLI, registry and backend-recovery cases cover these Linux/Windows seams; another broad battery is not justified. The design steward confirmed that Unix management is currently claimed on macOS, separately from unclaimed detector containment. #195 queues measured existing management-only native/CLI coverage after #194's active workflow edit, without a broad matrix or detector workload. No native macOS validation or containment clearance is claimed. This entry's completed runs predate #180 and do not establish its post-merge main validation.
lossyrob commentedon Oct 25, 2026
Landed Watcher management: first actual-main execution and test cost
Watcher PR #180's exact merge
0d5347a6e1b8a1716b524c537c4886d3f4d49b65passed 21/21 jobs in fresh push run 40223537702, attempt 1. Both retained workspace logs independently show that checkout SHA. This is actual-main evidence, separate from the PR's historical CI and from the earlier failures.These are observed test-binary elapsed times, not measured incremental PR cost. The Windows library duration and roughly 23-second CLI suite are useful baselines for choosing a focused macOS subset, not grounds to trim deliberate refusal/timeout or large/deep projection checks. No additional Linux/Windows battery is proposed.
The three canonical station-boundary jobs also passed: 47s Ubuntu, 149s Windows and 53s macOS, totaling 249 aggregate job-seconds. That is a fifth uncontrolled consolidated-main timing sample; it does not prove causal savings or clear the real-200ms grace/unknown historical
Overrandebt.The design steward's public clarification separates currently claimed Unix management from unclaimed macOS detector containment. Focused existing macOS management/native-CLI coverage is queued behind the active #194 workflow edit, with measured marginal cost and no detector workload or broad matrix. Current results do not establish native macOS management validation.
Retained raw workspace logs: Windows job
121260033811, 399391 bytes, SHA256a7c9fc076f3b65acee68bee0c8a4e0cc6337986270d42a8880407201fe6179db; Ubuntu job121260033911, 367825 bytes, SHA256e44cadef928f1308b52bb0704369dccead01e69737e12a1aeb0e170ae40e4662. Numbered executable/result excerpts and the exact source/run/job mapping are retained inmain-watch17-proof.json.lossyrob commentedon Oct 25, 2026
Second Windows crash-reattach refusal at the same public boundary
Main
5da770cf, run 40225451733, attempt 1, finished 20/21. The Windows InstalledCurrent Postgres job failedinstalled_current_crash_restart_and_reattach_recovers_membershipattests/installed_current_process.rs:508: reattach returnedDaemonBootstrap(ForeignDaemon)before the successor-PID, publication-marker and spawn-count assertions. Public fixtures passed 6/0/1 ignored; InstalledCurrent finished 29/1/2 ignored.Local independently matched both hashes, all supplied log excerpts and 61 immutable source lines, and classified this as the second retained occurrence of
local-daemon-windows-installed-current-crash-reattach-foreign-daemon, after4deaf30c/ run40222860912. It attaches to the one existing candidate, not a new record or the older Ubuntu timeout. Root test blob1f66b6784e33a2836fda6329e572d79c19ac62d6matches exactly; the earlier source predates Watcher #180 and the newer one includes it. Neither the identical test blob nor5da's own metadata-only delta is a controlled before/after comparison or establishes a shared cause, an actual foreign process or #180/#192/#194 attribution. No retry, probe, new issue/owner or #166/#193 scope expansion follows.Retained failed job 121265733770: raw log 50306 bytes, SHA256
a04f54f705aa72676d583174df364791424a17dfc0e4ce2958387de611645dbf; custody 18930 bytes, SHA256fec7ad9cba12fbed4547d09d69b00521e2c61c980dfa5b9daaaf8342d9a55b61. Direct checkout, result excerpts and numbered source are retained inmain-40225451733-evidence-custody.json.Later metadata-only heads passed independently:
f0053e3b/ 40226604399 anddf37d568/ 40227092687, both 21/21. Neither replaces the failure or demonstrates a repair. The four newly reviewed commits change only Local/Watcher ledgers, recording separate #166 owner promotion and accepted Watcher delivery; no product/test/workflow change or additional Sentinel test is justified by those records.All nine canonical boundary jobs in these three runs passed, totaling 184, 245 and 228 aggregate job-seconds in chronological order. Nine consolidated-main timing samples now span 182-249 seconds; these remain uncontrolled runner/cache/head observations, not causal savings or debt clearance. The selected reviewed-main cohort is 53 runs, 37 successful / 16 failed, not an independent failure-rate estimate. Focused macOS management coverage remains queued behind active workflow ownership; native AV and intent-seed repairs stay separate.
46 remaining items
lossyrob commentedon Oct 27, 2026
Admission-to-recovery seam coverage on main
PR #241, merged as
ca6ed12c309deedae8f5092a620fe51a4840f703, moves Application Client connection/bootstrap/capability admission before reserving a new send or reply operation, then uses that same connection. This protects receipt-gated consumers from a pre-write outage leaving a new operation Pending without retry-authorizing evidence.Two added scenarios bring the existing shared SQLite/PostgreSQL conformance inventory to 30. They cover typed admission failure, exact-handle absence with unchanged retention generation before and after a fresh runtime, exact retry with one message, offline replay/mismatch, preservation of existing outcomes, and concurrent identical attempts. They reuse the existing backend batteries and isolation/build structure without a new job or matrix leg. Incremental scenario runtime was not isolated here; no saving is claimed. No additional Sentinel test is justified for this seam.
Main CI 40445276093, attempt 1 succeeded with all 22 jobs successful; the separate Docs run 40445276122 also succeeded at that exact merge. This is producer coverage, not #234's still-separate consumer integration proof. No new failure occurrence is recorded.
lossyrob commentedon Oct 27, 2026
Post-merge coverage and cost: PR #239 at aae96d3.
Original main push CI https://github.com/lossyrob/telex/actions/runs/40449436237, attempt 1, passed all 22 jobs. The preceding e32051b push, run 40447696369 attempt 1, also passed all 22. No new failure or shared-flake occurrence is recorded here.
The landed Watcher recovery proofs exercise the Application Client admission/recovery seam added by #241. They protect sender non-readiness, paused attempts and send budget, unchanged unaccepted state, stable runtime identity/heartbeat, automatic recovery and exact durable deliveries. The interrupted-daemon case forces a real send past readiness/budget admission before the fault and requires the original recovery identity to be accepted exactly once. That barrier is not a claim that every PostgreSQL connection-fault case forces an interrupted send.
Actual Linux and Windows main logs contain passing lines for both new SQLite cases and the existing SQLite matrix. Both live PostgreSQL jobs ran three passing cases: the existing matrix and two new cases, each of which exercises daemon replacement and fixture-private connection termination. Isolation, stop-before-schema-drop and all fault/order variants remain. The only landed CI edits expand those two PostgreSQL selectors; there is no new job, checkout-policy change or change to the native macOS management steps. I found no additional seam test justified by this merge.
The PostgreSQL expansion has material cost:
Baseline run: https://github.com/lossyrob/telex/actions/runs/40445276093 (attempt 1). The harness figures exclude compilation; step figures include it. These are uncontrolled observations at different heads, not isolated per-case cost, a promised saving, a critical-path attribution or a future timing allowance.
The next performance candidate is the resilience fixture's serialized setup/waits. Keep the existing checks while investigating: preserve both backends and native OSes, both fault types/orderings, the strict interrupted-send proof, the ten-second no-budget-burn observation, cleanup and evidence. No extra end-to-end test, timing workaround or product scheduling change is proposed by this entry.
Docs was not triggered: none of the seven landed paths matches docs.yml's root src/, docs/guide/, Cargo manifest/lock or docs.yml filters. This is not a Docs pass or skip. #195 remains completed and Campaign-accepted; these results grant no #167 repair or #182 failure-resolution credit.
lossyrob commentedon Oct 27, 2026
Post-merge diagnostic coverage: PR #240 landed as 8e09538, instrumentation only, Refs #167. It is not a repair or a new recurrence; #167 remains open.
The existing delayed-inherited-writer case now records a pre-write marker and native-owner drain progress. Timeout capture is drop(operation), non-reaping drain snapshot, the existing reaping obligation query, then marker I/O. Invocation ID and owner PID bind counts and monotonic timestamps; no command text or output bytes are retained. The existing 5s budget, 22-byte pretrim/18-byte trimmed fixture, result/password/empty-obligation assertions and single case per invocation remain. No new test, job, matrix, diagnostic repetition or timeout allowance was added.
These are post-cancel independent loads, not a coherent deadline-state snapshot. The observed outcome is sampled after the timeout future returns. Later polls can overwrite earlier observations; byte counts are not EOF or child-exit proof; shell exit is the owner's observation. Writer FILETIME is wall clock, marker I/O affects timing and can fail, and a missing marker means unpublished rather than unreached. These signals alone do not select child startup/pre-write/write latency versus stopped inherited-pipe drain.
Full landed source and existing owner-loop context were inspected. Native drain/cleanup behavior and test assertions are retained; the accessor does not join, reap, publish or release permits. There is real new per-poll timestamp/atomic work in PostgreSQL-enabled builds, including non-Windows, plus Windows fixture marker I/O. Its marginal cost is unmeasured, not zero. No additional Sentinel test or experiment is justified by this landing.
The Campaign gate explicitly retained the premerge execution gap after #239: the earlier green PR run did not execute that combined tree. Exact-merge main CI https://github.com/lossyrob/telex/actions/runs/40451635695 was still in progress at intake; no composed-tree pass is claimed here. Exact-merge Docs https://github.com/lossyrob/telex/actions/runs/40451635758 passed attempt 1, both jobs. Natural first-failure recording remains mandatory before any rerun; credential-target residuals route through Local/#167, and Watcher PostgreSQL-step failures go to Campaign for classification.
The other first-parent advance records Campaign-accepted #238 completion and release of its historical owner. All other App graph fields/nodes/edges and the prior contract summary are unchanged; that metadata supplies no new #234 or shared-failure credit.
Separate cost routing: Campaign placed #242 in the unassigned, non-blocking backlog. No implementation or owner launch now. Reopen only if a live-PG job becomes the CI critical path or nears its timeout, or repeated main runs show growth. Sentinel continues only its existing natural watch.
lossyrob commentedon Oct 27, 2026
Terminal qualifier for the PR #240 instrumentation landing: exact-merge main CI https://github.com/lossyrob/telex/actions/runs/40451635695 at 8e09538 passed attempt 1, all 22 jobs. This supplies the previously pending first execution of the tree composed with #239. Exact-merge Docs 40451635758 also passed attempt 1, both jobs.
The retained Windows job 122015471345 records event/provider/checkout = 8e09538. Both ordinary credential invocations contain the final delayed-writer test PASS and 11 passed / 0 failed / 2 ignored target summaries. Their successful phase lines are not failure discrimination, drain-at-deadline evidence, cause selection or repair proof. #167 remains open; the closed diagnostic sampling ledger is unchanged. Raw CLI stdout is preserved: 456324 bytes, 1637 NULs, SHA256 8e74d9db454c1dee234623da86dffd30db9650ec5fcaed53f42c414d8c4739a0.
The other frozen result, a0c21d6 CI 40451401010, also passed attempt 1, all 22 jobs. Source has advanced only through the 744bcaa Watcher #234 accepted-completion metadata: 365 lines read, zero product bytes, only status and a summary prefix change, with all other graph fields/edges and the prior contract preserved. This records neither owner release nor an external installation/upgrade result. No additional test is justified.
Natural CI cost observations now cover three consecutive main runs:
Linux whole-job and step observations rose across these runs; Windows did not show the same trend. The steps include compilation, and heads/runners vary: this is not isolated harness growth or a cause attribution. Neither PG job limited any of these runs. At 8e, Linux PG finished 474s and Windows PG 620s before the last Windows Build and test job; neither PG job hit a timeout.
This passive repeated-growth signal is being sent to Campaign under its recorded #242 routing condition. Sentinel has not changed that issue's backlog routing, launched an owner, added an observer or run a diagnostic experiment. All fault/order variants, the 10s no-budget-burn observation and strict interrupted-send proof remain unchanged.
Campaign qualification: the three-sample packet above is NOT a #242 trigger. Campaign judged Linux +17s step / +108s job, Windows flat-to-down and both PG jobs' 474/620s slack to be run-to-run noise-level rather than sustained growth; #242 remains backlog/unassigned.
The sharpened natural-watch triggers are: either PG job's slack to the critical path falls below 120s, or the same platform's Watcher step grows on 5 consecutive main runs or exceeds 400s. Observe only the existing natural cadence; no experiment or owner launch. Source: actual Campaign message toolu_01AVgg5S5qQWA1LA9b194VWx. This qualification supersedes the earlier broad repeated-growth interpretation, not the retained measurements or their uncontrolled limits.
lossyrob commentedon Oct 27, 2026
Main watch: new Windows cap-setup failure retained; failed-run PG slack needs qualification
Original main CI 40463549751 attempt 1 at 7ebac77 is terminal: 21 jobs succeeded, Windows Build and test job 122053497821 failed in step 20 (Test).
https://github.com/lossyrob/telex/actions/runs/40463549751/job/122053497821
The workspace library failed station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes at src/station_intent.rs:3788:41: "write under cap" / "late successful acquisition on attempt 1; guard released". Summary: 610 passed, 1 failed, 5 ignored, 57.58s. This is the initial under-cap setup, before held mutation and the overflow/guidance/count/rewrite assertions; loop iteration is unknown. It identifies the fail-closed admission branch, not latency or a holder/contention/scheduling/filesystem cause.
Event, provider and actual checkout all match 7ebac77. The station-intent blob bf909638 is exactly the same as the prior PR240 sample-1 site (6432295605), which Local associated with the residual cap/setup inventory under completed #193 debt 2 / #209 (historical #194/#209 co-track). This is ONE new natural main observation sent to Local for association, not the named #208 scoped-scan case, repaired #193 outcome, #167 discriminating failure or a new family/owner. Historical cohorts and six #193 debts are unchanged.
Credential step 29 was skipped. The retained capture has no #167 iteration/phase markers or credential integration-runner lines; this is not absence of work everywhere and does not reopen/reset/refund the closed diagnostic ledger. Unmodified gh job-log stdout is retained: 364269 bytes, 1637 NULs, SHA256 15c8fcaa51d7cf44725702c8b9b568596af1223fe13eb7a9606a1cfe0e6071b9, with six byte-offset/hash-verified NUL-free excerpts and four immutable source excerpts. Failure, panic and summary are lines 2227, 2235-2236 and 2260.
#242 natural timing flag: Linux PG 782s / Watcher step 276s, Windows PG 708s / step 327s; both jobs and steps succeeded. Linux PG was the last job at 07:04:07Z, yielding observed finish slack 0s / 74s, below the literal 120s condition. HOWEVER, Windows workspace had failed early and ended at 06:57:08Z; this is not an all-green critical-path or sustained-growth baseline. Neither platform has five consecutive growing steps, and neither exceeds 400s. Sent to Campaign for disposition; #242 is not self-rerouted and no experiment or owner is launched.
The new commit changes only Watcher metadata: accepted #216 completion and retirement of #211 debt 4, with Postgres optimizer-plan index-use debt retained. No product/test/CI bytes changed; no new test is justified by that bookkeeping. Docs is N/A under its unchanged path filter. No rerun, dispatch, timeout increase, waiver, repair or runtime action was performed; 480ms/25-attempt and other protected admission/page semantics remain unchanged.
Campaign disposition after this report: #242 is NOT triggered. Slack is evaluated only on all-green runs; this failed attempt's 0s/74s is not a valid slack sample because the Windows workspace failure truncated the critical path. No five-run growth or >400s step condition applies; #242 stays backlog/unassigned.
Campaign accepted the cap/setup failure as ONE new natural observation of the known #193 debt 2 / #209 residual family, with Local association, no new owner and no #167 credit. After the immutable first-failure capture above was retained, Campaign reran ONLY the failed job as attempt 2, to restore main's green signal, not as a waiver, timing change or repair. Sentinel will observe it through the sole ordinary cadence and report only if it fails. The original observation and source-time pending disposition above remain preserved.
Local independently verified the packet, raw capture, all six raw excerpts and four immutable source excerpts at 07:13:46.662Z, and accepted this ONE observation into the existing residual #193 debt 2 / #209 setup-site inventory. This is site/refusal-branch association, not common cause. The current failing write was refused before its held mutation; prior setup iterations are not claimed mutation-free. The original cohorts, six debts, 480ms/25-attempt policy and closed diagnostic ledger remain unchanged; no duplicate public entry or new action follows.
lossyrob commentedon Nov 11, 2026
One additional residual cap-fixture setup refusal; no rerun or repair performed.
PR #250, CI run 43086360336, attempt 1, Windows Station-intent boundary job 130751666200: step 5,
Exercise complete station-intent unit boundary, failed.d7bfd6e961cc80176d055c046c7af8500f4098b6; provider/generated merge:9584b240bb977f771b01562855874811b9e15f88(parents4b3ff8f508598a32cef542831967e2bee69c2f5aandd7bfd6e961cc80176d055c046c7af8500f4098b6). Both trees are17bb3786d2d26051f13bc869113db53fe369c64e; the job explicitly checked out the source head.cargo test --locked --no-default-features --features sqlite --lib station_intent::tests:: -- --nocapture.station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes,src\station_intent.rs:3788:41.Result: 65 passed, 1 failed, 0 ignored, 489 filtered, command exit 1. 36.80s is whole-suite duration, not acquisition latency. The failure occurred while populating Live records below the cap, before the cap-overflow/no-deletion/rewrite assertions; the failing loop iteration is unknown. The earlier injected worker panic was intentional and its test passed.
The explicit branch reports a successful first acquisition rejected after its 480ms budget, with the guard released. It does not establish a competing holder, native-call latency, scheduling/filesystem cause, a production policy defect or a common cause. The 480ms/25-attempt contract is unchanged. The source and workflow blobs match the PR base:
src/station_intent.rs=bf909638a1937d2706b565addc537b29442cdfe9;.github/workflows/ci.yml=35ce79ca28921f66c50e9799b9cb027d9f2bb125.Co-track this as one additional residual setup-site observation under retained #193 debt 2 / historical #194 and #209 inventory, following the existing scope distinction. It is not #208's named scoped-scan recurrence: that test passed here. #193 remains completed; its six-debt inventory and historical cohorts are not changed by this entry.
Checkout, source/isolation verification, toolchain setup and compilation passed. Steps 6 (owner-private filesystem validation) and 7 (real pending seed/proof-write refusal) were skipped after failure. The macOS management steps were OS-inapplicable. At intake, 2026-11-11 19:26:19Z, the overall run was still in progress; only this job was terminal. Ordinary CI is not green.
Retained owner evidence:
639602507bc650beb1b48e7aa2284b94a1a9a1816c69f428d73f1f3a8177cefb; checkout/source line 128, failure lines 567-568/583, summary 592.3e97287c1050c2066c64be53ab210150a45b2ac3a6650e37ca860c276c7491b1.Exact run/job dedup found no prior entry on #182 before posting. No issue creation, source change, assertion/timeout/selector change, owner reactivation, rerun or merge/publication action. The existing sole observer continues the original run; any later disposition must retain this failure.
Later outcome: authorized single-job retry
At 2026-11-11 19:57:04Z, readback confirmed attempt 2 and its sole new execution, job 130768932503, succeeded. The job ran 19:53:23Z-19:55:18Z. This was the one Campaign-authorized unchanged-head, failed-job-only retry, dispatched after attempt 1 finished with 21 successful jobs and the single recorded failure.
The retry explicitly checked out
d7bfd6e961cc80176d055c046c7af8500f4098b6; provider event SHA remained9584b240bb977f771b01562855874811b9e15f88. PR base remained4b3ff8f508598a32cef542831967e2bee69c2f5a. No source, fixture, assertion, selector or deadline changed.write_cap_produces_a_typed_error_and_never_deletespassed. The deliberate cancellable-worker panic test also passed. macOS management steps remained platform-inapplicable. The other 21 attempt-2 job entries carry forward prior successes; this is one fresh execution, not 22 new executions.The original entry above is preserved as history: its nonterminal-run and no-rerun statements describe the initial intake before this later authorization. Its original 3,500-byte body is retained privately with SHA-256
f7ca5989e3966bf08a73140beeaf1d9f15349450eecd59e68fe30b99e85af4bc. Retry log: 58,104 bytes, SHA-2564b1a9f88476caf9a7328185b60c644df5caa58d5b2ddb6141f434c028f31819c; source identity line 128, cap-test pass 567, suite verdicts 569/606/630.This green retry satisfies only the retried check. It does not erase the original refusal, establish its cause or a repair, alter the residual-debt/cohort accounting, or grant merge/release acceptance. No second retry, observer rearm, source fix, merge, tag, publication or shared-runtime action was performed.
lossyrob commentedon Nov 12, 2026
One additional residual cap-fixture setup refusal on macOS; original evidence retained before the authorized bounded retry.
PR #255 at exact source head
eb6b802813ec52b2f51bcdc213e9521a2ff08f40, baseb0d63dbb794e40e1f130f50b2243db9fcb1a7aed:Exercise complete station-intent unit boundary, rancargo test --locked --no-default-features --features sqlite --lib station_intent::tests:: -- --nocapture.eb6b802813ec52b2f51bcdc213e9521a2ff08f40; provider event SHA:9d621782a1ff3f332c224ccbb08d4e17b791fbae. The job explicitly checked out the source head, not the generated merge.Failure:
Result: 65 passed, 1 failed, 0 ignored, 474 filtered; whole module duration 3.20s, not measured acquisition latency. Failure is in initial under-cap fixture population, before overflow, guidance, no-deletion/count and rewrite assertions. The failing loop iteration and physical cause are unknown. The earlier injected worker panic was intentional and its named test passed. Subsequent owner-private filesystem, pending-seed/proof-write refusal and native macOS management steps were skipped.
Classification: this is another observation of the existing cap-setup late-success refusal boundary, as retained in the prior main observation and PR #250's Windows occurrence, now on macOS. The source blob
bf909638a1937d2706b565addc537b29442cdfe9is identical at the PR head and its base. This does not establish a common platform cause, contention, scheduling/filesystem latency, a product regression or a repair. It is not the named scoped-scan #208 case or the repaired #187 contention-timing assertion. Historical #193/#194/#209 residual-debt associations and cohorts are not changed; those closed records are not reopened.Retained owner evidence:
7be35013666f9e05e65c0bbe82d4a1b32d034cbbd0cd73db445e2dde7bcd6b0d; actual source line 125, failure lines 652-654, result line 667.ab1087bda605373de7abdd7c202ea8db5ceee1d16fbeaf65acb7703ee20c7d17.files\docs254-ci43306999770-a1evidence directory.At initial intake only this job was terminal; the sole assigned waiter reported the original overall CI and Windows archive runs still in progress. This entry is published before retry. Campaign authorized one normal bounded unchanged-head retry after classification/tracker recording; the intended action is this failed job only, after the original run finishes. No rerun has yet been dispatched. A repeated failure escalates rather than receiving another automatic retry.
No source, selector, assertion, fixture or deadline changed. No docs-only waiver, RC1 exception, new observer/poller, merge clearance, native product acceptance, release or installation action is implied. Existing waiter 2a4 remains the sole CI observer. Any retry outcome will be recorded separately and will not erase this failure.
lossyrob commentedon Nov 12, 2026
PR #255: three additional failed jobs share a branch-binary fixture-resolution frontier; separate from the recorded macOS cap-setup refusal.
All observations below are CI run 43306999770, attempt 1, pull_request event, source head
eb6b802813ec52b2f51bcdc213e9521a2ff08f40and baseb0d63dbb794e40e1f130f50b2243db9fcb1a7aed.Each job's four failures are the applicable SQLite or Postgres variants of runtime hardening, backend resilience, sender outage and the runtime matrix. Every one reaches
tests/support/telex_isolation.rs:56:5, the setup assertion:The logs show the running Watcher unit-test executable itself beneath that same
build/telex-watcher/<hash>/outdirectory:35df3a119a1c8c4a, log line 756; first failure lines 768-769, suite result 892.7072adc7b8b4270d, log line 2555; first failure lines 2645-2646, suite result 2753.2c8b3de22586177b, log line 871; first failure lines 883-884, suite result 991.Concrete classification: branch-binary resolution/setup failure, not a reached runtime-contract assertion or an observed Postgres outage. In
branch_binary()at lines 41-61, a valid compile-timeCARGO_BIN_EXE_telexwould be used first; otherwise the fallback walks up only from a parent nameddeps, then appendstelex[.exe]. With the observedoutparent it constructs exactly the missing path in these logs. The logs do not establish whether the compile-time value was absent or invalid, or the continued existence/location of another branch binary at the assertion.All three logs report stable updating from Rust 1.99.1 to 1.100.0. That is a co-observation consistent with a layout-compatibility investigation, not a controlled proof of a toolchain cause. No local or old-toolchain reproduction was run. Existing workspace build commands had succeeded before the fixture assertions; blindly repeating them is not established as a repair.
Checkout distinction: unlike the macOS lock job, these three jobs actually checked out provider merge
9d621782a1ff3f332c224ccbb08d4e17b791fbae. Its authenticated Git object has parents baseb0d63dbb...and sourceeb6b8028..., and tree9f1fd3680a472ac21b004311aeaaf86ea29cd331, identical to the source-head tree. The fixture file blob is41359c65e78073f2196e03acd3d05f8b69d72057at both base and head. None of the six docs/canonical changes edits this helper.Raw owner evidence, retained in
files\docs254-ci43306999770-a1:995edce0e3382be92a0a97ba5f683e028ad9cb073e0f2c40c40febc5fa088f1de2ad41e5a3d43d8339e6d4efe46d2481ac58b7eba7e897ab576cba4da8143c50ae7d498ee7a26d684aa062ee16bd8f01355afbbc6322b552ead7b78588176ac7Per-job metadata is retained alongside each log. This is three actual failed jobs and twelve failed test invocations at one shared setup frontier, not twelve independent root causes. The macOS cap-setup occurrence is a separate fourth job and remains separately classified.
Tracker disposition: retain these occurrences in existing open health log #182. Targeted tracker search found no dedicated issue for this exact branch-binary message. No new issue, owner activation or historical closed-item reopening is inferred. Campaign must select any revised retry/fix custody; the earlier single macOS-job retry plan does not resolve or authorize action on this expanded set. A fixture-resolver compatibility correction is the concrete investigation frontier, not an accepted fix.
The sole observer's interim report still had the original run nonterminal (21 completed, 1 running), with four completed failures, and the archive nonterminal (2 successful, 1 running). No retry, source change, selector/assertion/deadline change, alternate toolchain run or duplicate observer has been started. Head remains unchanged. No docs-only waiver, RC1 exception, merge clearance or macOS product acceptance is claimed.
lossyrob commentedon Nov 12, 2026
PR #255 terminal attempt: fifth failed job independently preserved and classified.
Build and test (windows-latest), job 131492812306, CI 43306999770, attempt 1, failed step 20
Testand completed at 2026-11-12T21:22:09Z.The independently downloaded log shows these four failed SQLite cases:
runtime_hardening_tests::sqlite_watcher_runtime_hardeningruntime_resilience_tests::sqlite_watcher_backend_resilienceruntime_resilience_tests::sqlite_watcher_sender_outageruntime_tests::sqlite_watcher_runtime_matrixAll four panic at the setup assertion
tests/support/telex_isolation.rs:56:5, with the missing candidate:The actual running Watcher test executable is
target\debug\build\telex-watcher\efcc4a720278c8de\out\telex_watcher-efcc4a720278c8de.exe(line 2982). First failure and candidate are at lines 3076-3077; result line 3284 is 78 passed, 4 failed, 2 ignored, 0 filtered, 143.61s whole Watcher library duration. That duration is not branch-resolution or lock-acquisition latency.This independently observed frontier matches the three earlier branch-binary fixture-resolution occurrences:
branch_binary()only walks upward when the executable's parent isdeps, not the observedoutparent. It fails before the four runtime-contract assertions. The reason for the compile-time branch-path fallback and toolchain causality remain unproven; Rust 1.100.0 and the relocated test executable are observed, not a controlled old/new-toolchain comparison.Actual checkout and the structured CI context identify provider merge
9d621782a1ff3f332c224ccbb08d4e17b791fbae, not direct source checkout. Its tree equals source headeb6b802813ec52b2f51bcdc213e9521a2ff08f40's tree9f1fd3680a472ac21b004311aeaaf86ea29cd331; parents are baseb0d63dbb794e40e1f130f50b2243db9fcb1a7aedand that source head. The fixture blob remains unchanged from base.Raw Windows job log: 1,634,457 bytes, SHA-256
0b87c4b24195dd0a07bf36f2ad75658e247961a49f0762aa0774f70a25a0a8bc. Job JSON: 8,185 bytes, SHA-25641bcc1128f1fe08c60eadcc6be16e4bd23500baea7efd4771105b86ee33b7239. Both are retained in the owner'sfiles\docs254-ci43306999770-a1directory.The original exact-head pull_request attempt is now terminal FAILURE: 22 actual jobs, 17 successful and 5 failed. Complete failed-job set is 131492811850, 131492812268, 131492812296, 131492812306 and 131492812408. Four jobs now have independently retained branch-binary fixture setup failures; the macOS cap-setup job is a separate boundary.
The terminal-attempt JSON (12,525 bytes, SHA-256
0bd23e93c2726bb4737cae6cb8155a7e503c2f6e069615b98cb0a63181c32716) and all-job JSON (66,976 bytes, SHA-2561b2e835a9cc7c99b6a686efccb37aac5a8250082fd05b2ab5c047442757f3902) are retained. Earlier nonterminal statements describe their earlier intake times.No retry or fix has been performed. Campaign owns revised expanded-set retry/fix custody; the earlier single-macOS-job plan is not automatic authority for this set. The sole waiter reports its wait done and schedule cleared; no ongoing observation is implied. The separate Windows archive was still nonterminal at that report and is not green. Source head is unchanged; no duplicate poller, docs-only waiver, RC1 exception, merge clearance or product acceptance is claimed.
lossyrob commentedon Nov 12, 2026
PR #258 exact-head Live Postgres occurrence, distinct from the earlier branch-binary setup frontier and the separate macOS cap-fixture occurrence.
d5ae063b653d2711124ee83701fd41d2030133e6The repaired resolver passed its prior setup boundary in this job: the four targeted Postgres Watcher tests reached runtime assertions rather than failing with
branch telex binary not found.Observed terminal failures:
postgres_watcher_runtime_hardening:condition not reached; the staged detector accumulated consecutiveprocess-failureattempts without reaching the expected state.postgres_watcher_backend_resilience:state did not advance; the detector accumulated sevenprocess-failureattempts.postgres_watcher_sender_outage:state did not advance; the detector accumulated sevenprocess-failureattempts.postgres_watcher_runtime_matrix:actor failed to complete; the detector accumulated fiveprocess-failureattempts.The resilience evidence showed the Application Client membership registered with
sender_ready: true,receive_ready: false, and no inbound pending/actionable rows. That proves only sender membership/readiness at the observed point; it does not prove detector completion or establish the runtime failure's cause.Same-head evidence at classification time:
131529370835: success.131529370341: failure.Current disposition: Campaign merge HOLD remains. Existing #256 owner session
818c549c-a544-4d97-8542-d7d6ca5c5fb9is classifying whether this runtime frontier is tightly coupled to the fixture change or a distinct unmet prerequisite. No blind retry, assertion or timeout waiver, detector/product rewrite, duplicate writer/observer, root-cause claim, or merge authorization follows from this receipt. Any successor head requires fresh independent exact-head review and all four required Windows/Linux SQLite/Postgres axes.lossyrob commentedon Nov 12, 2026
PR #258 run 43317541046 terminal follow-up to the earlier Live Postgres receipt: all four required axes are now terminal on exact head
d5ae063b653d2711124ee83701fd41d2030133e6.Required-axis result:
binary_owned_stall_detector_reads_live_management_and_readonly_storefailed its direct branch-Telex path assertion attelex-watcher/tests/cli_v2.rs:453.Ubuntu details:
sqlite_watcher_runtime_hardening: expected condition not reached after repeated detectorprocess-failureattempts.sqlite_watcher_backend_resilienceandsqlite_watcher_sender_outage: state did not advance after seven detectorprocess-failureattempts.sqlite_watcher_runtime_matrix: actor did not complete after five detectorprocess-failureattempts.sender_ready, which is not detector completion or root-cause evidence.Windows detail:
binary_owned_stall_detector_reads_live_management_and_readonly_storecomputes the Telex candidate directly from the integration-test executable ascurrent_exe().parent().parent().join("telex.exe").run this test with cargo test --workspace.tests/support/telex_isolation.rshelper. Whether it is the same relocated-layout defect and belongs in [test-sentinel] Resolve branch Telex binary from relocated Watcher test executables #256, or a distinct unmet prerequisite, remains under owner classification.Disposition remains unchanged: Campaign merge HOLD, existing #256 owner
818c549c-a544-4d97-8542-d7d6ca5c5fb9, no retry or scope expansion yet. The owner must determine whether the Linux detector failures and Windows direct-path failure are tightly coupled to the fixture correction or require normal upstream/discovered-work routing. No assertion/timeout waiver, detector/product rewrite, duplicate writer/observer, green inference, or merge authorization follows. Any successor head requires fresh independent exact-head review and all four required axes.lossyrob commentedon Nov 12, 2026
PR #258 exact-head successor CI disposition before any retry or head movement.
79a9127871249d76fe81d584c33fd819a2ba0bd83bb75ce397a58f790429bee5b348d1ea0c9106a1Required-axis result:
Linux details:
Prepare Linux Watcher test guardians, the full workspaceTeststep, and the complete SQLite Application Client suite all passed first.Record Watcher capacity and recovery sizing evidencestep then rebuilt the all-features Watcher unit host after the one-time staging step:target/debug/build/telex-watcher/2c8b3de22586177b/out/telex_watcher-2c8b3de22586177b.sqlite_watcher_runtime_hardeningrepeated seven immediateprocess-failureattempts with 5 ms latest duration, null stderr, and no committed event, then panicked attelex-watcher/src/runtime_hardening_tests.rs:199.b69ed6b11ebb0cb4f6bf23f15c2700c5c046287cfce331281aa353a908143cc3.Windows details:
credential_command_shell_exit_preserves_delayed_inherited_writertest exceeded its existing five-second budget attests/credential_command.rs:848.Current disposition: Campaign merge HOLD remains. Existing #256 owner
818c549c-a544-4d97-8542-d7d6ca5c5fb9is absorbing only the omitted all-features guardian staging into the same narrow fixture correction. No unchanged-head retry, detector/product rewrite, assertion or timeout waiver, duplicate writer, root-cause claim, CI-green inference, or merge authorization follows. Head movement invalidates the pending review and all current CI evidence; a successor requires fresh independent exact-head review and all four required axes.lossyrob commentedon Nov 13, 2026
PR #258 final exact-head candidate packet for #256.
7eb562f07cabebe914e639ec99b2abba718fb8a23bb75ce397a58f790429bee5b348d1ea0c9106a1The final candidate keeps the original strict branch-Telex resolver, reuses it for the Windows CLI fixture, and stages the exact Cargo-reported
watcher_process_guardbeside each relocated Linux Watcher test host. The late all-features capacity step now performs its own exact-feature no-run preparation and guardian staging before execution, inside the unchanged ten-minute budget. Production process lookup, detector logic, assertions, selectors, backend coverage, and Windows commands/budgets are unchanged.Required exact-head axes:
All 25 reported checks are terminal success on the same head. This includes the corrected late Linux all-features capacity path and a non-recurrence of the separately routed Windows credential timeout. The green Windows observation does not close or select a cause for reopened #167.
Fresh independent exact-head implementation review:
25e371bf-96e5-4b8f-bbc4-b2e864a2adb058a3e6b4-f765-4f78-a25a-af763cdb9ee5gpt-6.1-sol,xhigh,long_context, interactive7eb562f07cabebe914e639ec99b2abba718fb8a2, base3bb75ce397a58f790429bee5b348d1ea0c9106a1, complete six-file diffgit diff --check;cargo fmt --checkCurrent disposition: candidate, review, and affected CI evidence are complete. Existing owner remains
818c549c-a544-4d97-8542-d7d6ca5c5fb9. Campaign merge authorization is still required. No release/install/host action, #167 closure, #172 dependency, #252 change, detector/product rewrite, assertion or timeout waiver, or canonical discovery-ledger publication follows from this packet.lossyrob commentedon Nov 13, 2026
New Windows production-clock Live-fixture seed occurrence; evidence-only residual intake.
PR #260 exact source: f122c6d, tree 9f82ef57737bfb804882dd9f3e5c87940d138cab (not a synthetic checkout).
Run 43463739914, attempt 1; Station-intent lock boundary (Windows), job 132017202712:
https://github.com/lossyrob/telex/actions/runs/43463739914/job/132017202712
The command cargo test --locked --no-default-features --features sqlite --lib station_intent::tests:: -- --nocapture failed only station_intent::tests::write_cap_produces_a_typed_error_and_never_deletes, at src/station_intent.rs:3788:41. Original log638-639 reports "write under cap" and "late successful acquisition on attempt 1; guard released". The failure occurs in the up-to-512-record seed loop BEFORE overflow, typed CapExceeded or no-delete assertions. Result: 65 passed, 1 failed, 0 ignored, 489 filtered; 27.04s is suite duration, not acquisition latency.
The first acquisition returned a guard; the real post-acquisition elapsed-time check found the 480ms budget exhausted and released it before record mutation. This does not establish a WouldBlock/retry loop, a competing writer, cap violation, deletion, or the duration of a particular syscall. Filesystem, scheduling and other underlying contributions remain unresolved. The earlier "injected worker failure" belongs to a deliberate passing test, not this failure or a native crash.
The station_intent.rs blob bf909638a1937d2706b565addc537b29442cdfe9 is unchanged across known 6a/215da/d71/f122/a9a4 sources. This is not a demonstrated PR #260 regression and must not be absorbed into its macOS bootstrap repair.
Existing precise co-tracking:
#208 (comment)
That record already distinguishes this cap-test seed refusal from #208's named scoped-scan test. Retain it as ADDITIONAL production-clock Live-setup debt under the #194/#209 residual boundary, not a new named scoped-scan recurrence or proof of a shared host cause. #208 remains unresolved; #193 is not reopened and #167 is not implicated.
The sole CI observer supplied the log; the retained reviewer classified those existing bytes; Local independently verified all ten files in the frozen three-job packet. Campaign read and hash-verified this 4658-byte classification, SHA256 51fc16fb704f4567e2161ae10d3071587d77ceb4daf1d29fb1a7b478230d573b. Original log: 60665 bytes, SHA256 3134a3720f0a5741b88e99658105c1d936a0d84baa7742d0bf2b993e386b0815.
No new implementation owner, repair, runtime reproduction, provider log fetch, rerun, deadline relaxation, assertion waiver, CI acceptance or release clearance follows. Whole-run completion is not established by this classified job. Existing sole-observer diagnostic custody and PR #260 source/native/public-sender holds remain separate.
lossyrob commentedon Nov 13, 2026
Actual RC2 merged-source CI failed because one macOS diagnostic job never acquired a hosted runner. Record this before any bounded job rerun; it is not a product-test failure or a waiver.
9facc00444852c324c315d371dbdef6e5a5a1b66, tree9cbcc03c954a45f953cb06b92846b70560db2ecd.Local verified the retained annotation (1400 bytes, SHA256
c9da308a87bf59ab7777dc5cd00a0d94303e7a0ae963328345d0326c2afdb895), jobs metadata (66408 bytes, SHA25623d60c08c96a611cbb0b37b7eb63a40c9b7c1db004fbab0874a95f8cfe28c551) and run metadata (11739 bytes, SHA256074ba2e947b453dff5507038a89d681915604995d89a6e73737efeb134d5686e). Campaign is retaining that attributed evidence, not claiming another CI/log query.The provider binds the head SHA only. There is no emitted checkout/tree/host or product assertion from the cancelled job. Hosted-runner acquisition failed; no deeper infrastructure cause or native macOS failure is inferred. Separate successful default+Entra native release proofs and the Watcher archive do not turn this diagnostic axis green.
Campaign verified the documented job-rerun endpoint reruns the selected job and any dependent jobs. Immutable ci.yml at this source has no needs or YAML merge keys, so this independent diagnostic job has no downstream jobs to rerun. A separately authorized single POST to this job's rerun endpoint can therefore preserve the 21 successful jobs; no full-workflow or failed-jobs rerun is intended.
This record is not a rerun result. Ordinary CI remains nongreen and RC2 tag/publication remains stopped. No source change, assertion/budget waiver, automatic second retry, PR260/macOS-repair dependency or shared-install/runtime action follows.