Repository navigation
feat(datastore): preserve history through legacy bucket aliases - #836
TimeToBuildBob wants to merge 51 commits into
Conversation
…_id,name)) Adds `device_id` to the `buckets` table so buckets from different devices can share a bucket name without colliding. The new uniqueness constraint is `UNIQUE(device_id, name)`, replacing the old `UNIQUE(name)`. Migration (v6→v7): - Recreates the buckets table with the new schema (SQLite cannot alter UNIQUE constraints in-place). - Stamps all existing rows with `device_id = 'local'` (locally-created). - Also drops the stale `data_deprecated` column (unused since v3). - FK enforcement is disabled around the recreation to avoid the immediate events→buckets FK check firing on DROP TABLE mid-batch. Read-only opens (aw-sync peer DBs at v4-v6): - `get_stored_buckets` falls back to `'local'` literal when `db_version < 7`, so aw-sync can still read older peer databases without migrating them. `create_bucket` stamps `device_id = "local"` for any bucket created without an explicit device_id (local watchers). Peer buckets imported via aw-sync carry their own device_id. Also fixes pre-existing duplicate `#[allow]` attributes in aw-query that were breaking the clippy pre-commit hook under the current toolchain. Git-Session-Id: 77b9
Git-Session-Id: c66cd073-07ec-4980-883b-2f3c24dd5be1
Complete the fix across all test and benchmark Bucket struct literals that weren't updated when device_id was added to the model. Git-Session-Id: da0eaaaf-bdea-5882-addc-bba90b82acea
buckets_cache is keyed by bucket name only. A second insert with the same name but a different device_id would silently overwrite the first cache entry, hiding its history and routing new events to the wrong bucket. Reject until the cache key includes device_id (follow-up refactor). Git-Session-Id: da0eaaaf-bdea-5882-addc-bba90b82acea
…te rejection Add three tests for the storage guarantees introduced by the device_id column: - empty device_id is stamped "local" on creation - explicit device_id survives the round-trip - same-name bucket with a different device_id is rejected (cache guard) Git-Session-Id: da0eaaaf-bdea-5882-addc-bba90b82acea
…new_in_memory(false) Git-Session-Id: dde9bcf8-3b8e-589f-bdc8-7b22fa16c358
…ckets The guard added in d446f37 rejected same-name buckets with a different device_id to protect the name-keyed buckets_cache, but UNIQUE(device_id, name) exists precisely to allow (A, "x") and (B, "x") to coexist — the guard made the feature unusable for its stated purpose. Remove the guard and let the UNIQUE constraint be the sole duplicate guard (same device_id + name still maps to BucketAlreadyExists). Test flipped from asserting rejection to asserting coexistence plus the same-device duplicate rejection path. Git-Session-Id: 836f0e43-c054-59df-bd31-567140d6ae5d
Git-Session-Id: 9295d79d-f715-58c1-aca0-8588355870ff
Git-Session-Id: 18163d54-c6a9-5fd9-a7c3-e85d1aed668f
Git-Session-Id: b59ff0c6-06f5-5ed5-8eaa-6141ff07f3d9
Git-Session-Id: b59ff0c6-06f5-5ed5-8eaa-6141ff07f3d9
… migration lock - rename_bucket checks the destination name inside the UPDATE so a stale cache cannot create an ambiguous (device_id, name) pair - v6->v7 migration carries the AUTOINCREMENT high-water mark over the table rebuild so deleted bucket ids are never reused - re-read user_version after taking the exclusive lock so a concurrent opener's finished migration is not rebuilt over its device IDs Git-Session-Id: b51e5cce-319b-4ec1-8c25-2a1eb9a205e6
After merging master (ActivityWatch#776), _infer_db_version returned 2 for the v7 schema: v7 drops data_deprecated, which the v2/v3 check reads as 'no rename yet'. Recognise device_id first. The early-schema dump-restore test now asserts v7 dropped data_deprecated instead of preserving it. Git-Session-Id: 0c1512d2-dbce-4257-97f6-9b08bbe8dcb8
Git-Session-Id: d8c65c8d-a38a-5d02-bbdc-63034939f80f
Git-Session-Id: b01e1b01-a683-5ca8-805e-c6dd7d8436c0
A concurrent opener that read v5 before the first opener finished the v7 migration would regress user_version back to 6 (or 5) when its own migration step finally took the exclusive lock, causing the v7 rebuild to re-run and overwrite stored device IDs with 'local'. Mirror the existing v6->v7 in-lock recheck in _migrate_v4_to_v5 and _migrate_v5_to_v6: after taking the lock, skip the step if the version is already applied. Git-Session-Id: ea4f2eca-681c-5fde-b350-3502103e7d4c
Git-Session-Id: 9749f553-4dc2-5998-9ffd-3d98a62bae37
…story Git-Session-Id: 0773711b-88a1-4a62-aa84-a1721b7a3cc2
Git-Session-Id: 0773711b-88a1-4a62-aa84-a1721b7a3cc2
[High impact] The changes reach bucket storage and the server’s device-scoped data routes. The PR needs fixes before merging because tests cannot compile and device-scoped writes can remove another device's events. Findings
SummaryAdds hostname aliases and device-scoped APIs while preserving stored bucket names.
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
Request[Device-scoped request] --> Normalize[Normalize local UUID]
Normalize --> Handle[Scoped datastore handle]
Handle --> Worker[Serialized worker]
Worker --> Resolve[Resolve exact name or hostname alias]
Resolve --> Bucket[Stored bucket row]
Bucket --> Events[Read or write events]
Worker --> Export[Grouped export]
Export --> UUID[Replace local sentinel with source UUID]
UUID --> File[Stage JSON in private tempfile]
Reviews (2) · Last reviewed commit: "Merge origin/master into feat/device-add..." · Reviewed by Greptile |
Git-Session-Id: 1fe79215-f972-41ee-9d1a-2ef78e3f9410
…ests Git-Session-Id: 1fe79215-f972-41ee-9d1a-2ef78e3f9410
Git-Session-Id: 1fe79215-f972-41ee-9d1a-2ef78e3f9410
|
The 2/5 summary reviews def5ea9, not current head 1e0a3c5. Its three findings already have pushed repairs and individual replies: rename alias protection in 96b1e34; creation/write target agreement and hostname-sensitive reader refresh in 1216619, retained through the later cache/routing changes. I checked the current source and all three named regression tests; I did not rerun tests in this monitoring pass. The implementation owner is actively extending this draft in the same leased worktree. I left that work untouched and handed back the review refresh for the next verified push. No additional fix or Greptile retrigger from this dispatch; the old score is not evidence that the current head has passed review. |
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: 73203c26-75f4-4756-b450-2d9f5877e9fc
Git-Session-Id: ee485138-49ef-455b-9c15-d08a96a24bd7
|
Follow-up |
|
Convergence adjudication deferred by the freshness gate. Rechecked 2026-10-23 17:44 UTC: current head No findings classified, dismissed or resolved in this adjudication pass; no review retrigger or merge. The detector reports Updated this existing deferral rather than adding another status comment. Earlier checks at |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #836 +/- ##
===========================================
+ Coverage 70.81% 87.33% +16.51%
===========================================
Files 51 94 +43
Lines 2916 14938 +12022
===========================================
+ Hits 2065 13046 +10981
- Misses 851 1892 +1041 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
|
Integration update at ee18948 (fix 6f9efed): reproduced both current correctness findings before repair. A forged explicit event ID moved a different device’s event (200); guarded peer creation returned 304 although scoped GET was 404. Whole-batch ID ownership preflight now precedes insertion, covering event and heartbeat inserts while preserving same-bucket replacement. Rejected batches leave no accepted prefix. Guarded peer creation and import collision return 409. Datastore/query/server suites and production-target strict Clippy pass; 14 device HTTP tests and 14 resolver fixtures pass. ee18948 verifies scoped alias privacy uses the stored bucket name without filtering a different peer name. Compilation was already repaired by a3c3d64. Both duplicate-name guards remain; no full same-name acceptance or release claim. Shared real-HTTP fixtures in ActivityWatch/aw-server#200 pass both transfer directions and alias/query/heartbeat behavior (8 cases). |
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
|
Further guarded acceptance is pushed through a3bca14. 1a16373 keys heartbeat cache/invalidation/privacy fallback by (device_id, stored id); an internal same-name fixture reproduced cross-device cached event-ID failure before repair. 9a81923 includes ownership in reader signatures (old schemas remain supported), with real open-reader ownership-repair coverage. a3bca14 propagates signature/reload failures instead of serving stale identity: a real-file fixture reproduces a rejected ambiguous catalogue, verifies reads/list fail closed, then verifies recovery after repair. Full datastore/query/server suites and strict production-target Clippy pass; 16 resolver fixtures pass. Both duplicate-name guards remain. Remaining work is coherent same-name legacy-index/list/export/rename integration plus the complete acceptance matrix. #826 remains separately owned; its delete/rename invalidation will need the pair keys. No production migration or complete identity claim. |
Git-Session-Id: e9187c4d-6f8e-4dcd-8348-b5182b82f9d2
Git-Session-Id: e9187c4d-6f8e-4dcd-8348-b5182b82f9d2
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: 6ce10d23-d723-441f-a1a9-a57de198bdd8
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
Git-Session-Id: c24867b5-ab58-475a-a9fd-78ac3d0e096f
|
Guarded continuation pushed through d9b1805. Three reproduced repairs: 8c95cad scopes hostname repair and verifies open-reader aliases; 3e9f4ec retains local authority through legacy Python import/canonicalization/dedup with caller-scope restoration; f47b8c9 refreshes migration destinations before mutation, rejecting a newly created peer destination without changing history. d9b1805 adds exact v6→v7 row/name/archive/event-association preservation, including pre-epoch nanoseconds and binary archive bytes; existing migration code passes. Local verification: datastore/query/server 348 passed / 2 ignored, strict production-target Clippy and formatting passed; workspace test targets compile; aw-sync 221 passed; eight shared real-HTTP Python/Rust cases pass (distinct stored names sharing aliases). Normal hooks passed. Both duplicate-name guards remain; this is not enabled same-name coexistence, completed identity acceptance, upstream release or production migration. Contributor-owned #826 remains separate and its eventual invalidation integration must use pair keys. Deferred recovery retains fresh rollback backups on incomplete attempts; no completion markers or retained history were weakened. |
Git-Session-Id: f5ebad7c-cb7d-440b-ad2a-d8782ae33b6e
Git-Session-Id: f5ebad7c-cb7d-440b-ad2a-d8782ae33b6e
Git-Session-Id: f5ebad7c-cb7d-440b-ad2a-d8782ae33b6e
Git-Session-Id: c81281b2-9cbe-4e63-a85e-e227184324d1
…ing-e78b Git-Session-Id: 5f08faae-877d-49bf-8a88-03199ec2ec47
The merge of ActivityWatch#826 left its invalidation calls removing a bare bucket id from the (device_id, id)-keyed cache, which does not compile. Drop the resolved device's entry on DeleteEventsById, DeleteBucket and both ends of RenameBucket; without a resolved device, drop that id on every device. Same-name regressions: deleting a device's cached event or recreating its bucket starts a new event for that device, while a peer with the same bucket id keeps merging into its own event. Git-Session-Id: 5f08faae-877d-49bf-8a88-03199ec2ec47
Git-Session-Id: 5f08faae-877d-49bf-8a88-03199ec2ec47
Summary
Device-identity integration for ActivityWatch/activitywatch#302, stacked on schema #786 at
d74c27eb425b. Do not merge before #786. Remains draft: Python parity and full identity acceptance are incomplete.(device_id, id)bucket cache; scoped worker requests retain scope through event/list/export/query operations and restore it after errors. Reader signatures include hostname repairs.localselect the same storage namespace; peerhostname="!local"is rejected.87a8619: complete local-plus-stored-peer catalogue and lossless grouped list/export/import. Grouped envelopes use{"devices":{"<source-device-id>":{"buckets":{"<bucket-id>":{...}}}}}. Local sentinel is exported as source UUID, so importing on another server cannot silently localize source history. Events serialize one row at a time from one worker transaction to a private tempfile before HTTP success. Full envelope identity preflight precedes writes; runtime failures can leave earlier groups imported, with cache invalidation on failure. Peer hostname is a deterministic representative of bucket metadata, not a latest-name registry.Routes
GET /api/0/devices— local UUID (including empty server) and stored peersGET /api/0/devices/buckets— grouped metadataGET /api/0/devices/export,POST /api/0/devices/import— grouped transfer/api/0/devices/{device_id}/buckets— GET/api/0/devices/{device_id}/buckets/{id}— GET/POST/DELETE/events,/events/{event_id},/events/count,/events/deleteand/heartbeat?pulsetime=.../api/0/devices/{device_id}/buckets/{id}/exportand/export/csv/api/0/devices/{device_id}/export,/import,/queryBoth global duplicate-name creation and duplicate-cache-load guards remain. Still missing: Python addressing parity, enabled same-name coexistence and full backend-specific migration/reader acceptance. Watcher rollout and V2 segment import remain separate follow-ups. No production database or published package changed.
Verification at
87a8619cargo test -p aw-datastore -p aw-query -p aw-server: passed, including 11 device HTTP fixtures. Additional configured API-key fixture covering all four grouped routes passed separately.cargo clippy -p aw-datastore -p aw-query -p aw-server --lib --bins -- -D warnings, formatting, whitespace and normal commit hooks passed. Web UI not built. Strict all-target Clippy is not claimed green (pre-existing query test-fixture warning).!localmetadata, import UUID normalization and collision status.make test-fast: 1505 passed / 1 skipped;make typecheck: 286 package and 1803 script files passed. These are not Rust coverage. Upstream current-head CI/review remains separate.The Greptile 2/5 summary reviews
def5ea9, not current head. All three named findings already have individual replies and pushed repairs: rename protection96b1e34, creation/write agreement and hostname-sensitive reader refresh1216619, retained through later integration. Heartbeat rename/delete invalidation is separately owned by #826.