Repository navigation
refactor: lift the CLI core, health and write helpers into the engine - #55
Merged
Merged
Conversation
Each destination package carried a copy of the command line, the flag and environment parsers, the entrypoint check, an identical health module, an otel module differing only by service name, and the write-path helpers. All of it now lives once in @workos/sync-engine. `runCli(spec, argv, deps)` is the command: commands, flags, exit codes, help text, health server and OpenTelemetry bootstrap. A package's `cli.ts` is a `CliSpec` of what differs: how its destination is built, what `init` reports, the DDL `init --print-sql` prints, and the words the help needs. The row helpers took the strictest of the four versions: `keyValues` refuses an object where a key should be, `chunk` refuses a size that would never advance, and the timestamp error code is INVALID_TIMESTAMP everywhere. The duplicated unit tests moved with the code; each package keeps the tests that run its command against its database.
[High impact] The PR appears safe to merge; the previous Postgres timestamp mismatch is fixed. SummaryThe PR moves shared commands, health settings, OpenTelemetry startup, and row helpers into
|
…e duplicates by its own parser `Date.parse` reads a timestamp without a zone as the process's local time, where every destination stores it as UTC, so a batch could keep the wrong duplicate. The shared helper reads such a timestamp as UTC, and `dedupeByKey` takes a destination's own parser; MongoDB passes its strict one, so the rows it discards are judged by the rules the rows it keeps are written by, and a bad timestamp fails before any row is discarded, as it did before.
…session TimeZone The writes cast `updatedAt` with `::timestamptz`, which reads a string without a zone in the session's zone, where the catalog's `parse_iso8601`, the other destinations and the engine's `isoInstant` read it as UTC; a pooler can hand out sessions in any zone. A string with an offset still goes through `timestamptz`, which honours it; one without is read as UTC. Text that is no timestamp still fails the cast.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
init --print-sql), the seven flag and environment parsers, the entrypoint check, an identicalhealth.ts, anotel.tsdiffering only by service name, and the write-path helpers (readPath,keyValues,dedupeByKey,wavesByKey,chunk,micros,isBefore, withjsonEqual,parseUpdatedAtandcanonicalJsonin two or three). All of it now lives once, in@workos/sync-engine.runCli(spec, argv, deps)is the command: commands, flags, exit codes, help text, health server and OpenTelemetry bootstrap. Each package'scli.tsis aCliSpecof what differs: how its destination is built, whatinitreports, the DDLinit --print-sqlprints, and the words the help needs. The CLIs are now 81 to 111 lines each.keyValuesrefuses an object where a key should be (MongoDB's rule),chunkrefuses a batch size that would never advance (MongoDB's), and the timestamp error code isINVALID_TIMESTAMPeverywhere (MySQL'sER_INVALID_TIMESTAMPis gone; its classifier never keyed on it).startOteltakes the service name from the spec, and the SDK moves to the engine's optional dependencies, where the import now happens; the packages keep theirs for the container image.Net: 46 files, roughly 1,400 lines added and 4,300 removed.
Testing
Engine 164, Postgres 226, MySQL 160, MongoDB 140 and ClickHouse 163 tests pass locally, with lint, typecheck and a full build clean. The CLIs' behaviour is unchanged, which the per-package command tests (print-sql, init, backfill, reconcile, rejected, redact, help text) cover against each database.