Skip to content

refactor: lift the CLI core, health and write helpers into the engine - #55

Merged
gjtorikian merged 4 commits into
mainfrom
refactor/lift-shared-into-engine
Nov 6, 2026
Merged

gjtorikian merged 4 commits into
mainfrom
refactor/lift-shared-into-engine

Conversation

@gjtorikian

Copy link
Copy Markdown
Collaborator

Summary

  • The four destination packages each carried a copy of the command line (about 566 lines, differing only in wording, the CA variable, the client type and init --print-sql), the seven flag and environment parsers, the entrypoint check, an identical health.ts, an otel.ts differing only by service name, and the write-path helpers (readPath, keyValues, dedupeByKey, wavesByKey, chunk, micros, isBefore, with jsonEqual, parseUpdatedAt and canonicalJson in two or three). All of it now lives once, in @workos/sync-engine.
  • runCli(spec, argv, deps) is the command: commands, flags, exit codes, help text, health server and OpenTelemetry bootstrap. Each package's cli.ts is a CliSpec of what differs: how its destination is built, what init reports, the DDL init --print-sql prints, and the words the help needs. The CLIs are now 81 to 111 lines each.
  • The row helpers took the strictest of the four versions: keyValues refuses an object where a key should be (MongoDB's rule), chunk refuses a batch size that would never advance (MongoDB's), and the timestamp error code is INVALID_TIMESTAMP everywhere (MySQL's ER_INVALID_TIMESTAMP is gone; its classifier never keyed on it).
  • startOtel takes the service name from the spec, and the SDK moves to the engine's optional dependencies, where the import now happens; the packages keep theirs for the container image.
  • The duplicated unit tests moved with the code: parsers, entrypoint, help rendering, health options, the SDK bootstrap and the row helpers are tested once in the engine; each package keeps the tests that run its command against its database.

Net: 46 files, roughly 1,400 lines added and 4,300 removed.

Testing

Engine 164, Postgres 226, MySQL 160, MongoDB 140 and ClickHouse 163 tests pass locally, with lint, typecheck and a full build clean. The CLIs' behaviour is unchanged, which the per-package command tests (print-sql, init, backfill, reconcile, rejected, redact, help text) cover against each database.

Each destination package carried a copy of the command line, the flag
and environment parsers, the entrypoint check, an identical health
module, an otel module differing only by service name, and the
write-path helpers. All of it now lives once in @workos/sync-engine.

`runCli(spec, argv, deps)` is the command: commands, flags, exit codes,
help text, health server and OpenTelemetry bootstrap. A package's
`cli.ts` is a `CliSpec` of what differs: how its destination is built,
what `init` reports, the DDL `init --print-sql` prints, and the words
the help needs. The row helpers took the strictest of the four versions:
`keyValues` refuses an object where a key should be, `chunk` refuses a
size that would never advance, and the timestamp error code is
INVALID_TIMESTAMP everywhere. The duplicated unit tests moved with the
code; each package keeps the tests that run its command against its
database.
@greptile-apps

greptile-apps Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; the previous Postgres timestamp mismatch is fixed.

Summary

The PR moves shared commands, health settings, OpenTelemetry startup, and row helpers into @workos/sync-engine.

  • All four destination command lines now use the engine's shared command flow.
  • All four databases now use the engine's shared row-writing helpers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  C[Destination commands] --> S[CliSpec]
  S --> R[runCli]
  R --> E[SyncEngine]
  E --> D[Destination writes]
  H[Shared row helpers] --> D
Loading

Reviews (3) · Last reviewed commit: "fix(postgres): read an updatedAt without..." · Reviewed by Greptile

Comment thread packages/sync-mongodb/src/db/upsert.ts Resolved
…e duplicates by its own parser

`Date.parse` reads a timestamp without a zone as the process's local
time, where every destination stores it as UTC, so a batch could keep
the wrong duplicate. The shared helper reads such a timestamp as UTC,
and `dedupeByKey` takes a destination's own parser; MongoDB passes its
strict one, so the rows it discards are judged by the rules the rows it
keeps are written by, and a bad timestamp fails before any row is
discarded, as it did before.
Comment thread packages/sync-engine/src/rows.ts Resolved
…session TimeZone

The writes cast `updatedAt` with `::timestamptz`, which reads a string
without a zone in the session's zone, where the catalog's
`parse_iso8601`, the other destinations and the engine's `isoInstant`
read it as UTC; a pooler can hand out sessions in any zone. A string
with an offset still goes through `timestamptz`, which honours it; one
without is read as UTC. Text that is no timestamp still fails the cast.
@gjtorikian
gjtorikian merged commit 68b9e3f into main Nov 6, 2026
25 checks passed
@gjtorikian
gjtorikian deleted the refactor/lift-shared-into-engine branch November 6, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant