Repository navigation
Conversation
jwils
changed the title
jwils
force-pushed
the
joshuaw/proto-record-conversion
branch
from
November 2, 2026 00:07
7ec74bf to
7fb2796
Compare
This was referenced Nov 2, 2026
jwils
force-pushed
the
joshuaw/proto-record-conversion
branch
2 times, most recently
from
November 5, 2026 20:08
2af56fe to
11abf6f
Compare
jwils
force-pushed
the
joshuaw/proto-record-conversion
branch
from
November 9, 2026 15:58
11abf6f to
bda3128
Compare
Keep scalar metadata when built-in names are customized. Accept numeric enum values from JRuby and avoid native message singleton mocks. Include a reproducible preparation benchmark.
jwils
added a commit
that referenced
this pull request
Nov 9, 2026
## Why The shared indexer specs need real protobuf publishers before they can exercise protobuf ingestion. Their protobuf round trip is still a placeholder. ## What Add a test publisher that compiles `schema.proto`, encodes event batches, and decodes native protobuf records. Extract the shared `protoc` locator from the existing schema test helper. ## How The publisher walks protobuf descriptors and uses the generated runtime metadata for scalar and enum overrides. Each instance owns its descriptor pool, so tests can encode with an older publisher and decode with the current schema. Focused specs cover proto2 and proto3, field presence, nested lists and abstract types, exact 64-bit integers, timestamps, and disk-backed artifacts. The publisher stays independent of the production record preparer. ## Risk Adds test support and declares runtime dependencies on `elasticgraph-indexer` and `google-protobuf`. Protobuf ingestion remains inactive; existing indexing examples stay pending. ## Testing No manual testing. ## Bigger picture This prepares the shared indexer parity suite for real protobuf publishing. Record preparation (#1442) and adapter integration plus parity-test activation (#1450) follow in this stack.
jwils
force-pushed
the
joshuaw/proto-record-conversion
branch
from
November 9, 2026 16:29
9d326d9 to
bcddf29
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The indexer needs datastore hashes, but protobuf publishers produce native messages. Converting them into public record hashes first adds a full traversal and extra allocations.
What
Prepare native protobuf messages directly into datastore records in one pass using the generated protobuf runtime metadata and
google-protobufdescriptors.How
ProtoIngestion::RecordPreparercaches field plans built from protobuf descriptors. Sparse runtime metadata supplies field-name overrides, enum names, and scalar types. Renamed built-in scalars retain their indexing preparers, and enum symbols and numbers map consistently across Ruby runtimes.The preparer applies shared scalar rules, unwraps abstract
oneofrecords, formatsgoogle.protobuf.Timestampvalues directly fromsecondsandnanos, and parses and canonicalizesUntypedleaves. The shared indexing pipeline adds__countsafterward.Preserves field presence, explicit defaults, exact 64-bit integers, nested lists, and concrete types in a single pass.
Risk
Adds
ProtoIngestion::RecordPreparerandRuntimeMetadata::Schema.from_schema_artifacts, which become active when #1450 registers the protobuf ingestion adapter. Runtime dependencies areelasticgraph-indexerandgoogle-protobuf.Testing
No manual testing. Covered by integration specs across
proto2andproto3inrecord_preparer_spec.rb.Bigger picture
Builds on #1441's protobuf publishing test support. Native record validation, adapter registration, and activation of the shared indexing parity suite follow in #1450.