Skip to content

Add protobuf ingestion adapter and indexing parity - #1450

Draft
jwils wants to merge 1 commit into
joshuaw/proto-record-conversionfrom
joshuaw/proto-ingestion-adapter
Draft

jwils wants to merge 1 commit into
joshuaw/proto-record-conversionfrom
joshuaw/proto-ingestion-adapter

Conversation

@jwils

@jwils jwils commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The indexer has a format-neutral IngestionAdapter seam and shared for_each_ingestion_format test suite (#1412), but the :proto format is still pending because elasticgraph-proto_ingestion does not yet validate native protobuf records or register a "proto" ingestion adapter.

What

Add ProtoIngestion::RecordValidator and ProtoIngestion::IngestionAdapter, wire ProtoIngestion::IndexerExtension to register "proto", and enable :proto across the shared indexer test suite.

How

ProtoIngestion::RecordValidator validates ElasticGraph constraints that protobuf wire decoding cannot express: unknown enum integer values, JsonSafeLong range, finite Float values, calendar Date strings, DateTime Timestamp bounds, LocalTime, TimeZone, and Untyped JSON payloads. Retained wire-contract checks (JsonSafeLong and enum membership) stay unsampled when skip_record_validation: true.

ProtoIngestion::IngestionAdapter validates incoming Event[Google::Protobuf::Message] records, unwraps top-level union/interface messages to their concrete subtype, verifies required rollover timestamps, and pairs events with ProtoIngestion::RecordPreparer without JSON schema version checks.

Risk

Activates "proto" ingestion when protobuf runtime metadata is present in the schema artifacts. Existing JSON ingestion paths are unchanged.

Testing

No manual testing.

Bigger picture

Completes the 3-PR protobuf indexing parity stack (#1441, #1442, #1450), running the shared elasticgraph-indexer unit, integration, and acceptance specs against both :json and :proto.

@jwils
jwils force-pushed the joshuaw/proto-ingestion-adapter branch from 1cc170f to 9ec7bb3 Compare November 4, 2026 23:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant