feat(outputs.kafka): Handle partial writes without retrying delivered metrics by kVinsom · Pull Request #19964 · influxdata/telegraf

kVinsom · 2026-10-22T16:08:11Z

Summary

When only part of a Kafka batch fails, retrying the entire batch duplicates messages that were already delivered.
Conversely, treating one permanent error as success for the entire batch can discard unrelated failed messages.

Return internal.PartialWriteError with the original metric indices: acknowledge successful deliveries, reject individual permanent failures, and retain transient failures for retry.
The indices remain correct when serialization skips metrics or Sarama reports errors out of order.
An error without per-message delivery results retains every submitted metric because its delivery status is unknown.

Permanent failures include serialization errors, broker size/timestamp errors, and per-message sarama.ConfigurationError values, including wrapped errors.
Sarama uses the latter for messages exceeding the local max_message_bytes limit and headers unsupported by the configured Kafka version.
These metrics are rejected individually instead of being retried on every flush.

Following review, this PR contains no new configuration option: topic_error_behavior, drop-mode tests, and the associated sample configuration/documentation have been removed for a separate feature change.
Unknown-topic-or-partition errors retain the existing retry behavior because they can be temporary.
A batch filled entirely with unavailable-topic messages can still block newer metrics until the topic or partition recovers; this PR does not introduce an opt-in data-dropping policy.

Regression coverage includes mixed permanent/transient errors, wrapped errors, original indices across serialization gaps, and unclassified batch errors.
Tests using a real Sarama producer against MockBroker exercise Telegraf's buffer and delivery tracking, including oversized messages in mixed/full batches, unsupported headers, later-batch progress, and recovery of a temporarily missing topic.

No dependency, delivery-timeout, or build-configuration changes are included.
#19735 addresses stalled deliveries separately; both changes touch Write() and may need reconciliation depending on merge order.

Validation

All three new real-producer configuration-error scenarios failed against the previous PR code and pass after the fix.
Windows/Go 1.27.0: go test -short ./plugins/outputs/kafka ./plugins/common/kafka ./models -count=1 -timeout=180s passes.
Linux/WSL/Go 1.27.0: go test -race -short ./plugins/outputs/kafka ./plugins/common/kafka ./models -count=1 -timeout=180s passes. The mock broker starts only after plugin initialization to avoid racing Sarama's global logger setup.
go vet ./plugins/outputs/kafka ./plugins/common/kafka ./models passes.
Project-pinned golangci-lint v2.15.0 on ./plugins/outputs/kafka/...: 0 issues.
gofmt, Markdown lint, the project's README linter, and git diff --check pass.
The Kafka README was regenerated using the project's config includer.
The Docker-backed Kafka integration test was not run because the Docker daemon is unavailable.
The real-producer MockBroker tests ran without Docker.
The repository-wide test suite was not run locally; fresh CI is required for this revision.

Checklist

No AI generated code was used in this PR
AI generated code used in this PR follows the InfluxData Policy on AI-Generated Code Contributions

AI-assisted implementation and validation using OpenAI Codex.

Related issues

Part of #10864.

Review follow-up (4 November 2026)

Merged the configuration-error regression tests into kafka_test.go and combined the plugin configuration into one struct initializer, as requested. The logger is still initialized before starting the mock broker.

Merged current master (f6d5b919a). The Kafka output, common Kafka code and models pass Linux race tests; the project-pinned Kafka linter reports zero issues. The Docker-backed integration test remains unrun locally.

…metrics Track per-message delivery results using PartialWriteError. Keep transient failures for retry and add an opt-in drop policy for unknown topics or partitions. Cover mixed failures, serialization gaps, buffer draining and topic recovery. Co-Authored-By: OpenAI Codex (GPT-6) <noreply@openai.com>

telegraf-tiger · 2026-10-22T16:08:20Z

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

kVinsom · 2026-10-22T16:14:55Z

!signed-cla

skartikey

@kVinsom Thanks, the client-side size and header case is covered now.
I checked sarama v1.61.1 only returns a per-message ConfigurationError for those two, so rejecting it is safe, and the MockBroker tests show the buffer draining past a full batch of them.

srebhan

Thanks for your contribution @kVinsom! Two small comments from my side. I also changed this PR to "feature" as this is how we treated it on other plugins (and because it's a feature actually). ;-)

srebhan · 2026-11-02T09:56:25Z

Please merge this file into kafka_test.go!

srebhan · 2026-11-02T09:56:51Z

+			plugin := &Kafka{
+				Topic:        "existing",
+				Headers:      tt.headers,
+				Log:          testutil.Logger{},
+				producerFunc: sarama.NewSyncProducer,
+				serializer:   serializer,
+			}
+			plugin.Version = tt.version
+			plugin.MaxMessageBytes = 256
+			plugin.RequiredAcks = 1


Please merge the two blocks!

telegraf-tiger · 2026-11-04T18:04:39Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 6.82 % for linux amd64 (new size: 329.3 MB, nightly size 308.2 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

. DEB	. RPM	. TAR . GZ	. ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

telegraf-tiger Bot added area/kafka fix pr to fix corresponding bug plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins labels Oct 22, 2026

skartikey reviewed Oct 29, 2026

View reviewed changes

Comment thread plugins/outputs/kafka/kafka.go Resolved

Comment thread plugins/outputs/kafka/kafka.go Resolved

skartikey self-assigned this Oct 29, 2026

kVinsom added 2 commits October 31, 2026 02:57

fix(outputs.kafka): reject permanent client-side message errors

bddcc8c

test(outputs.kafka): initialize logger before mock broker

fe1dd46

skartikey approved these changes Nov 2, 2026

View reviewed changes

skartikey assigned srebhan and unassigned skartikey Nov 2, 2026

skartikey added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Nov 2, 2026

srebhan reviewed Nov 2, 2026

View reviewed changes

srebhan changed the title ~~fix(outputs.kafka): handle partial writes without retrying delivered metrics~~ feat(outputs.kafka): Handle partial writes without retrying delivered metrics Nov 2, 2026

telegraf-tiger Bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Nov 2, 2026

srebhan added waiting for response waiting for response from contributor and removed fix pr to fix corresponding bug labels Nov 2, 2026

kVinsom added 2 commits November 4, 2026 19:35

test(outputs.kafka): consolidate configuration error coverage

48f390f

Merge master after Kafka test review

067b60c

Navigation Menu

Repository navigation

feat(outputs.kafka): Handle partial writes without retrying delivered metrics - #19964

feat(outputs.kafka): Handle partial writes without retrying delivered metrics#19964
kVinsom wants to merge 5 commits into
influxdata:masterfrom
kVinsom:fix/10864-kafka-partial-writes

kVinsom commented Oct 22, 2026 •

edited

Loading

Uh oh!

telegraf-tiger Bot commented Oct 22, 2026

Uh oh!

kVinsom commented Oct 22, 2026

Uh oh!

Uh oh!

Uh oh!

skartikey left a comment

Uh oh!

srebhan left a comment

Uh oh!

srebhan Nov 2, 2026

Uh oh!

srebhan Nov 2, 2026

Uh oh!

telegraf-tiger Bot commented Nov 4, 2026

Artifact URLs

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kVinsom commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Checklist

Related issues

Review follow-up (4 November 2026)

Uh oh!

telegraf-tiger Bot commented Oct 22, 2026

Uh oh!

kVinsom commented Oct 22, 2026

Uh oh!

Uh oh!

Uh oh!

skartikey left a comment

Choose a reason for hiding this comment

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

srebhan Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

srebhan Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

telegraf-tiger Bot commented Nov 4, 2026

Artifact URLs

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kVinsom commented Oct 22, 2026 •

edited

Loading