Skip to content

send PROXY protocol v2 on configured inbound ports - #4625

Open
daniel-garcia wants to merge 17 commits into
linkerd:mainfrom
daniel-garcia:proxy-protocol-v2-inbound-ports
Open

daniel-garcia wants to merge 17 commits into
linkerd:mainfrom
daniel-garcia:proxy-protocol-v2-inbound-ports

Conversation

@daniel-garcia

@daniel-garcia daniel-garcia commented Sep 18, 2026 •

Copy link
Copy Markdown

Problem

Backend applications behind the inbound proxy see every opaque TCP connection as originating from the loopback, with no way to learn the real client address or the mTLS-verified client identity — the TCP counterpart of what l5d-client-id provides for HTTP. linkerd/linkerd2#15474 asks for opt-in PROXY protocol support on inbound TCP.

Solution

An opt-in port list, LINKERD2_PROXY_INBOUND_PORTS_PROXY_PROTOCOL_V2 (comma-separated ports/ranges, empty by default), selects which inbound ports get a HAProxy PROXY protocol v2 header, prepended on the connection the inbound proxy opens to the local application before splicing bytes:

  • The real client address/port goes in the standard PP v2 address fields. This works for meshed and unmeshed clients; mixed address families are encoded via the IPv6-mapped form.
  • When the connection was mutually authenticated, the verified client identity goes in a custom TLV: type 0xED, subtype 0x01, laid out [type][len][subtype][identity] like the AWS, Azure and Alibaba TLVs. The custom range 0xE0–0xEF has no registry; 0xED avoids the assignments known to be in use (0xE0 Google Cloud, 0xE1 Alibaba Cloud, 0xE4 Teleport, 0xEA AWS, 0xEC SAP Archer, 0xEE Azure, and 0xE0–0xE6 in ByteDance g3), which are documented next to the constant. For reference, Istio's ztunnel sends a peer SPIFFE ID as a flat TLV of type 0xD0, outside the custom range.
  • Headers are generated by the proxy from its own connection metadata and never forwarded from client input, so applications cannot receive forged addresses.

Implementation: a ProxyProtocol connector middleware (modeled on outbound's TaggedTransport) in the shared TCP forward stack, covering both the origin-destination path and the direct transport-header path; TcpEndpoint now carries the client address and server-side TLS status across the forwarding boundary. Opaque/TCP forwarding only — HTTP proxying is unaffected. Config parsing mirrors ENV_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION. No new dependencies. The control-plane surface (annotation → env var) is linkerd/linkerd2#15676.

Per review, PROXY protocol v1 support has been dropped; v2 supersedes it and v1 can be revisited if a use case comes up.

Validation

7 unit tests:

  • byte-exact golden header encoding for IPv4 with identity, IPv4/IPv6 without identity, and both mixed-family directions
  • connect middleware: header written on a configured port, passthrough on an unconfigured one (tokio_test::io)

cargo test -p linkerd-app-inbound green and cargo clippy -p linkerd-app-inbound --all-targets -- -D warnings clean, in ghcr.io/linkerd/dev:v51.

The end-to-end run on kind with a companion linkerd2 build (application received a v2 header with the identity TLV only for meshed clients) was done before the TLV type change and has not been repeated yet.

Part of linkerd/linkerd2#15474

@daniel-garcia daniel-garcia changed the title WIP: feat(inbound): send PROXY protocol v2 on configured inbound ports WIP: feat(inbound): send PROXY protocol v1/v2 on configured inbound ports Sep 29, 2026
@daniel-garcia daniel-garcia changed the title WIP: feat(inbound): send PROXY protocol v1/v2 on configured inbound ports feat(inbound): send PROXY protocol v1/v2 on configured inbound ports Oct 11, 2026
@daniel-garcia daniel-garcia changed the title feat(inbound): send PROXY protocol v1/v2 on configured inbound ports send PROXY protocol v1/v2 on configured inbound ports Oct 11, 2026
Comment thread linkerd/app/inbound/src/lib.rs Resolved
@toidiu

toidiu commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

@daniel-garcia left some comments on the PR.

Linkerd exposes the verified client identity to HTTP applications via
the l5d-client-id header, but has no equivalent for opaque TCP
protocols: the application sees a connection from the loopback with no
caller identity. This adds an opt-in mechanism to relay that connection
metadata on the TCP forwarding path.

When LINKERD2_PROXY_INBOUND_PORTS_PROXY_PROTOCOL_V2 (a comma-separated
port/range list, empty by default) includes the target port, the
inbound proxy prepends a HAProxy PROXY protocol v2 header to the
connection it opens to the local application, before splicing bytes.
The header carries the real client address in the standard PP v2
address fields and, when the connection was mutually authenticated, the
verified client identity in a custom TLV of type 0xE0. Mixed address
families are encoded by promoting the IPv4 address to its IPv6-mapped
form.

The header is written by a new SendProxyProtocol connector middleware
(modeled on the outbound TaggedTransport) in the shared TCP forward
stack, covering both the origin-destination path and the direct
transport-header path. TcpEndpoint now carries the client address and
server-side TLS status across the forwarding boundary to make that
metadata available at connect time. HTTP proxying is unaffected.

The control-plane configuration surface (annotation
config.linkerd.io/proxy-protocol-v2-inbound-ports rendering this
environment variable) lands separately in linkerd/linkerd2.

See linkerd/linkerd2#15637

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: the shorter name reads better at the call site in
server.rs.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: encode() was hard to follow, building a throwaway Vec
for each address in four near-identical match arms.

Introduce AddrBlock, which owns the address-family decision (including
promoting a mixed v4/v6 pair to IPv6-mapped form) and writes the
addresses and ports itself, and write the header through BufMut. The
encoded bytes are unchanged; the existing golden tests still pass.
Also cover the IPv6-client/IPv4-server mixed case, which had no test.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: TLV type 0xE0 is PP2_TYPE_GCP, which Google Cloud
Private Service Connect uses for its connection ID, so a backend behind
both could not tell the two apart.

Send the identity under type 0xED instead, in the subtype layout AWS,
Azure and Alibaba use: [type][len][subtype][value], with subtype 0x01
for the client identity. Later Linkerd fields can then be added as new
subtypes without claiming more of the 16-value custom range.

The custom range has no registry, so document the assignments known to
be in use next to the constant: Google Cloud (0xE0), Alibaba Cloud
(0xE1), Teleport (0xE4), AWS (0xEA), SAP Archer (0xEC) and Azure
(0xEE), plus software that assigns 0xE0-0xE6 to its own fields.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
@daniel-garcia
daniel-garcia force-pushed the proxy-protocol-v2-inbound-ports branch from 1ae8e95 to b3510eb Compare October 26, 2026 06:41
@daniel-garcia daniel-garcia changed the title send PROXY protocol v1/v2 on configured inbound ports send PROXY protocol v2 on configured inbound ports Oct 26, 2026
@daniel-garcia

Copy link
Copy Markdown
Author

Thanks for the review. I pushed fixes for all of it and rebased on main. CI needs a maintainer to approve the workflow run before it starts.

addrs.put(&mut buf);
if let Some(id) = client_id {
buf.put_u8(TLV_TYPE_LINKERD);
buf.put_u16((1 + id.len()) as u16);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why 1 +?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 1 is the subtype byte. The TLV length only counts the value, not the type and length bytes before it, and our value is a 1-byte subtype (TLV_SUBTYPE_CLIENT_ID) followed by the identity. I added a comment there in a20674c that quotes the spec.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replaced these with constants.

Comment on lines +9 to +10
//! This is only ever used on the opaque/TCP forwarding path: HTTP
//! connections proxied by this process never carry this header.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we check that PP ports are also part of the opaque list?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing checks it so frar. I don't think the proxy can check it well. The proxy's own opaque list (LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION) is only a starting default. The real answer comes from the policy controller (the pod's opaque-ports annotation or a Server), and I think it can change at runtime.

It does matter, however. On a PP port that is not opaque, HTTP traffic goes through the HTTP stack and doesn't get a header, and everything else gets one. So maybe it should be enforced in the control plane PR (linkerd/linkerd2#15676). The injector would add the PP ports to the pod's opaque-ports annotation, which the policy controller and destination already use. I'll also document that the Service should be opaque too. Does that sound OK?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#4625 (comment) Lmk what you think.. this way we can enforce it in the proxy itself.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I like this better. Each connection is checked against its live policy, so it also handles policy changes. I'll keep the opaque-ports annotation in linkerd/linkerd2#15676 as the normal way to set it up, but the proxy will enforce it. Details are on the server.rs thread.

@toidiu

toidiu commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Left some comments. The encoding is a bit difficult to follow because its complicated. I am wondering if you could add additional comments that quote the PP spec to make it super obvious where the constant numbers are coming from. Ascii diagrams are also welcome.

daniel-garcia and others added 3 commits November 2, 2026 22:19
…ity length

Review feedback: the header encoding was hard to follow and the client
identity length was not bounded.

Link the spec in the module docs, add a diagram of the header, and quote
the spec next to the constants and length sums so it is clear where each
number comes from.

Cap the client identity at 2048 bytes. DNS names are at most 253 bytes,
but SPIFFE IDs can be longer, so 253 would drop valid identities. An
identity over the cap is left out of the header, not cut short, because a
cut-down identity could match a different real one. This also keeps the
u16 length fields from overflowing.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: keep functions inside impl blocks, so there is a clear
place to attach context to them.

Add a small private Header type with new() and encode(). The identity
length check now lives in Header::new, so a Header can never hold an
identity too long for the u16 length fields. encode() is no longer
pub(crate), since only this module uses it.

Move to_ipv6 into impl AddrBlock, since only AddrBlock::new calls it.

No change in behavior.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: name it encode, to match Header::encode.

Co-authored-by: toidiu <apoorv@toidiu.com>
Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
@daniel-garcia
daniel-garcia force-pushed the proxy-protocol-v2-inbound-ports branch from 1365e90 to ff6445f Compare November 3, 2026 04:46
Review feedback: add a one-line comment to each function, and explain
what call() does.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
@daniel-garcia
daniel-garcia marked this pull request as ready for review November 3, 2026 05:00
@daniel-garcia
daniel-garcia requested a review from a team as a code owner November 3, 2026 05:00
Comment thread linkerd/app/inbound/src/server.rs Outdated
self.map_stack(|config, rt, connect| {
let proxy_protocol_v2_ports = Arc::new(config.proxy_protocol_v2_ports.clone());
connect
.push(proxy_protocol::ProxyProtocol::layer(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only want to support PP on opaque ports right? Lets just enforce that.

We should be able to move this layer into into_tcp_connect, which every inbound stack shares. Push it before push_connect_timeout, so the timeout also bounds the header write. We also get bytes written metrics for free this way:

svc::stack(transport::ConnectTcp::new(*keepalive, *user_timeout))
// Limits the time we wait for a connection to be established.
.push_connect_timeout(*timeout)
// Prevent connections that would target the inbound proxy port from looping.
.push_filter(move |t: T| {
let addr = t.param();
let port = addr.port();
if port == proxy_port {
return Err(Loop(port));
}
Ok(addr)
})

On a PP port, the layer then writes the header for opaque TCP and refuses everything else (HTTP, Detect). Rough sketch:

svc::stack(transport::ConnectTcp::new(*keepalive, *user_timeout))
    .push(proxy_protocol::ProxyProtocol::layer(pp_ports))
    .push_connect_timeout(*timeout)
    ....

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I'll move it into into_tcp_connect, below the connect timeout, and refuse anything that isn't opaque on a PP port.

One correction: the header bytes still won't show up in the write metrics. metrics::Client only starts counting after the connect finishes, and it sits above into_tcp_connect.

A few questions before I start:

  1. Should Protocol::Tls ports count as opaque? They skip HTTP detection like opaque ports do, and PP in front of app TLS is a common setup. I'd allow it.
  2. On the direct path, I'd decide "opaque" from the server policy, not from the client's hint. So if the Server says Detect, the connection is refused. OK?
  3. For HTTP on a PP port, I'd refuse with an io::Error so the client gets a 502. Otherwise it shows up as a 500, and on H2 the reconnect loop keeps retrying until failfast. Or would you rather refuse HTTP earlier, at protocol detection?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d07749a. I went ahead with the answers I suggested above. They're easy to change if you'd rather do it another way.

  • The layer is now in into_tcp_connect, below the connect timeout.
  • Every connect target now provides a proxy_protocol::Peer. Forward and AuthorizedLocalTcp get it from the server policy. opaque and tls get a header, and anything else (Detect) is refused. The HTTP router's target is always refused.
  • A refusal is an io::Error (ConnectionRefused) that wraps a NotOpaqueError, so HTTP clients get a 502. Each refused connection also logs a warning.

}

#[cfg(test)]
mod tests {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unit tests look good. Can you also add integration tests:

  • inbound_tcp (unmeshed client): check that the proxy sends the PP header, and have the server assert it.

  • opaque_hinted (meshed client, direct path): enable PP on the port, and have the server assert the header carries the client identity (0xED TLV).

  • h2_hinted (negative case): enable PP on the port; the client sends an HTTP/2 transport header, so the connection should be refused.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in b8a3f0a, in tests/proxy_protocol.rs. In each test the application is a raw TCP server that parses the header itself.

  • inbound_tcp: an opaque port and an unmeshed client. It checks the addresses, and that there is no identity TLV.
  • inbound_tcp_requires_opaque: the same, but the port's policy is Detect. The client connection is closed, and the application never gets a connection.
  • opaque_hinted: the direct path. It checks that the 0xED TLV carries bar.ns1.serviceaccount.identity.linkerd.cluster.local.
  • h2_hinted: the client gets a 502, and the application never gets a connection.

While doing this I found that the existing direct::opaque_hinted and direct::h2_hinted never reached the inbound proxy. mk_outbound set no outbound policy, so the outbound proxy connected to the app port itself. Fixed in 4070b7e. mk_outbound now sets outbound_default, and each test now checks that the inbound proxy handled the request. h2_hinted checks for l5d-client-id, and opaque_hinted checks the inbound tcp_open_total for the client's identity. I checked that both tests fail again without the policy.


// Not a configured port: connect as normal, with no header.
if !self.ports.contains(&server_addr.port()) {
return Box::pin(self.inner.connect(target).err_into::<Error>());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this boxes the connect future on every connection, even when no PP ports are configured (the default). Could we skip pushing the layer when the port set is empty, or return an Either future here? This matters more once it moves into into_tcp_connect, since every inbound connection would go through it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5f29478. Ports without PP now return the inner future through future::Either, so they aren't boxed. Only PP ports still box. I picked this over skipping the layer because it also covers non-PP ports when some PP ports are set.


let header = Header::new(client_addr, server_addr, client_id.as_deref()).encode();
debug!("writing PROXY protocol v2 header");
io.write_all(&header).await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this write fails we return a bare io::Error, which looks the same as the app refusing the connection. Could we wrap it in a typed error (e.g. "failed to write PROXY protocol header")? It can sit next to the error for refused non-opaque connections.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d5d5523.

// Section 2.2 says a TLV is a 1-byte type, a 2-byte length, then the
// value. That is 3 bytes before the value. Our value is a 1-byte
// subtype followed by the identity.
let tlv_len = self.client_id.map_or(0, |id| 3 + 1 + id.len());

@toidiu toidiu Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets avoid magic numbers and instead use named constants. Same for other magic numbers also please.

/// Type (1 byte) + length (2 bytes).
const TLV_HEADER_LEN: usize = 3;
/// The subtype byte before the client identity.
const TLV_SUBTYPE_LEN: usize = 1;
...

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in dce0603. The numbers are now named constants: PREFIX_LEN, IPV4_ADDR_BLOCK_LEN, IPV6_ADDR_BLOCK_LEN, TLV_HEADER_LEN and TLV_SUBTYPE_LEN. Each one quotes the line of the spec it comes from.

I left the bytes in the golden tests as plain numbers on purpose, so the tests still check the constants.

Review feedback: replace magic numbers with named constants.

Add PREFIX_LEN, IPV4_ADDR_BLOCK_LEN, IPV6_ADDR_BLOCK_LEN, TLV_HEADER_LEN
and TLV_SUBTYPE_LEN. Each one quotes the part of the spec it comes from.
The subtype byte is our own, not the spec's, and the comment says so.

Add a compile-time check that the longest header still fits in the
2-byte len field.

Fix the MAX_CLIENT_ID_LEN comment: the SPIFFE-ID spec recommends 2048
bytes but does not cap it. Also correct a few details in the list of
known custom TLV types.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
…fails

Review feedback: a failed header write returned a bare io::Error, which
looks the same as the application refusing the connection.

Wrap it in WriteHeaderError ("failed to write PROXY protocol header"),
keeping the io::Error as its source, and add a test for it.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
…otocol

Review feedback: every connection was boxed, even when no PROXY protocol
ports are configured (the default).

Return a future::Either instead. Ports without PROXY protocol use the
inner connect future as is. Only connections that write a header are
boxed.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: move the PROXY protocol layer into `into_tcp_connect`,
below the connect timeout, so every inbound connection goes through it
and the timeout also bounds the header write.

On a PROXY protocol port, a connection now gets a header only when the
server policy for the port is `opaque` or `tls`. Everything else is
refused: TCP that was forwarded because detection did not find HTTP,
and HTTP, where one connection to the application can carry requests
from many clients. The refusal is a `ConnectionRefused` `io::Error`, so
HTTP clients get a 502, and each one is logged as a warning.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: cover PROXY protocol end to end.

- inbound_tcp: an unmeshed client on an opaque port gets a header with
  its address and no identity.
- inbound_tcp_requires_opaque: on a port whose policy is not opaque,
  the connection is refused and the application never sees it.
- opaque_hinted: a meshed client on the direct path gets a header with
  its identity in the 0xED TLV.
- h2_hinted: HTTP on the direct path gets a 502, and the application
  never sees a connection.

The direct-path tests give the outbound proxy an outbound policy.
Without one, it connects to the application port itself and the
inbound proxy never sees the connection.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
`direct::opaque_hinted` and `direct::h2_hinted` set no outbound policy,
so the outbound proxy connected to the application port itself and the
inbound proxy never saw the traffic. Both passed without testing the
direct path.

`mk_outbound` now sets `outbound_default` for the destination. Each test
also checks that the inbound proxy handled the request: `h2_hinted`
checks that the application sees `l5d-client-id`, and `opaque_hinted`
checks the inbound `tcp_open_total` for the client's identity. Both
checks fail without the outbound policy.

The PROXY protocol tests now reuse `mk_outbound`.

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
daniel-garcia added a commit to daniel-garcia/linkerd2 that referenced this pull request Nov 11, 2026
The proxy (linkerd/linkerd2-proxy#4625) now writes a PROXY protocol
header only on connections whose server policy is opaque or TLS, and
refuses every other connection to those ports, including HTTP.

Say so in the Helm value and annotation docs: each port must also be in
the `config.linkerd.io/opaque-ports` annotation, or be selected by a
Server with `proxyProtocol: opaque` (or `TLS`).

Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants