Repository navigation
send PROXY protocol v2 on configured inbound ports - #4625
daniel-garcia wants to merge 17 commits into
Conversation
|
@daniel-garcia left some comments on the PR. |
Linkerd exposes the verified client identity to HTTP applications via the l5d-client-id header, but has no equivalent for opaque TCP protocols: the application sees a connection from the loopback with no caller identity. This adds an opt-in mechanism to relay that connection metadata on the TCP forwarding path. When LINKERD2_PROXY_INBOUND_PORTS_PROXY_PROTOCOL_V2 (a comma-separated port/range list, empty by default) includes the target port, the inbound proxy prepends a HAProxy PROXY protocol v2 header to the connection it opens to the local application, before splicing bytes. The header carries the real client address in the standard PP v2 address fields and, when the connection was mutually authenticated, the verified client identity in a custom TLV of type 0xE0. Mixed address families are encoded by promoting the IPv4 address to its IPv6-mapped form. The header is written by a new SendProxyProtocol connector middleware (modeled on the outbound TaggedTransport) in the shared TCP forward stack, covering both the origin-destination path and the direct transport-header path. TcpEndpoint now carries the client address and server-side TLS status across the forwarding boundary to make that metadata available at connect time. HTTP proxying is unaffected. The control-plane configuration surface (annotation config.linkerd.io/proxy-protocol-v2-inbound-ports rendering this environment variable) lands separately in linkerd/linkerd2. See linkerd/linkerd2#15637 Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: the shorter name reads better at the call site in server.rs. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: encode() was hard to follow, building a throwaway Vec for each address in four near-identical match arms. Introduce AddrBlock, which owns the address-family decision (including promoting a mixed v4/v6 pair to IPv6-mapped form) and writes the addresses and ports itself, and write the header through BufMut. The encoded bytes are unchanged; the existing golden tests still pass. Also cover the IPv6-client/IPv4-server mixed case, which had no test. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: TLV type 0xE0 is PP2_TYPE_GCP, which Google Cloud Private Service Connect uses for its connection ID, so a backend behind both could not tell the two apart. Send the identity under type 0xED instead, in the subtype layout AWS, Azure and Alibaba use: [type][len][subtype][value], with subtype 0x01 for the client identity. Later Linkerd fields can then be added as new subtypes without claiming more of the 16-value custom range. The custom range has no registry, so document the assignments known to be in use next to the constant: Google Cloud (0xE0), Alibaba Cloud (0xE1), Teleport (0xE4), AWS (0xEA), SAP Archer (0xEC) and Azure (0xEE), plus software that assigns 0xE0-0xE6 to its own fields. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
1ae8e95 to
b3510eb
Compare
|
Thanks for the review. I pushed fixes for all of it and rebased on main. CI needs a maintainer to approve the workflow run before it starts. |
| addrs.put(&mut buf); | ||
| if let Some(id) = client_id { | ||
| buf.put_u8(TLV_TYPE_LINKERD); | ||
| buf.put_u16((1 + id.len()) as u16); |
There was a problem hiding this comment.
The 1 is the subtype byte. The TLV length only counts the value, not the type and length bytes before it, and our value is a 1-byte subtype (TLV_SUBTYPE_CLIENT_ID) followed by the identity. I added a comment there in a20674c that quotes the spec.
There was a problem hiding this comment.
I replaced these with constants.
| //! This is only ever used on the opaque/TCP forwarding path: HTTP | ||
| //! connections proxied by this process never carry this header. |
There was a problem hiding this comment.
Do we check that PP ports are also part of the opaque list?
There was a problem hiding this comment.
Nothing checks it so frar. I don't think the proxy can check it well. The proxy's own opaque list (LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION) is only a starting default. The real answer comes from the policy controller (the pod's opaque-ports annotation or a Server), and I think it can change at runtime.
It does matter, however. On a PP port that is not opaque, HTTP traffic goes through the HTTP stack and doesn't get a header, and everything else gets one. So maybe it should be enforced in the control plane PR (linkerd/linkerd2#15676). The injector would add the PP ports to the pod's opaque-ports annotation, which the policy controller and destination already use. I'll also document that the Service should be opaque too. Does that sound OK?
There was a problem hiding this comment.
#4625 (comment) Lmk what you think.. this way we can enforce it in the proxy itself.
There was a problem hiding this comment.
Yes, I like this better. Each connection is checked against its live policy, so it also handles policy changes. I'll keep the opaque-ports annotation in linkerd/linkerd2#15676 as the normal way to set it up, but the proxy will enforce it. Details are on the server.rs thread.
|
Left some comments. The encoding is a bit difficult to follow because its complicated. I am wondering if you could add additional comments that quote the PP spec to make it super obvious where the constant numbers are coming from. Ascii diagrams are also welcome. |
…ity length Review feedback: the header encoding was hard to follow and the client identity length was not bounded. Link the spec in the module docs, add a diagram of the header, and quote the spec next to the constants and length sums so it is clear where each number comes from. Cap the client identity at 2048 bytes. DNS names are at most 253 bytes, but SPIFFE IDs can be longer, so 253 would drop valid identities. An identity over the cap is left out of the header, not cut short, because a cut-down identity could match a different real one. This also keeps the u16 length fields from overflowing. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: keep functions inside impl blocks, so there is a clear place to attach context to them. Add a small private Header type with new() and encode(). The identity length check now lives in Header::new, so a Header can never hold an identity too long for the u16 length fields. encode() is no longer pub(crate), since only this module uses it. Move to_ipv6 into impl AddrBlock, since only AddrBlock::new calls it. No change in behavior. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: name it encode, to match Header::encode. Co-authored-by: toidiu <apoorv@toidiu.com> Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
1365e90 to
ff6445f
Compare
Review feedback: add a one-line comment to each function, and explain what call() does. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
…nbound-ports Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
| self.map_stack(|config, rt, connect| { | ||
| let proxy_protocol_v2_ports = Arc::new(config.proxy_protocol_v2_ports.clone()); | ||
| connect | ||
| .push(proxy_protocol::ProxyProtocol::layer( |
There was a problem hiding this comment.
We only want to support PP on opaque ports right? Lets just enforce that.
We should be able to move this layer into into_tcp_connect, which every inbound stack shares. Push it before push_connect_timeout, so the timeout also bounds the header write. We also get bytes written metrics for free this way:
linkerd2-proxy/linkerd/app/inbound/src/server.rs
Lines 142 to 153 in 31a2326
On a PP port, the layer then writes the header for opaque TCP and refuses everything else (HTTP, Detect). Rough sketch:
svc::stack(transport::ConnectTcp::new(*keepalive, *user_timeout))
.push(proxy_protocol::ProxyProtocol::layer(pp_ports))
.push_connect_timeout(*timeout)
....There was a problem hiding this comment.
Sounds good. I'll move it into into_tcp_connect, below the connect timeout, and refuse anything that isn't opaque on a PP port.
One correction: the header bytes still won't show up in the write metrics. metrics::Client only starts counting after the connect finishes, and it sits above into_tcp_connect.
A few questions before I start:
- Should
Protocol::Tlsports count as opaque? They skip HTTP detection like opaque ports do, and PP in front of app TLS is a common setup. I'd allow it. - On the direct path, I'd decide "opaque" from the server policy, not from the client's hint. So if the Server says Detect, the connection is refused. OK?
- For HTTP on a PP port, I'd refuse with an
io::Errorso the client gets a 502. Otherwise it shows up as a 500, and on H2 the reconnect loop keeps retrying until failfast. Or would you rather refuse HTTP earlier, at protocol detection?
There was a problem hiding this comment.
Done in d07749a. I went ahead with the answers I suggested above. They're easy to change if you'd rather do it another way.
- The layer is now in
into_tcp_connect, below the connect timeout. - Every connect target now provides a
proxy_protocol::Peer.ForwardandAuthorizedLocalTcpget it from the server policy.opaqueandtlsget a header, and anything else (Detect) is refused. The HTTP router's target is always refused. - A refusal is an
io::Error(ConnectionRefused) that wraps aNotOpaqueError, so HTTP clients get a 502. Each refused connection also logs a warning.
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { |
There was a problem hiding this comment.
The unit tests look good. Can you also add integration tests:
inbound_tcp(unmeshed client): check that the proxy sends the PP header, and have the server assert it.
opaque_hinted(meshed client, direct path): enable PP on the port, and have the server assert the header carries the client identity (0xEDTLV).
h2_hinted(negative case): enable PP on the port; the client sends an HTTP/2 transport header, so the connection should be refused.
There was a problem hiding this comment.
Added in b8a3f0a, in tests/proxy_protocol.rs. In each test the application is a raw TCP server that parses the header itself.
inbound_tcp: an opaque port and an unmeshed client. It checks the addresses, and that there is no identity TLV.inbound_tcp_requires_opaque: the same, but the port's policy is Detect. The client connection is closed, and the application never gets a connection.opaque_hinted: the direct path. It checks that the0xEDTLV carriesbar.ns1.serviceaccount.identity.linkerd.cluster.local.h2_hinted: the client gets a 502, and the application never gets a connection.
While doing this I found that the existing direct::opaque_hinted and direct::h2_hinted never reached the inbound proxy. mk_outbound set no outbound policy, so the outbound proxy connected to the app port itself. Fixed in 4070b7e. mk_outbound now sets outbound_default, and each test now checks that the inbound proxy handled the request. h2_hinted checks for l5d-client-id, and opaque_hinted checks the inbound tcp_open_total for the client's identity. I checked that both tests fail again without the policy.
|
|
||
| // Not a configured port: connect as normal, with no header. | ||
| if !self.ports.contains(&server_addr.port()) { | ||
| return Box::pin(self.inner.connect(target).err_into::<Error>()); |
There was a problem hiding this comment.
nit: this boxes the connect future on every connection, even when no PP ports are configured (the default). Could we skip pushing the layer when the port set is empty, or return an Either future here? This matters more once it moves into into_tcp_connect, since every inbound connection would go through it.
There was a problem hiding this comment.
Done in 5f29478. Ports without PP now return the inner future through future::Either, so they aren't boxed. Only PP ports still box. I picked this over skipping the layer because it also covers non-PP ports when some PP ports are set.
|
|
||
| let header = Header::new(client_addr, server_addr, client_id.as_deref()).encode(); | ||
| debug!("writing PROXY protocol v2 header"); | ||
| io.write_all(&header).await?; |
There was a problem hiding this comment.
If this write fails we return a bare io::Error, which looks the same as the app refusing the connection. Could we wrap it in a typed error (e.g. "failed to write PROXY protocol header")? It can sit next to the error for refused non-opaque connections.
| // Section 2.2 says a TLV is a 1-byte type, a 2-byte length, then the | ||
| // value. That is 3 bytes before the value. Our value is a 1-byte | ||
| // subtype followed by the identity. | ||
| let tlv_len = self.client_id.map_or(0, |id| 3 + 1 + id.len()); |
There was a problem hiding this comment.
Lets avoid magic numbers and instead use named constants. Same for other magic numbers also please.
/// Type (1 byte) + length (2 bytes).
const TLV_HEADER_LEN: usize = 3;
/// The subtype byte before the client identity.
const TLV_SUBTYPE_LEN: usize = 1;
...There was a problem hiding this comment.
Done in dce0603. The numbers are now named constants: PREFIX_LEN, IPV4_ADDR_BLOCK_LEN, IPV6_ADDR_BLOCK_LEN, TLV_HEADER_LEN and TLV_SUBTYPE_LEN. Each one quotes the line of the spec it comes from.
I left the bytes in the golden tests as plain numbers on purpose, so the tests still check the constants.
Review feedback: replace magic numbers with named constants. Add PREFIX_LEN, IPV4_ADDR_BLOCK_LEN, IPV6_ADDR_BLOCK_LEN, TLV_HEADER_LEN and TLV_SUBTYPE_LEN. Each one quotes the part of the spec it comes from. The subtype byte is our own, not the spec's, and the comment says so. Add a compile-time check that the longest header still fits in the 2-byte len field. Fix the MAX_CLIENT_ID_LEN comment: the SPIFFE-ID spec recommends 2048 bytes but does not cap it. Also correct a few details in the list of known custom TLV types. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
…fails
Review feedback: a failed header write returned a bare io::Error, which
looks the same as the application refusing the connection.
Wrap it in WriteHeaderError ("failed to write PROXY protocol header"),
keeping the io::Error as its source, and add a test for it.
Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
…otocol Review feedback: every connection was boxed, even when no PROXY protocol ports are configured (the default). Return a future::Either instead. Ports without PROXY protocol use the inner connect future as is. Only connections that write a header are boxed. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: move the PROXY protocol layer into `into_tcp_connect`, below the connect timeout, so every inbound connection goes through it and the timeout also bounds the header write. On a PROXY protocol port, a connection now gets a header only when the server policy for the port is `opaque` or `tls`. Everything else is refused: TCP that was forwarded because detection did not find HTTP, and HTTP, where one connection to the application can carry requests from many clients. The refusal is a `ConnectionRefused` `io::Error`, so HTTP clients get a 502, and each one is logged as a warning. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Review feedback: cover PROXY protocol end to end. - inbound_tcp: an unmeshed client on an opaque port gets a header with its address and no identity. - inbound_tcp_requires_opaque: on a port whose policy is not opaque, the connection is refused and the application never sees it. - opaque_hinted: a meshed client on the direct path gets a header with its identity in the 0xED TLV. - h2_hinted: HTTP on the direct path gets a 502, and the application never sees a connection. The direct-path tests give the outbound proxy an outbound policy. Without one, it connects to the application port itself and the inbound proxy never sees the connection. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
`direct::opaque_hinted` and `direct::h2_hinted` set no outbound policy, so the outbound proxy connected to the application port itself and the inbound proxy never saw the traffic. Both passed without testing the direct path. `mk_outbound` now sets `outbound_default` for the destination. Each test also checks that the inbound proxy handled the request: `h2_hinted` checks that the application sees `l5d-client-id`, and `opaque_hinted` checks the inbound `tcp_open_total` for the client's identity. Both checks fail without the outbound policy. The PROXY protocol tests now reuse `mk_outbound`. Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
The proxy (linkerd/linkerd2-proxy#4625) now writes a PROXY protocol header only on connections whose server policy is opaque or TLS, and refuses every other connection to those ports, including HTTP. Say so in the Helm value and annotation docs: each port must also be in the `config.linkerd.io/opaque-ports` annotation, or be selected by a Server with `proxyProtocol: opaque` (or `TLS`). Signed-off-by: Daniel Garcia <dgarcia@infoblox.com>
Problem
Backend applications behind the inbound proxy see every opaque TCP connection as originating from the loopback, with no way to learn the real client address or the mTLS-verified client identity — the TCP counterpart of what
l5d-client-idprovides for HTTP. linkerd/linkerd2#15474 asks for opt-in PROXY protocol support on inbound TCP.Solution
An opt-in port list,
LINKERD2_PROXY_INBOUND_PORTS_PROXY_PROTOCOL_V2(comma-separated ports/ranges, empty by default), selects which inbound ports get a HAProxy PROXY protocol v2 header, prepended on the connection the inbound proxy opens to the local application before splicing bytes:0xED, subtype0x01, laid out[type][len][subtype][identity]like the AWS, Azure and Alibaba TLVs. The custom range0xE0–0xEFhas no registry;0xEDavoids the assignments known to be in use (0xE0Google Cloud,0xE1Alibaba Cloud,0xE4Teleport,0xEAAWS,0xECSAP Archer,0xEEAzure, and0xE0–0xE6in ByteDance g3), which are documented next to the constant. For reference, Istio's ztunnel sends a peer SPIFFE ID as a flat TLV of type0xD0, outside the custom range.Implementation: a
ProxyProtocolconnector middleware (modeled on outbound'sTaggedTransport) in the shared TCP forward stack, covering both the origin-destination path and the direct transport-header path;TcpEndpointnow carries the client address and server-side TLS status across the forwarding boundary. Opaque/TCP forwarding only — HTTP proxying is unaffected. Config parsing mirrorsENV_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION. No new dependencies. The control-plane surface (annotation → env var) is linkerd/linkerd2#15676.Per review, PROXY protocol v1 support has been dropped; v2 supersedes it and v1 can be revisited if a use case comes up.
Validation
7 unit tests:
tokio_test::io)cargo test -p linkerd-app-inboundgreen andcargo clippy -p linkerd-app-inbound --all-targets -- -D warningsclean, inghcr.io/linkerd/dev:v51.The end-to-end run on kind with a companion linkerd2 build (application received a v2 header with the identity TLV only for meshed clients) was done before the TLV type change and has not been repeated yet.
Part of linkerd/linkerd2#15474