Skip to content

align failed handshake behavior - #401

Open
kares wants to merge 3 commits into
masterfrom
ssl-align-alert
Open

kares wants to merge 3 commits into
masterfrom
ssl-align-alert

Conversation

@kares

@kares kares commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

expected to address #399

primary fix is the same as in #400 but we have some extra work to e.g. align the reported verify-result (with MRI)

kares added 3 commits November 3, 2026 19:27
When BCJSSE's unwrap() fails during handshake (e.g. the trust manager
rejects the server cert), the fatal alert is queued for output. unwrap()
does not throw: saves the exception (ProvSSLEngine.deferredException)
and asks for NEED_WRAP.

The next wrap() throws the saved exception and writes nothing.

doShutdown() stopped at that first exception, so no alert was sent;
a server whose peer kept the TCP socket open then blocked in accept
indefinitely.

regression since 0.19.2 - reduced shutdown to one doShutdown() call
rejected peer certificate raised a bare "certificate verify failed" on
SunJSSE and a synthetic "Socket closed" on BCJSSE - in both cases
`SSLSocket#verify_result` stayed 0;
while MRI reports the verify error in the message and in verify_result

trust manager now throws VerifyCertificateException; which carries the
verify error, connect/accept find it in the cause chain of the JSSE
exception and set the message and verify_result.

BCJSSE holds trust failure back until the next wrap(); when unwrap hits
end of stream while the engine still asks for NEED_WRAP, doHandshake now
runs that wrap, so real failure is raised instead of "Socket closed".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server accept hangs when the client rejects the certificate (since 0.19.2)

1 participant