gh-148292: Update SSLSocket.read() for OpenSSL 4

[vstinner]

Add _got_eof attribute to avoid calling SSL_read_ex() again after SSL_ERROR_EOF.

• Issue: 3.15.0a8 fails test_urllib2_localnet with OpenSSL 4.0.0beta1 #148292

[vstinner]

@picnixz@gpshead: Would you mind to review this (draft) change?

I marked the PR as a draft since I'm not sure if the fix makes sense and is correct.

See #148600 (comment) to reproduce the issue and build Python 3.15 with OpenSSL 4.0.0.

Note: #148601 (Add Modules/_ssl_data_40.h data) has no effect on test_urllib2_localnet (it does still fail).

[vstinner]

See #146217 (comment) for differences between OpenSSL 3 and OpenSSL 4.

In short:

• On OpenSSL 3, the 3rd read fails with SSLEOFError("UNEXPECTED_EOF_WHILE_READING"), and the following 4th read fails with SSLEOFError("EOF occurred in violation of protocol").
• On OpenSSL 4, the 3rd read fails with SSLEOFError("UNEXPECTED_EOF_WHILE_READING"), and the following 4th read fails with SSLError("A failure in the SSL library occurred").

[picnixz]

I don't think it's the correct change because I need to investigate. The reason why I don't think it's correct is because the code path being taken to trigger "A failure in the SSL library occurred" means that the last OpenSSL error code was not set (either we cleared it accidently or they didn't set it correctly) and this is something that can happen elsewhere.

[picnixz]

FTR, this may be related #148594.

[vstinner]

FTR, this may be related #148594.

It's not related. test_urllib2_localnet still fails with this change:

diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index 4e563379098..d17cd308628 100644
--- a/Modules/_ssl.c+++ b/Modules/_ssl.c@@ -2938,6 +2938,7 @@ _ssl__SSLSocket_read_impl(PySSLSocket *self, Py_ssize_t len,
do {
Py_BEGIN_ALLOW_THREADS;
+ ERR_clear_error();
retval = SSL_read_ex(self->ssl, mem, (size_t)len, &count);
err = _PySSL_errno(retval == 0, self->ssl, retval);
Py_END_ALLOW_THREADS;

[vstinner]

On OpenSSL 4, the 3rd read fails with SSLEOFError("UNEXPECTED_EOF_WHILE_READING"), and the following 4th read fails with SSLError("A failure in the SSL library occurred").

The 3rd read fails with SSL_R_UNEXPECTED_EOF_WHILE_READING:

• SSL_read_ex(): 0
• errno: 0
• SSL_get_error(ssl, retcode): SSL_ERROR_SSL (1)
• ERR_peek_last_error(): 167772454
• ERR_GET_LIB(e): ERR_LIB_SSL (20)
• ERR_GET_REASON(e): SSL_R_UNEXPECTED_EOF_WHILE_READING (294)

Related code in Modules/_ssl.c:

#if defined(SSL_R_UNEXPECTED_EOF_WHILE_READING)
/* OpenSSL 3.0 changed transport EOF from SSL_ERROR_SYSCALL with * zero return value to SSL_ERROR_SSL with a special error code. */if (ERR_GET_LIB(e) ==ERR_LIB_SSL&&ERR_GET_REASON(e) ==SSL_R_UNEXPECTED_EOF_WHILE_READING) {
p=PY_SSL_ERROR_EOF;
type=state->PySSLEOFErrorObject;
errstr="EOF occurred in violation of protocol";
}
#endif

The 4th read doesn't seem to set any specific error :-( It only says that it's an "SSL error".

• SSL_read_ex(): 0
• errno: 0
• SSL_get_error(ssl, retcode): SSL_ERROR_SSL (1)
• ERR_peek_last_error(): 0
• ERR_GET_LIB(e): 0
• ERR_GET_REASON(e): 0

[vstinner]

@picnixz:

(...) the code path being taken to trigger "A failure in the SSL library occurred" means that the last OpenSSL error code was not set (either we cleared it accidently or they didn't set it correctly) and this is something that can happen elsewhere.

How can I debug such issue? According to my previous comment, OpenSSL doesn't set any error. It only says that it's a "SSL error". Even if I check ERR_peek_last_error() just after SSL_read_ex() it's still 0.

It seems like OpenSSL 4 changed SSL_read_ex() error when called after SSL_R_UNEXPECTED_EOF_WHILE_READING.

[picnixz]

This may be a bug in OpenSSL 4. Something changed in their errors:

SSL_get_error() no longer depends on the state of the error stack, so it is no longer necessary to empty the error queue before the TLS/SSL I/O operations.

So.... maybe we may ourselves be doing bad things elsewhere. I honestly do not know if the problem is on our side or their side. It may be on our side because we maybe assume that the stack entry is still there? (idk)

[vstinner]

Ok, here is a simpler Python reproducer script only using socket and ssl modules:

importos.pathimportsocketimportsslimportthreadingCERT=os.path.join('Lib', 'test', 'certdata', 'keycert.pem')
HOST='127.0.0.1'HOSTNAME='localhost'classServer(threading.Thread):
def__init__(self):
super().__init__()
self.listening=threading.Event()
self.address=Nonedefrun(self):
context=ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain(CERT)
server_sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)#server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)server_sock.bind((HOST, 0))
server_sock.listen(5)
self.address=server_sock.getsockname()
self.listening.set()
sock, addr=server_sock.accept()
sslconn=context.wrap_socket(sock, server_side=True)
request=b''whileTrue:
chunk=sslconn.recv(65537)
request+=chunkifb'\r\n\r\n'inrequest:
breakprint(f"server got request: {request!r}")
print("server sendall")
sslconn.sendall(
b'HTTP/1.0 200 OK\r\n'b'Server: TestHTTP/ Python/3.15.0a8+\r\n'b'Date: Thu, 16 Apr 2026 12:42:37 GMT\r\n'b'Content-type: text/plain\r\n\r\n')
sslconn.sendall(b'we care a bit')
print("server shutdown write")
sslconn.shutdown(socket.SHUT_WR)
print("server close socket")
sslconn.close()
server_sock.close()
defmain():
server=Server()
server.start()
server.listening.wait()
port=server.address[1]
context=ssl.create_default_context(cafile=CERT)
sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, port))
sslsock=context.wrap_socket(sock, server_hostname=HOSTNAME)
sslsock.sendall(b'GET /bizarre HTTP/1.0\r\n\r\n')
sslobj=sslsock._sslobjdefread(prefix, sslobj):
try:
data=sslobj.read(1024)
result=repr(data)
exceptssl.SSLErrorasexc:
result=f'<{exc!r}>'print(prefix, result)
foriinrange(1, 5):
read(f"client read #{i}:", sslobj)
sslsock.close()
server.join()
if__name__=="__main__":
main()

Output with OpenSSL 3:

server got request: b'GET /bizarre HTTP/1.0\r\n\r\n'
server sendall
server shutdown write
server close socket
client read #1: b'HTTP/1.0 200 OK\r\nServer: TestHTTP/ Python/3.15.0a8+\r\nDate: Thu, 16 Apr 2026 12:42:37 GMT\r\nContent-type: text/plain\r\n\r\n'
client read #2: b'we care a bit'
client read #3: <SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:2970)')>
client read #4: <SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2970)')>

Output with OpenSSL 4:

server got request: b'GET /bizarre HTTP/1.0\r\n\r\n'
server sendall
server shutdown write
server close socket
client read #1: b'HTTP/1.0 200 OK\r\nServer: TestHTTP/ Python/3.15.0a8+\r\nDate: Thu, 16 Apr 2026 12:42:37 GMT\r\nContent-type: text/plain\r\n\r\n'
client read #2: b'we care a bit'
client read #3: <SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:2984)')>
client read #4: <SSLError(1, 'A failure in the SSL library occurred (_ssl.c:2984)')>

OpenSSL 4 behaves differently on the last (4th) read (after UNEXPECTED_EOF_WHILE_READING): it fails with a generic A failure in the SSL library occurred error instead of EOF occurred in violation of protocol (OpenSSL 3).

[vstinner]

@picnixz:

This may be a bug in OpenSSL 4. Something changed in their errors:
(...)
So.... maybe we may ourselves be doing bad things elsewhere. I honestly do not know if the problem is on our side or their side. It may be on our side because we maybe assume that the stack entry is still there? (idk)

I went ahead and I created an issue in OpenSSL bug tracker openssl/openssl#30894 to ask if the new behavior was made on purpose or not.

[vstinner]

The PR is no longer a draft, it's ready for review.

I went ahead and I created an issue in OpenSSL bug tracker openssl/openssl#30894 to ask if the new behavior was made on purpose or not.

So in short, yes, the new behavior was made on purpose. After UNEXPECTED_EOF_WHILE_READING, the connection no longer exists, and SSL_read_ex() fails with a generic protocol error.

IMO this change is an acceptable solution to get the same behavior on OpenSSL 4 and on OpenSSL 3 (and older).

But I'm not sure if the fix is done at the right level. I modified SSLSocket in Lib/ssl.py, but maybe the change should be done in the C code (Modules/_ssl.c) instead.

[picnixz]

I don't have much bandwidth to check this but I think it should be done at the C level instead. AFAIR, the high-level Python part is really just a thin wrapper and there may be users that directly use the C class. Problem is that the SSL code is really large and I can't tell you exactly which places will need to be fixed...

[vstinner]

@picnixz:

I think it should be done at the C level instead. AFAIR, the high-level Python part is really just a thin wrapper and there may be users that directly use the C class

Yeah, I agree. So I wrote gh-149102 which moves the logic to the C code (Modules/_ssl.c) and adds a test.

@picnixz:

Problem is that the SSL code is really large and I can't tell you exactly which places will need to be fixed...

I searched for _ssl._SSLSocket methods calling PySSL_SetError() which behave differently on EOF: I found read(), sendfile() and write(). The pending() returns 0 on EOF, and shutdown() raises an OSError error (not a SSL error) on EOF. The do_handshake() is the first called method, it should not be called after it gets an EOF (correct me if I'm wrong).

[vstinner]

I close this PR. I merged the PR changing the C code instead (PR gh-149102).