OpenSSL 3.0 performance issue: SSLContext.set_default_verify_paths / load_verify_locations about 5x slower

[fcfangcc]

Bug report
Example code in ubuntu20.04(openssl1.1) is much faster than ubuntu22.04(openssl3.x)
Not just speed, CPU occupancy ubuntu22.04(openssl3.x) is many times of ubuntu20.04(openssl1.1)
I'm not sure whether it's OpenSSL or Python adaptation problem

import socket
import ssl
import time

import certifi
hostname = 'www.python.org' # any support https hostname
times = 100
pem_where = certifi.where()
context = ssl.create_default_context()
verify_total_time = 0

for i in range(times):
 with socket.create_connection((hostname, 443)) as sock:
 with context.wrap_socket(sock, server_hostname=hostname) as ssock:
 verify_start_time = time.time()
 context.load_verify_locations(pem_where)
 verify_total_time += time.time() - verify_start_time
 ssock.version()
 
print(f"total {verify_total_time:.4f}, avg {verify_total_time/times:.4f}")

in my environment with docker:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 3321 root 20 0 304140 81148 12792 S 42.0 0.8 0:29.69 ipython (ubuntu22.04)
 3850 root 20 0 203348 52632 11576 S 16.7 0.5 0:06.34 ipython (ubuntu20.04)

total 5.8634, avg 0.0586 (ubuntu22.04)
total 0.6753, avg 0.0068 (ubuntu20.04)

Your environment

• CPython versions tested on: 3.10.5
• Operating system and architecture: ubuntu20.04(openssl1.1) and ubuntu22.04(openssl3.0.2), build from source
• certifi==2022.6.15

[tiran]

It is a problem in OpenSSL 3.0. Python upstream does not support OpenSSL 3.0 for good reasons. It has performance and backwards compatibility issue. On my system load_verify_locations is about 5 times slower when using system certificates.

3.12.0a0 (heads/main-dirty:88e4eeba25d, Jul 20 2022, 08:22:18) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
OpenSSL 3.0.5 5 Jul 2022
100 loops of 'load_verify_locations' in 3.965sec

3.12.0a0 (heads/main-dirty:88e4eeba25d, Jul 20 2022, 08:22:18) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
OpenSSL 1.1.1n 15 Mar 2022
100 loops of 'load_verify_locations' in 0.871sec

By the way you should not combine ssl.create_default_context() with certifi. A default context already loads the system cert store.

[tiran]

import ssl
import sys
import time

LOOPS = 100

print(sys.version)
print(ssl.OPENSSL_VERSION)

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

start = time.monotonic()
for i in range(LOOPS):
 ctx.load_verify_locations('/etc/pki/tls/cert.pem')
dur = time.monotonic() - start
print(f"{LOOPS} loops of 'load_verify_locations' in {dur:0.3f}sec")

[fcfangcc]

Thansk.Example is separate from requests,httpx......
It seems that using ubuntu22 with Python is not a good idea, it`s default openssl3.

[tiran]

I recommend that you raise a bug with OpenSSL. Their SSL_CTX_set_default_verify_paths and SSL_CTX_load_verify_locations functions are much slower in 3.0 than in 1.1.1.

[tiran]

According to "perf', OpenSSL 3.0 is spending a lot of time in pthread_rwlock lock/unlock followed by sa_doall, getrn, and several libcrypto string functions (ossl_lh_strcasehash, ossl_tolower, OPENSSL_strcasecmp, OPENSSL_sk_value).

[arhadthedev]

a lot of time in pthread_rwlock lock/unlock

Fortunately, the issue is known (so no need to report it once more): openssl/openssl#16791 initial report and openssl/openssl#18814 pointing to a root issue.

[iritkatriel]

Should we close this as a third party issue?

[risicle]

As an outsider, it appears to me that there's always going to be a risk of this becoming a severe performance bottleneck as long as python's ssl API doesn't expose the ability to re-use an X509_STORE - forcing the system's CA bundle to be re-parsed for every new SSLContext.

[ThomasChr]

This one bit me today. And it did bite quite hard!

Beginning with cPython 3.11.5 (on Windows) we're shipping OpenSSL 3 instead of 1.1
So when I installed 3.12.0 on a customer system it took a few minutes until I got a call. The system is blocked, one Python Process takes all of the CPU and no one can work anymore.
Turns out that my Python Process which uses 32 Threads and does nothing more then sending some simple numbers into the Internet took all of the CPU - instead of 20% as it was with Python 3.11. Thats a very bad performance regression.

One of my other processes has some more accurate numbers - before:

15 seconds total, 1 second on CPU

and after:

30 seconds total, 15 seconds on CPU

Pretty sure you could have cooked coffee on the cpu after that.
(With cProfile you could see most of the CPU time in {method 'load_verify_locations' of '_ssl._SSLContext' objects})

On the upper level I'm using the requests library in my code and there where two solutions to the problem:

1. Add verify=False to all of the requests
2. Install Python 3.11.4

I used Method 2, but now I need to stay on Python 3.11.4 for all eternity. Hopefully this will be fixed some day soon.

Hope this post helps other people with the same problem. I will see if I'll add some info to the linked OpenSSL Issues also.

[ThomasChr]

As a quick solution:
It seems that the time is spent when OpenSSL verifys a certificate - can't we try to cache this verification? In my code I'm making thousands of requests to the exact same host - I don't need to verify it every time.
It would be a solution for the user to add verify=False beginning with the second request to the same host himself - but that means he needs to find out the root course first, which is not an easy task.

I'm a little bit sad that people will say that Python is dog-slow when really it isn't our fault. But saying "not our fault" won't help much here. Having a plan B would be great.
(I'm not entirely convinced that OpenSSL will fix this problem soon...)