Support Jetty's live cert reload on HTTPS frontend

[mlsorensen]

Description

This PR implements Jetty's hot certificate reload per jetty/jetty.project#5042

When the keystore changes, the API server begins using the new certificate. Note that this functionality doesn't support live change of keystore password, only certificate.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested locally with self-signed certs. Changed the cert in the keystore while management server was running, reloaded the website, and inspected the https certificate.

Tested against cmk, which continued to function after the cert change. Also tested against UI - website needed to be refreshed due to browser security. Perhaps a future enhancement would catch these errors and auto-refresh?

Here you can see the local cloudstack management server using a new cert after it is loaded into a keystore:

[root@kvmlab1 management]# echo | openssl s_client -showcerts -connect localhost:8443 2>/dev/null | openssl x509 -inform pem -noout -text | grep "Not Before"
Not Before: Mar 21 21:09:50 2023 GMT

change cert

[root@kvmlab1 management]# openssl x509 -req -in cloud.csr -signkey cloud.key > cloud.crt
Signature ok
subject=C = US, ST = UT, L = Alpine, O = Marcus, OU = Marcus, CN = kvmlab1, emailAddress = example@example.com
Getting Private key
[root@kvmlab1 management]# openssl pkcs12 -export -in cloud.crt -inkey cloud.key -name cloud -passout pass:****** > cloud-localhost.pk12
[root@kvmlab1 management]# keytool -importkeystore -srckeystore cloud-localhost.pk12 -srcstoretype PKCS12 -destkeystore cloud.jks -deststoretype JKS -srcstorepass ****** -deststorepass ****** -destkeypass ****** -noprompt
Importing keystore cloud-localhost.pk12 to cloud.jks...
Warning: Overwriting existing alias cloud in destination keystore
Entry for alias cloud successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

check cert

[root@kvmlab1 management]# echo | openssl s_client -showcerts -connect localhost:8443 2>/dev/null | openssl x509 -inform pem -noout -text | grep "Not Before"
Not Before: Mar 21 21:24:47 2023 GMT

[codecov]

Codecov Report

Merging #7355 (cc00c6a) into main (2aa3f98) will not change coverage.
The diff coverage is n/a.

@@ Coverage Diff @@## main #7355 +/- ##
=========================================
Coverage 12.68% 12.68% Complexity 8656 8656 =========================================
Files 2718 2718 Lines 256169 256169 Branches 39932 39932 =========================================
Hits 32504 32504 Misses 219531 219531 Partials 4134 4134 

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[weizhouapache]

code lgtm

[yadvr]

Code LGTM - but haven't tested it.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[mlsorensen]

Merging based on 2x LGTM and manual testing of feature.

[yadvr]

Sorry @mlsorensen per community guidelines, there were no @blueorangutan smoketests were performed and generally, it's not preferred that PR author serves manual tests.

However, it's possible there's no regression as smoketests in Github Actions against simulator have passed. I'll kick a round of BO smoketests on #7344