Skip to content

gh-119400: make_ssl_certs: update reference test data automatically, pass in expiration dates as parameters #119400 - #119401

Merged
encukou merged 4 commits into
python:mainfrom
kanavin:fix-make-ssl-certs
Sep 25, 2024
Merged

gh-119400: make_ssl_certs: update reference test data automatically, pass in expiration dates as parameters #119400 #119401
encukou merged 4 commits into
python:mainfrom
kanavin:fix-make-ssl-certs

Conversation

@kanavin

@kanavinkanavin commented May 22, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

As discussed here:
#107594

make_ssl_certs.py has a few shortcomings. In particular:

  • it generates certificates, but does not update reference data in tests that use them, instead asking the user to copy paste the data by hand (expiration dates and serial numbers in particular)
  • it is supposed to be run by hand and isn't executed in builds, which means its output has to be checked into git, cluttering the source tree, and complicating reviews of pull requests that change that output.
  • expiration dates are hardcoded into the tool and can't be passed in as parameters

This pull request aims to address first and last issue, so then #107594 can move forward on top of them.

@bedevere-appbedevere-appBot mentioned this pull request May 22, 2024
Open
@kanavin

kanavin commented May 22, 2024

Copy link
Copy Markdown
ContributorAuthor
OSError: [Errno 30] Read-only file system: '/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/certdata'

This means CI is set up so that modifying the source tree is not possible. Suggestions? I still think it's worth making make_ssl_certs execution a part of the build, but the complication is that its output needs to be written into build dir, and both installation and tests needs to find it there.

@kanavinkanavin mentioned this pull request May 22, 2024
Merged
@kanavin

kanavin commented May 23, 2024

Copy link
Copy Markdown
ContributorAuthor
OSError: [Errno 30] Read-only file system: '/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/certdata'

This means CI is set up so that modifying the source tree is not possible. Suggestions? I still think it's worth making make_ssl_certs execution a part of the build, but the complication is that its output needs to be written into build dir, and both installation and tests needs to find it there.

I've concluded that this is not feasible for now:

  • needs invasive changes to Makefile
  • requires openssl executable at build time
  • breaks build reproducibility as every build is going to have different certificates installed, even if they're only used for testing.

I'll drop that from this PR, and make it only about not hardcoding reference certificate data and expiration parameters.

@kanavinkanavin changed the title gh-119400: make_ssl_certs: run at build time, update reference test data automatically #119400 gh-119400: make_ssl_certs: update reference test data automatically, pass in expiration dates as parameters #119400 May 23, 2024
@kanavin
kanavinforce-pushed the fix-make-ssl-certs branch from 87c0672 to be2c50fCompareMay 23, 2024 09:59
@kumaraditya303
kumaraditya303 removed their request for review June 23, 2024 07:54
@kanavin

kanavin commented Sep 6, 2024

Copy link
Copy Markdown
ContributorAuthor

This seems to be not getting any attention, is there something I can do to push it forward?

@gvanrossum
gvanrossum requested review from encukou and sethmlarson and removed request for 1st1, asvetlov and gvanrossumSeptember 6, 2024 15:59
encukou
encukou reviewed Sep 13, 2024
View reviewed changes

@encukouencukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay; my review queue grew too long and I had to trim it.
This looks great, but let's change a few details:

Comment threadLib/test/test_asyncio/utils.py Outdated
Comment threadMisc/NEWS.d/next/Build/2024-05-22-13-18-02.gh-issue-119400.WEt83v.rst Outdated
Comment threadLib/test/certdata/allsans.pem
Comment threadLib/test/certdata/make_ssl_certs.py Outdated
kanavinand others added 2 commits September 25, 2024 11:37
@kanavin@encukou
Lib/test/certdata: do not hardcode reference cert data into tests
0e82723
The script was simply printing the reference data and asking
users to update it by hand into the test suites. This can
be easily improved by writing the data into files and
having the test cases load the files.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
@kanavin
make_ssl_certs: make it possible to pass in expiration dates from com…
4bff2d1
…mand line
Note that the defaults are same as they were, so if nothing is
specified, the script works exactly as before.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
@kanavin

kanavin commented Sep 25, 2024

Copy link
Copy Markdown
ContributorAuthor

@encukou Thanks, everything should be addressed now.

encukou
encukou reviewed Sep 25, 2024
View reviewed changes
Comment threadLib/test/certdata/make_ssl_certs.py Outdated
@encukou
encukou enabled auto-merge (squash) September 25, 2024 20:59
@encukou
encukou merged commit 1ff1b89 into python:mainSep 25, 2024
emilyemorehouse added a commit to lysnikolaou/cpython that referenced this pull request Sep 26, 2024
@emilyemorehouse
Merge branch 'main' into fix-help-double-prompt
4cdb3ae
* main: (69 commits)
Add "annotate" SET_FUNCTION_ATTRIBUTE bit to dis. (python#124566)
pythongh-124412: Add helpers for converting annotations to source format (python#124551)
pythongh-119180: Disallow instantiation of ConstEvaluator objects (python#124561)
For-else deserves its own section in the tutorial (python#123946)
Add 3.13 as a version option to the crash issue template (python#124560)
pythongh-123242: Note that type.__annotations__ may not exist (python#124557)
pythongh-119180: Make FORWARDREF format look at __annotations__ first (python#124479)
pythonGH-58058: Add quick reference for `ArgumentParser` to argparse docs (pythongh-124227)
pythongh-41431: Add `datetime.time.strptime()` and `datetime.date.strptime()` (python#120752)
pythongh-102450: Add ISO-8601 alternative for midnight to `fromisoformat()` calls. (python#105856)
pythongh-124370: Add "howto" for free-threaded Python (python#124371)
pythongh-121277: Allow `.. versionadded:: next` in docs (pythonGH-121278)
pythongh-119400: make_ssl_certs: update reference test data automatically, pass in expiration dates as parameters python#119400 (pythonGH-119401)
pythongh-119180: Avoid going through AST and eval() when possible in annotationlib (python#124337)
pythongh-124448: Update Windows builds to use Tcl/Tk 8.6.15 (pythonGH-124449)
pythongh-123884 Tee of tee was not producing n independent iterators (pythongh-124490)
pythongh-124378: Update test_ttk for Tcl/Tk 8.6.15 (pythonGH-124542)
pythongh-124513: Check args in framelocalsproxy_new() (python#124515)
pythongh-101100: Add a table of class attributes to the "Custom classes" section of the data model docs (python#124480)
Doc: Use ``major.minor`` for documentation distribution archive filenames (python#124489)
...
kanavin added a commit to kanavin/cpython that referenced this pull request Sep 26, 2024
@kanavin
Revert "pythongh-119400: make_ssl_certs: update reference test data a…
496df6c
…utomatically, pass in expiration dates as parameters python#119400 (pythonGH-119401)"
This reverts commit 1ff1b89.
halstead pushed a commit to openembedded/openembedded-core that referenced this pull request Feb 19, 2026
@kanavin@rpurdie
time64.inc: clean up and add upstream tickets where issues remain
640e2f9
Most issues were resolved via upstream version updates that bring in
needed fixes:
glib-2.0 update to 2.78.0 that includes:
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3547https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3550
curl update to 8.3.0 that includes
curl/curl#11610
util-linux update to 2.39 that includes
util-linux/util-linux#2430util-linux/util-linux@3ab9e69util-linux/util-linux#2435
glib-networking update to 2.78.0 that includes
https://gitlab.gnome.org/GNOME/glib-networking/-/merge_requests/241
python3-cryptography update to 42.0.0 which resolvespyca/cryptography#9370 via
pyca/cryptography#9964
perl update to 5.40.0 which includes
Perl/perl5#21379
python3 update to 3.13.0 which includes
python/cpython#118425
python3 update to 3.13.1 which includes
python/cpython#124972
python3 update to 3.14.0 which includes
python/cpython#119401python/cpython#125045python/cpython#107594python/cpython#125104
tcl update to 9.0.0 which includes
tcltk/tcl@4ca6172
(tcl8 recipe has a simple backport of this)
dbus update to 1.16.0 which includes
https://gitlab.freedesktop.org/dbus/dbus/-/merge_requests/444https://gitlab.freedesktop.org/dbus/dbus/-/merge_requests/289
openssh update to 10.0p1 which includes
openssh/openssh-portable#425https://bugzilla.mindrot.org/show_bug.cgi?id=3684https://marc.info/?l=openbsd-bugs&m=172561736524815&w=2https://lists.mindrot.org/pipermail/openssh-unix-dev/2024-October/041621.html
(all reporting the same issue)
gcc update to 15.1 which includes
llvm/llvm-project#99699
via gcc-mirror/gcc@fa32100
and allows dropping special flags and exceptions for gcc-sanitizers.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukouencukouencukou left review comments
@willingcwillingcAwaiting requested review from willingcwillingc is a code owner
@sethmlarsonsethmlarsonAwaiting requested review from sethmlarson

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kanavin@encukou