Run stubtest on stubs on different platforms by sobolevn · Pull Request #8923 · python/typeshed

sobolevn · 2022-10-18T11:43:13Z

Open problems:

How to install deps on other platforms. brew works well on macos, but no idea how to use windows (I've never touched it since 2008)
Listing platform specific packages. Which ones should we add?
I still need to change daily.yml to do the same thing for all stubs
How to test this? Not quite clear for now. I will need to tweak some values

sobolevn · 2022-10-18T12:02:17Z

I have no idea how to fix this Windows problem:


Run STUBS=$(
ParserError: D:\a\_temp\4a20ead6-174f-4640-96c9-b47d87c97584.ps1:8
Line |
8 | if [ -n "$STUBS" ]; then
| ~
| Missing '(' after 'if' in if statement.

It is valid sh, what more do you need? 🤣
Probably we need two different test runners: one for win and one for posix

AlexWaygood · 2022-10-18T12:04:27Z

It is valid sh, what more do you need? 🤣
Probably we need two different test runners: one for win and one for posix

The Windows shell is a whole different beast.

Can we use WSL in GitHub Actions? That might be an easy workaround.

sobolevn · 2022-10-18T12:13:28Z

Looks like it works! https://github.com/python/typeshed/actions/runs/3273273447/jobs/5385277526

Avasam · 2022-10-18T16:33:29Z

It is valid sh, what more do you need?

But not valid powershell ! Although you already figured out that you can specify the shell in github actions. Which is what I would've recommended over:

Can we use WSL in GitHub Actions? That might be an easy workaround.

How to install deps on other platforms

Chocolatey is what I see most often on windows (at least when developing with Node and Python).
Brew for MacOS.

Listing platform specific packages. Which ones should we add?

Whatever's needed as you develop/test this I guess?
Taking a look at stubtest_allowlist.txt files, and skip=true in METADATA.toml, seems to be a good starting point to get an idea of which stubs have os-specific needs.

Akuli · 2022-10-19T07:59:47Z

You can use shell: bash on Windows github actions.

sobolevn · 2022-10-19T08:00:11Z

I already do that, thanks!

AlexWaygood · 2022-10-19T09:55:26Z

Could you make this change to the METADATA.toml file for D3DShot as part of this PR, to test that the Windows part of this is working?

 [tool.stubtest]
- # The library only works on Windows; we currently only run stubtest on Ubuntu for third-party stubs in CI.- # See #8660- skip = true+ platform = ["win32"]

Akuli · 2022-10-19T09:59:40Z

Nit: I would prefer if the metadata key was named platforms instead of platform.

AlexWaygood · 2022-10-19T10:49:25Z

You might have to add a @tests/requirements-stubtest.txt file to D3DShot, and pin Pillow to a specific revision, in order to get it to install.

sobolevn · 2022-10-19T10:54:32Z

Output from windows daily run: https://github.com/sobolevn/typeshed/actions/runs/3280905801/jobs/5402262235

…/sobolevn/typeshed into run-stubs-on-different-platforms

sobolevn · 2022-10-19T17:37:25Z

Since we now have platform-specific stubtest failures: https://github.com/sobolevn/typeshed/actions/runs/3283716889/jobs/5408788907

I will introduce platform-specific allowlists. Something similar to what we have in stdlib/

AlexWaygood

Looks great to me! Thanks for all the work you've done on this!

github-actions · 2022-11-11T00:21:58Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

JelleZijlstra

Slight wording suggestion because many stubs may be "supported" on other platforms, we just don't need to run stubtest multiple times.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

github-actions · 2022-11-11T10:21:33Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

AlexWaygood

A few more tiny nits I spotted while looking over again

sobolevn · 2022-11-11T13:04:43Z

 [tool.stubtest]
-# The library only works on Windows; we currently only run stubtest on Ubuntu for third-party stubs in CI.
-# See #8660
+# TODO: re-enable


Suggested change
# TODO: re-enable

Wait, this TODO comment I did like -- it would be good if we could figure out how to enable stubtest for this package!

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

github-actions · 2022-11-11T13:17:43Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

github-actions · 2022-11-11T13:36:27Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

AlexWaygood · 2022-11-11T13:39:53Z

Time for this to go in.

Avasam · 2022-11-12T17:05:42Z

@sobolevn & @AlexWaygood There's an unforseen issue: Now I can't run stubtest locally on any stub that doesn't explicitely specify my non-linux OS!

If no platform is specified in METADATA.toml, it should be allowed to be run on any OS, but the CI should only run linux.

sobolevn · 2022-11-12T17:31:52Z

I think we can add --local-run flag to be sure that it is indeed runing on a local env.

Avasam · 2022-11-12T17:41:15Z

Proposed solution: #9173

sobolevn added 3 commits October 18, 2022 14:39

Run stubtest on stubs on different platforms
61deba6

Fix CI
0a96d99

Fix CI
8149e3c

This comment has been minimized.
Sign in to view

Fix CI
7d364f7

This comment has been minimized.
Sign in to view

AlexWaygood reviewed Oct 18, 2022
View reviewed changes

Comment threadtests/stubtest_third_party.py Outdated

sobolevn added 2 commits October 19, 2022 13:35

Address review
1a9af3a

Run daily on differeny platforms
f00714a

AlexWaygood reviewed Oct 19, 2022
View reviewed changes

Comment thread.github/workflows/stubtest_stubs.yml Outdated

Fix daily run
3f31bed

This comment has been minimized.
Sign in to view

Avasam mentioned this pull request Oct 19, 2022
pywin32: Remove IID that doesn't exist at runtime #8930
Merged

Merge branch 'master' into run-stubs-on-different-platforms
8eb0c83

This comment has been minimized.
Sign in to view

sobolevn added 2 commits October 19, 2022 20:29

Rename stubtest files
c8650b3

Merge branch 'run-stubs-on-different-platforms' ofhttps://github.com…
770c5aa
…/sobolevn/typeshed into run-stubs-on-different-platforms

This comment has been minimized.
Sign in to view

Add platform-specific allowlists for stubs
fdf13bb

Delete unused allowlist entries
7df5a1f

AlexWaygood approved these changes Nov 11, 2022
View reviewed changes

AlexWaygood requested a review from JelleZijlstra November 11, 2022 00:15

JelleZijlstra reviewed Nov 11, 2022
View reviewed changes

Comment threadCONTRIBUTING.md Outdated

JelleZijlstra reviewed Nov 11, 2022
View reviewed changes

Update CONTRIBUTING.md
949bf46
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

AlexWaygood reviewed Nov 11, 2022
View reviewed changes

Comment thread.github/workflows/stubtest_third_party.yml Outdated
Comment thread.github/workflows/stubtest_third_party.yml Outdated
Comment thread.github/workflows/stubtest_third_party.yml
Comment threadCONTRIBUTING.md Outdated

sobolevn commented Nov 11, 2022
View reviewed changes

Apply suggestions from code review
faac1be
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood reviewed Nov 11, 2022
View reviewed changes

Comment threadstubs/D3DShot/METADATA.toml

Add TODO comment back
9464335

AlexWaygood merged commit 9cd9f6f into python:mainNov 11, 2022

This was referenced Nov 11, 2022
Do not run stubtest_third_party on requirements change #9160
Merged
Run stubtest under different OS environments #8660
Closed

This was referenced Nov 12, 2022
Run stubtest on windows for colorama #9168
Merged
Run stubtest for psutil on all platforms #9169
Merged

Avasam mentioned this pull request Nov 12, 2022
stubtest_third_party.py: Allow running non-listed platforms locally #9173
Merged

This was referenced Nov 12, 2022
cffi: run stubtest on Windows #9178
Merged
Fix paramiko stubtest on Windows #9179
Merged
Run stubtest on Windows for waitress #9181
Merged

AlexWaygood mentioned this pull request Aug 16, 2023
Bump aiofiles to 23.2.* #10584
Merged

srittau mentioned this pull request Mar 17, 2025
Do we need --specified-platforms-only in stubtest_third_party? #13641
Open

Uh oh!

Uh oh!

Conversation

sobolevn commented Oct 18, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sobolevn commented Oct 18, 2022

Uh oh!

AlexWaygood commented Oct 18, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

sobolevn commented Oct 18, 2022

Uh oh!

This comment has been minimized.

Avasam commented Oct 18, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Akuli commented Oct 19, 2022

Uh oh!

sobolevn commented Oct 19, 2022

Uh oh!

AlexWaygood commented Oct 19, 2022

Uh oh!

Akuli commented Oct 19, 2022

Uh oh!

Uh oh!

AlexWaygood commented Oct 19, 2022

Uh oh!

sobolevn commented Oct 19, 2022

Uh oh!

This comment has been minimized.

This comment has been minimized.

sobolevn commented Oct 19, 2022

Uh oh!

This comment has been minimized.

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 11, 2022

Uh oh!

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 11, 2022

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sobolevnNov 11, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexWaygoodNov 11, 2022

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 11, 2022

Uh oh!

Uh oh!

github-actionsBot commented Nov 11, 2022

Uh oh!

AlexWaygood commented Nov 11, 2022

Uh oh!

Avasam commented Nov 12, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sobolevn commented Nov 12, 2022

Uh oh!

Avasam commented Nov 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

sobolevn commented Oct 18, 2022•
edited
Loading

AlexWaygood commented Oct 18, 2022•
edited
Loading

Avasam commented Oct 18, 2022•
edited
Loading

sobolevnNov 11, 2022•
edited
Loading

Avasam commented Nov 12, 2022•
edited
Loading