Make Azure Pipelines optional on GitHub PRs

[vstinner]

BPO	39837
Nosy	@brettcannon, @vstinner, @ned-deily, @zooba, @miss-islington, @shihai1991
PRs	bpo-39837: Remove .azure-pipelines/ directory #18769 bpo-39837: Disable macOS tests on Azure Pipelines #18818 [3.8] bpo-39837: Disable macOS tests on Azure Pipelines (GH-18818) #18820 [3.7] bpo-39837: Disable macOS tests on Azure Pipelines (GH-18818) #18821 Revert "bpo-39837: Disable macOS tests on Azure Pipelines (GH-18818)" #18835
Files	Annotation 2020-03-05 223851.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-30.18:36:36.085>
created_at = <Date 2020-03-03.17:05:19.756>
labels = ['3.8', '3.7', 'tests', '3.9']
title = 'Make Azure Pipelines optional on GitHub PRs'
updated_at = <Date 2020-05-15.15:12:05.138>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-05-15.15:12:05.138>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-30.18:36:36.085>
closer = 'brett.cannon'
components = ['Tests']
creation = <Date 2020-03-03.17:05:19.756>
creator = 'vstinner'
dependencies = []
files = ['48956']
hgrepos = []
issue_num = 39837
keywords = ['patch']
message_count = 50.0
messages = ['363279', '363289', '363291', '363292', '363382', '363384', '363394', '363405', '363425', '363447', '363455', '363458', '363460', '363466', '363498', '363568', '363569', '363570', '363571', '363808', '365460', '365496', '365503', '365519', '365535', '365561', '365604', '365609', '365799', '367578', '367593', '367600', '367627', '367659', '367673', '367678', '367769', '367781', '367798', '367799', '367800', '367801', '367802', '367804', '367805', '367806', '367807', '367861', '368358', '368952']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'vstinner', 'ned.deily', 'python-dev', 'steve.dower', 'miss-islington', 'shihai1991']
pr_nums = ['18769', '18818', '18820', '18821', '18835']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39837'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[vstinner]

The Azure Pipelines jobs have been reimplemented as GitHub actions which are better integrated with GitHub:

• Docs / Docs (pull_request)
• Tests / Windows (x64) (pull_request)
• Tests / macOS (pull_request)
• Tests / Ubuntu (pull_request)
• etc.

Azure Pipelines runs the same jobs, but it looks slower. It is voting and so prevents to merge a PR until it completes.

I propose to simply remove the job.

I already proposed it on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/message/NC2ZS4WSF5AYGUUMBB7I4YIQ4YJXWBA5/

In this thread:
https://mail.python.org/archives/list/python-dev@python.org/thread/2NSPJUEWULTLLALR3HY3H2PRYAUT474C/#NC2ZS4WSF5AYGUUMBB7I4YIQ4YJXWBA5

[zooba]

Deleting the files is not the right first step.

First, it needs to be changed to a non-required check.

Then, I can use the web UI to disable it starting.

*Then*, we can remove *some* of the files in the directory. Others are used for the official release, and have to stay.

PR 18769 should *not* be merged.

[vstinner]

PR 18769 should *not* be merged.

Ok, I closed it.

First, it needs to be changed to a non-required check.

Who is allowed to do that?

[zooba]

Who is allowed to do that?

Brett did it the first time. I'm having too much trouble with GitHub right now to find the current admins.

[brettcannon]

Yes, I can do it.

And to answer Victor's question on the PR that he closed, we can make any individual status check required. So probably:

• Docs
• Ubuntu
• Windows x86
• Windows x64

Just let me know when we are ready to merge a PR and I will switch off Azure Pipelines and make these checks required.

[zooba]

There's no PR required. We need to change the required check so when I disable new builds from running it won't break existing PRs.

As for actually changing the files in the repo, I don't see any hurry there. We can clean up a few of them, and maybe I can move the release build scripts into the PC folder (though would have to backport that through to 3.7).

But I'm ready to disable the builds from running as soon as they're not a required check.

[vstinner]

And to answer Victor's question on the PR that he closed, we can make any individual status check required.

Great! I saw a macOS failure this week which was caused by an internal CI error. I hope that it will be fixed soon. In the meanwhile, I would suggest to not make the macOS job mandatory, but pay attention manually to its status ;-) Apart of macOS, other jobs look reliable.

[brettcannon]

Actually, I just realized we can't make these status checks required because they don't always run. :) Our Actions are smart enough to not run when they aren't necessary, i.e. doc changes don't run the rest of the checks. And so making the OS-specific tests required would block doc tests.

Basically we would either have to waste runs on things that aren't really necessary but then be able to require runs, or we have to just rely on people paying attention to failures.

I'm personally for the latter.

[zooba]

Are you sure the required check isn't just for failures? Surely GitHub is smarter than requiring checks that it can tell aren't required (as it's their logic to include/exclude, not ours).

If they're not, I now have many internal contacts there, so we can probably get it fixed :)

[brettcannon]

It was an old issue that if required checks didn't run it would block, but hopefully it's fixed. :)

I have gone ahead and removed the Azure Pipelines requirement from 3.7, 3.8, and master and flipped on the check requirements for the ones I listed.

[zooba]

Looks like it isn't fixed... #18774