Migrate CI from azure pipelines to GitHub Actions by andersy005 · Pull Request #4730 · pydata/xarray

andersy005 · 2020-12-23T17:22:49Z

As discussed in today's dev meeting, it'd be convenient to migrate all our CIs from azure pipelines to GitHub Actions.

Consolidate all style/type checks (isort, black, mypy, flake8) into a single workflow and use the pre-commit GitHub action.
Create a main CI workflow and parametrize it for different settings
Add minimum version policy workflow
Add Doctests workflow
Remove azure pipelines config
Update docs: remove references to azure pipelines

keewis · 2020-12-23T23:15:37Z

I think the EOF issue is a bug in blackdoc, it shouldn't remove the newline immediately before EOF.

To fix the mypy errors I would change the hook configuration to exclude properties and asv_bench:

 - repo: https://github.com/pre-commit/mirrors-mypyrev: v0.790 # Must match ci/requirements/*.ymlhooks:
- id: mypyexclude: "properties|asv_bench"

if we do want to type check those we could add a empty __init__.py to properties and fix the errors in asv_bench

keewis

thanks, @andersy005, this looks great.

The attendance of that meeting was pretty low so we should probably wait on a few more reviews.

jhamman · 2020-12-29T04:52:10Z

Thanks @andersy005 for pulling this together. This will make for a much cleaner CI experience.

jhamman · 2021-01-07T18:16:21Z

@andersy005 - I merged #4688, so we just have some conflicts to sort out now.

andersy005 · 2021-01-07T18:20:47Z

@andersy005 - I merged #4688, so we just have some conflicts to sort out now.

I'm working on it. Will update this branch shortly

andersy005 · 2021-01-07T18:53:48Z

Heres' what we get with [skip-ci] tag in the git commit message: Cc @dcherian

keewis

one question, otherwise this looks good to me.

keewis · 2021-01-07T22:10:18Z

+ python xarray/util/print_versions.py
+ - name: Import xarray
+ run: |
+ python -OO -c "import xarray"


we currently don't use the -OO flag in the upstream-dev CI. Should we add it to the upstream-dev CI or remove it here and from CI Additional?

@andersy005 - thoughts on this one? Otherwise I think we merge this and follow up with dev doc updates.

I don't have a preference. I used the -OO flag because it was present in the previous azure pipelines CI. For consistency, I'm going to remove it

great, thanks. If we had a specific reason for -OO we can always add it back in a new PR, I guess.

keewis · 2021-01-07T22:25:04Z

actually, we have a picture in the contributing docs which is really out-of-date now (there's still stickler and appveyor). Should we update that here, or is it fine to keep it until we rewrite the contributing docs (see #4361)?

dcherian · 2021-01-07T22:35:43Z

Looks great. Thanks! I think we can wait to update the docs.

keewis · 2021-01-11T12:10:59Z

Merging. If any issues remain we can fix them in new PRs.

* upstream/master: (342 commits) fix decode for scale/ offset list (pydata#4802) Expand user dir paths (~) in open_mfdataset and to_zarr. (pydata#4795) add a version info step to the upstream-dev CI (pydata#4815) fix the ci trigger action (pydata#4805) scatter plot by order of the first appearance of hue (pydata#4723) don't skip the scheduled CI (pydata#4806) coords: retain str dtype (pydata#4759) Fix interval labels with units (pydata#4794) Always force dask arrays to float in missing.interp_func (pydata#4771) Print number of variables in repr (pydata#4762) install conda as a library in the minimum dependency check CI (pydata#4792) Migrate CI from azure pipelines to GitHub Actions (pydata#4730) use conda.api instead of parallel calls to the conda binary (pydata#4775) Speed up missing._get_interpolator (pydata#4776) Remove special case in guess_engines (pydata#4777) improve typing of OrderedSet (pydata#4774) CI: ignore some warnings (pydata#4773) DOC: update hyperlink for xskillscore (pydata#4778) drop support for python 3.6 (pydata#4720) Trigger upstream CI on cron schedule (by default) (pydata#4729) ...

max-sixty · 2021-01-19T23:58:22Z

This seems to be going well, thanks again!

The "Overriding CI behaviors" is fairly prominent in the PR template, when I wouldn't expect it to be used that often. Was that intended? Should it be a markdown comment?

andersy005 · 2021-01-20T00:04:43Z

The "Overriding CI behaviors" is fairly prominent in the PR template, when I wouldn't expect it to be used that often. Was that intended? Should it be a markdown comment?

The current choice was just a first stab at documenting the CI "special" behaviors.

👍🏽 to making the "overriding CI behaviors" section a markdown comment and documenting this in the contributing guide (for convenience) if need be.

andersy005 · 2021-02-25T22:03:22Z

+ triggered: ${{ steps.detect-trigger.outputs.trigger-found }}
+ steps:
+ - uses: actions/checkout@v2
+ - uses: ./.github/actions/detect-ci-trigger


This action was moved into its own repository.
You can replace uses: ./.github/actions/detect-ci-trigger with
uses: xarray-contrib/ci-trigger@v1

andersy005 added 3 commits December 23, 2020 10:19

Run isort, black, mypy, flake8 checks via pre-commit
4168a5f

Fix line ending
24c7d78

Fix end of line
855d6b5

andersy005 added 4 commits December 23, 2020 18:13

Fix mypy pre-commit hook. Thanks@keewis
8f9f81b

Add main CI
eb6581d

Add additional CI
b37d894

Fetch all history for all branches and tags
60d08eb

andersy005 added the CI Continuous Integration tools label Dec 24, 2020

andersy005 added 10 commits December 23, 2020 19:29

Add windows environment
bdd3986

Import xarray
26604a5

Add doctests workflow
6445b47

Add minimum version policy workflow
80da331

Simplify if logic
32114b2

Add flaky and backend-api-v2 settings
4eacb65

Fix if elif else statements
4c6252b

Fix typo
002ab25

Remove azure pipelines configurations
e65c655

Fix environment file name
06b8334

andersy005 marked this pull request as ready for review December 24, 2020 04:49

andersy005 added 5 commits December 24, 2020 18:08

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA
e70fb89

Upload code coverage for additional CI
edf8d14

Cache conda pkgs_dir
ec1e5fc

Fix cache key
257529b

Remove unnecessary cache number variable
26a0316

andersy005 added the needs review label Dec 28, 2020

keewis reviewed Dec 29, 2020
View reviewed changes

Comment thread.github/workflows/ci.yaml
Comment threadci/requirements/environment.yml
Comment thread.github/workflows/ci.yaml Outdated
Comment thread.github/workflows/ci-pre-commit.yml

keewis mentioned this pull request Dec 29, 2020
Trigger upstream CI on cron schedule (by default) #4729
Merged
3 tasks

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA
b2287ae

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA
1328110

andersy005 added 6 commits January 7, 2021 11:22

Use py37
7c4de89

Remove all references to py36 env file
c60dcd9

Add check for skip ci
5e6ddf2

rename job to detect ci trigger
93ff1ce

[skip ci] Empty commit
3cf7d3e

[skip-ci] Test skip CI trigger
a6f4d60

andersy005 requested review from jhamman and keewis January 7, 2021 18:45

Update PR template
fe18db1

andersy005 added 2 commits January 7, 2021 11:59

Fix typ
f4088f6

GH markdown doesn't like lists in <sub></sub>
c869ee2

jhamman approved these changes Jan 7, 2021
View reviewed changes

keewis approved these changes Jan 7, 2021
View reviewed changes

andersy005 removed the needs review label Jan 8, 2021

andersy005 added 2 commits January 10, 2021 09:01

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA
11fdc01

Remove the -OO flag for consistency
5c6c49b

keewis merged commit d241aa4 into pydata:masterJan 11, 2021

andersy005 deleted the ci/migrate-to-GHA branch January 11, 2021 12:24

max-sixty mentioned this pull request Jan 20, 2021
Move skip ci instructions to contributing guide #4829
Merged

andersy005 commented Feb 25, 2021
View reviewed changes

Uh oh!

Uh oh!

Conversation

andersy005 commented Dec 23, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keewis commented Dec 23, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keewis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhamman commented Dec 29, 2020

Uh oh!

jhamman commented Jan 7, 2021

Uh oh!

andersy005 commented Jan 7, 2021

Uh oh!

andersy005 commented Jan 7, 2021

Uh oh!

keewis left a comment

Choose a reason for hiding this comment

Uh oh!

keewisJan 7, 2021

Choose a reason for hiding this comment

Uh oh!

jhammanJan 8, 2021

Choose a reason for hiding this comment

Uh oh!

andersy005Jan 10, 2021

Choose a reason for hiding this comment

Uh oh!

keewisJan 11, 2021

Choose a reason for hiding this comment

Uh oh!

keewis commented Jan 7, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcherian commented Jan 7, 2021

Uh oh!

keewis commented Jan 11, 2021

Uh oh!

max-sixty commented Jan 19, 2021

Uh oh!

andersy005 commented Jan 20, 2021

Uh oh!

andersy005Feb 25, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andersy005 commented Dec 23, 2020•
edited
Loading

keewis commented Dec 23, 2020•
edited
Loading

keewis commented Jan 7, 2021•
edited
Loading