Use mamba to install Continuous Integration dependencies by weiji14 · Pull Request #841 · GenericMappingTools/pygmt

weiji14 · 2021-02-06T02:05:19Z

Description of proposed changes

To cut down on our Continuous Integration's installation time (~9 min on Windows), let's replace conda with a more efficient installer called mamba! Specifically using mambaforge which is a faster miniconda replacement that defaults to using the conda-forge channel and mamba instead of conda by default.

Continuation of #813 which sped up our Continuous Documentation builds.

Update September 2021: Below shows comparison on Windows - Python 3.9 with a 5min speedup (18m44s -> 13m42s), this is without caching.

Old using `conda` (18m 44s)	New using `mamba` (13m 42s)

More benchmarks:

Build	Old time	New time	Diff
ubuntu-latest - Python 3.7 / NumPy 1.18	10m27s	9m8s	-1m19s
macOS-latest - Python 3.7 / NumPy 1.18	14m42s	10m11s	-4m31s
windows-latest - Python 3.7 / NumPy 1.18	17m12s	13m0s	-4m12s
ubuntu-latest - Python 3.9 / NumPy 1.21	13m22s	7m32s	-5m50s
macOS-latest - Python 3.9 / NumPy 1.21	17m4s	13m15s	-3m49s
windows-latest - Python 3.9 / NumPy 1.21	19m12s	14m41s	-4m41s

References:

https://github.com/conda-incubator/setup-miniconda/tree/2fc91a2ef82dfb50ded2d80e56f83b7039ba5955#example-10-miniforge

Addresses #584

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

/format: automatically format and lint the code
/test-gmt-dev: run full tests on the latest GMT development version

Specifically using `mambaforge` which is a `miniconda` replacement that defaults to using the `conda-forge` channel and `mamba` instead of `conda` by default.

Delete the requirements-dev.txt file, and update the ci_test.yml to install GMT dependencies while setting up mamba.

seisman · 2021-02-19T00:46:16Z

Should we try the cache action?

weiji14 · 2021-02-19T00:57:07Z

Should we try the cache action?

Sure, feel free to try it out in this branch. I'll be heading out for most of the day.

seisman · 2021-02-25T07:34:19Z

As you can see, caching (bf1e97d) doesn't help.

See PR #932 for another test. It's clear that fewer packages mean faster speed.

This reverts commit bf1e97d.

defaults to see if it speeds up resolver. See also https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#strict-channel-priority

weiji14

As you can see, caching (bf1e97d) doesn't help.
See PR #932 for another test. It's clear that fewer packages mean faster speed.

I've reverted the caching step, and configured setup-miniconda to use strict channel priority (conda-forge only, with no packages from default anaconda channel) which speeds up the conda/mamba resolver step (see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#strict-channel-priority). See if this looks ok @seisman.

seisman · 2021-09-17T12:51:40Z

It looks great. Maybe also update ci_docs.yml, cache_data.yml and even vercel's package.json file.

weiji14 · 2021-09-17T21:30:13Z

It looks great. Maybe also update ci_docs.yml, cache_data.yml and even vercel's package.json file.

Ok, updated ci_docs.yml and cache_data.yml to use mamba! I won't update the vercel package.json file because using micromamba didn't seem to give a noticeable speedup last time we tried (ref #813 (review)).

weiji14

Thanks @seisman, I'll merge this PR on second approval or in ~12 hours, and that should give us some good CI time savings!

seisman · 2021-09-18T05:58:38Z

Don't forget to remove the python-version: 3.7 line before merging.

This reverts commit 4b93f4d.

…ingTools#841) Specifically using `mambaforge` which is a `miniconda` replacement that defaults to using the `conda-forge` channel and `mamba` instead of `conda` by default. * Set strict channel priority to conda-forge only with no defaults to see if it speeds up resolver. See also https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#strict-channel-priority * Use mamba on GMT Dev Tests CI build * Use mamba in CI docs and Cache Data workflows * Pin gmt dev version to 6.3 on Windows CI Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

Use mamba to install Continuous Integration dependencies
79cd804
Specifically using `mambaforge` which is a `miniconda` replacement that defaults to using the `conda-forge` channel and `mamba` instead of `conda` by default.

weiji14 added the maintenance Boring but important stuff for the core devs label Feb 6, 2021

weiji14 changed the title ~~Use mamba to install Continuous Integration dependencies~~WIP: Use mamba to install Continuous Integration dependenciesFeb 6, 2021

vercelBottemporarily deployed to Preview February 6, 2021 02:45 Inactive

vercelBottemporarily deployed to Preview February 6, 2021 03:15 Inactive

weiji14 force-pushed the mambaforge branch from dc020e8 to 650436dCompare February 6, 2021 03:24

vercelBottemporarily deployed to Preview February 6, 2021 03:24 Inactive

Move dev dependency specification to environment.yml
5612a68
Delete the requirements-dev.txt file, and update the ci_test.yml to install GMT dependencies while setting up mamba.

weiji14 force-pushed the mambaforge branch from 650436d to 5612a68Compare February 6, 2021 03:49

vercelBottemporarily deployed to Preview February 6, 2021 03:49 Inactive

weiji14 force-pushed the master branch from 1730198 to 65b099aCompare February 15, 2021 12:06

weiji14 added 2 commits February 18, 2021 14:41

Merge branch 'master' into mambaforge
2dd0890

Temporarily enable Windows Python 3.9 CI tests in draft mode
4b93f4d

vercelBottemporarily deployed to Preview February 18, 2021 01:44 Inactive

Use the cache action
bf1e97d

vercelBottemporarily deployed to Preview February 19, 2021 01:32 Inactive

seismanand others added 2 commits September 17, 2021 22:51

Revert "Use the cache action"
70f5921
This reverts commit bf1e97d.

Merge branch 'main' into mambaforge
1595cb1

vercelBottemporarily deployed to Preview September 17, 2021 11:18 Inactive

Set strict channel priority to conda-forge only with no
7543a26
defaults to see if it speeds up resolver. See also https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#strict-channel-priority

weiji14 force-pushed the mambaforge branch from c8f2fb9 to 7543a26Compare September 17, 2021 11:21

vercelBottemporarily deployed to Preview September 17, 2021 11:27 Inactive

weiji14 changed the title ~~WIP: Use mamba to install Continuous Integration dependencies~~Use mamba to install Continuous Integration dependenciesSep 17, 2021

Use mamba on GMT Dev Tests CI build
e2dbf0c

vercelBottemporarily deployed to Preview September 17, 2021 11:43 Inactive

weiji14 marked this pull request as ready for review September 17, 2021 11:44

weiji14 commented Sep 17, 2021
View reviewed changes

Comment thread.github/workflows/ci_tests.yaml Outdated

Use mamba in CI docs and Cache Data workflows
b962733

vercelBottemporarily deployed to Preview September 17, 2021 21:28 Inactive

weiji14 self-assigned this Sep 17, 2021

weiji14 added this to the 0.5.0 milestone Sep 17, 2021

weiji14 mentioned this pull request Sep 17, 2021
Make test suite more efficient to run in 10 minutes #584
Closed
4 tasks

seisman approved these changes Sep 17, 2021
View reviewed changes

weiji14 added the final review call This PR requires final review and approval from a second reviewer label Sep 18, 2021

weiji14 commented Sep 18, 2021
View reviewed changes

Comment thread.github/workflows/ci_tests_dev.yaml Outdated

Pin gmt dev version to 6.3 on Windows CI
deeafad

vercelBottemporarily deployed to Preview September 18, 2021 01:03 Inactive

weiji14 added 2 commits September 18, 2021 22:17

Merge branch 'main' into mambaforge
56e9433

Revert "Temporarily enable Windows Python 3.9 CI tests in draft mode"
504fadc
This reverts commit 4b93f4d.

vercelBottemporarily deployed to Preview September 18, 2021 10:20 Inactive

weiji14 merged commit 8409764 into mainSep 18, 2021

weiji14 deleted the mambaforge branch September 18, 2021 10:41

seisman removed the final review call This PR requires final review and approval from a second reviewer label Sep 18, 2021

weiji14 mentioned this pull request Dec 5, 2021
Unpin GMT dev version from conda-forge/label/dev on Windows #1645
Merged
6 tasks

Navigation Menu

Uh oh!

Use mamba to install Continuous Integration dependencies - #841

Use mamba to install Continuous Integration dependencies#841
weiji14 merged 13 commits into
mainfrom
mambaforge

weiji14 commented Feb 6, 2021•
edited
Loading

Uh oh!

seisman commented Feb 19, 2021

Uh oh!

weiji14 commented Feb 19, 2021

Uh oh!

seisman commented Feb 25, 2021

Uh oh!

weiji14 left a comment

Uh oh!

Uh oh!

seisman commented Sep 17, 2021

Uh oh!

weiji14 commented Sep 17, 2021

Uh oh!

weiji14 left a comment

Uh oh!

Uh oh!

seisman commented Sep 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

weiji14 commented Feb 6, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seisman commented Feb 19, 2021

Uh oh!

weiji14 commented Feb 19, 2021

Uh oh!

seisman commented Feb 25, 2021

Uh oh!

weiji14 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seisman commented Sep 17, 2021

Uh oh!

weiji14 commented Sep 17, 2021

Uh oh!

weiji14 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seisman commented Sep 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

weiji14 commented Feb 6, 2021•
edited
Loading