Skip to content

Cache GMT remote data as artifacts on Github - #530

Merged
weiji14 merged 14 commits into
masterfrom
cache-artifact
Jul 16, 2020
Merged

Cache GMT remote data as artifacts on Github#530
weiji14 merged 14 commits into
masterfrom
cache-artifact

Conversation

@weiji14

@weiji14weiji14 commented Jul 16, 2020

Copy link
Copy Markdown
Member

Description of proposed changes

Use GMT on macOS to download the GMT remote files from the GMT data server, and then upload them as artifacts on Github.

Separates the 'cache' task from our Tests CI. Meaning we can update our cache independently of our Continuous Integration tests, allowing the cache to be updated every Sunday for example without needing to manually trigger it with a PR.

References:

Fixes#523, Closes#527

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If adding new functionality, add an example to docstrings or tutorials.

Use GMT on macOS to download the GMT remote files from the GMT data server, and then upload them as artifacts on Github.
@weiji14weiji14 added the maintenance Boring but important stuff for the core devs label Jul 16, 2020
@weiji14weiji14 changed the title WIP: Cache GMT remote data as artifacts on GithubCache GMT remote data as artifacts on GithubJul 16, 2020
@weiji14
weiji14 marked this pull request as ready for review July 16, 2020 03:07
Comment thread.github/workflows/ci_tests.yaml Outdated
github_token: ${{secrets.GITHUB_TOKEN}}
workflow: cache_data.yaml
name: gmt-cache
path: ~/.gmt

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the path correct? Downloading artifacts is fast, but it may not be placed in the correct directories. GMT still tries to download all files during testing.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it looks like it's unzipped to the right location (e.g. ~/.gmt/server/earth/earth_relief/earth_relief_01d_g.grd), and that's what wget was doing before?? Maybe we should use $HOME or something?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can add a step listing all files in the ~/.gmt directory after downloading the artifacts.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, $HOME doesn't work either. Then I'm not sure where the action is actually unzipping the files to!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've some tests to do, and will make changes to this branch.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, go for it 👍

@weiji14weiji14Jul 16, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that a ~ folder is created 🤦. How about trying ./.gmt, since the action uses ./ as the default, see https://github.com/dawidd6/action-download-artifact/blob/2f29da2bb95f5580767170955fca0cf48714d48b/action.yml#L31. Also, we should use ls -alh to see hidden dot files.

Edit: What am I saying, I should just use .gmt!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw the .gmt directory in the root directory. At least we can download the artifact, and then move it to ~/.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's one way of doing it.

@vercel
vercelBot temporarily deployed to Preview July 16, 2020 03:40 Inactive
Comment thread.github/workflows/ci_tests.yaml Outdated
So we don't need to set a Github token
Comment thread.github/workflows/ci_tests.yaml
Comment thread.github/workflows/cache_data.yaml Outdated
@seismanseisman added this to the 0.2.x milestone Jul 16, 2020
Co-Authored-By: Dongdong Tian <seisman.info@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenanceBoring but important stuff for the core devs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The GitHub Actions caches don't work for PRs

2 participants

@weiji14@seisman