build: try to speed up Github workflows by ktmud · Pull Request #12090 · apache/superset

ktmud · 2020-12-17T04:19:02Z

SUMMARY

CI has been slow recently. This PR is the latest effort in trying to mitigate:

Update the Cancel Previous Run job added in build: cancel previous github action runs #11940 to cancel future jobs as well.
Skip dependabot branches when possible (e.g. don't run Python jobs for npm updates).
Merge lighter actions (Prefer Typescript and License Check) into one workflow (there are overheads in starting a job).

Add a script to allow committers to manually cancel CI jobs in queue. This is useful on a busy day when the CI pipeline got clogged by continuous new commits to open PRs. Users have to configure GITHUB_TOKEN in their env variables in order to use this.

(superset) ➜ incubator-superset git:(manage-github-workflow) ✗ ./scripts/cancel_github_workflows.py --help Usage: cancel_github_workflows.py [OPTIONS] BRANCH_OR_PULL
Cancel running or queued GitHub workflows by branch or pull request ID.
By default jobs that are already running or are the latest of the branch
or PR will not be cancelled.
Options:
--repo TEXT Default is apache/incubator-superset
--event [pull_request|push|issue]
--include-last / --no-include-last
Whether to also cancel the lastest run.
--include-running / --no-include-running
Whether to also cancel running workflows
--help Show this message and exit.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Test locally and CI

ADDITIONAL INFORMATION

Has associated issue: this is a rerun of build: try to speed up Github workflows #12089 because it was blocked by CI
Changes UI
Requires DB Migration.
Confirm DB Migration upgrade and downgrade tested.
Introduces new feature or API
Removes existing feature or API

codecov-io · 2020-12-17T04:33:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.22%. Comparing base (eb3c2b2) to head (85993fd).
⚠️ Report is 10942 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (eb3c2b2) and HEAD (85993fd). Click for more details.
HEAD has 7 uploads less than BASE
Flag BASE (eb3c2b2) HEAD (85993fd)
python 10 4
javascript 1 0

Additional details and impacted files

@@ Coverage Diff @@## master #12090 +/- ##
==========================================
- Coverage 67.14% 59.22% -7.93% 
==========================================
Files 1002 951 -51 Lines 49300 46658 -2642 Branches 5010 4312 -698 ==========================================
- Hits 33103 27633 -5470 - Misses 16072 19025 +2953 + Partials 125 0 -125

Flag	Coverage Δ
cypress	`51.44% <ø> (+<0.01%)`	⬆️
javascript	`?`
python	`63.59% <ø> (-0.57%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add a script to cancel previous GitHub workflows and optimize the workflows in hope to speed up CI. - Skip dependabot for python jobs - Merge Prefer TypeScript and License Check - Cancel all tests in one workflow

nytai · 2021-01-06T21:07:31Z

+ continue-on-error: true
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ with:


This check is essentially useless since the comments will never work for PRs coming from forks (due to the GITHUB_TOKEN access). This would be a good candidate to port over to the workflow_run approach mentioned in https://securitylab.github.com/research/github-actions-preventing-pwn-requests. Obviously, out of scope for this PR

villebro

A few non-blocking comments, LGTM.

villebro · 2021-01-07T08:05:28Z

+from typing import Iterable, List, Optional, Union
+
+import click
+import requests


Whenever possible, we should probably try to use urllib3 over requests to avoid licensing issues.

Isn't it using Apache 2.0 License as well?

Oh wow, they've apparently relicensed. There was a refactor effort some time ago to remove explicit dependency on requests as it wasn't Apache compatible back then (see #7643). Good to know.

Hmm, I can't find any reference of it ever having been anything but Apache 2.0. Apparently I'm misremembering something or this has been a major misunderstanding.

I couldn't let this one go.. 😄 I read old mailing list archives and turned up this: https://lists.apache.org/thread.html/49c4d03d3aadab5c2d1c20ccf1cb7ca02edbc94961e6429dbf6a082b%40%3Cdev.superset.apache.org%3E Turns out requests was pulling in chardet which is LGPL, which was the problem. Turns out chardet was dropped as of 09/2020: encode/httpx#1269 🎉

So it means it's OK to use requests again, I guess?

Uhh, I went down a link in this issue without noticing that it was pointing to another project: psf/requests#4848 Apparently requests still uses chardet, and there are no plans to remove it. As long as this is the case we're not allowed to reference requests as a required dependency, but it's ok to be had as an optional dependency. As long as we restrict the usage to development scripts we're fine, but we shouldn't use it in the core code.

Got it. Thanks for the clarification and investigation!

villebro · 2021-01-07T08:11:00Z

@@ -0,0 +1,141 @@
+# Python MySQL unit tests


I'm mostly indifferent to having these split up versus all in one file, but I'd like to avoid shuffling these around unnecessarily to avoid losing the git history.

There was a hypothesis that there are overheads involved in scheduling multiple workflows vs running them in one. Not sure how much it helps but I still think it makes sense to combine these three considering they are very similar in nature and not particularly expensive to run.

pull-request-sizeBot added the size/XL label Dec 17, 2020

ktmud force-pushed the manage-github-workflow branch from 16a2754 to c6ddbbfCompare December 21, 2020 07:47

ktmud force-pushed the manage-github-workflow branch 2 times, most recently from 03494b0 to 1c4aba0Compare January 4, 2021 18:36

pull-request-sizeBot added size/L and removed size/XL labels Jan 4, 2021

ktmud force-pushed the manage-github-workflow branch from 1c4aba0 to 3ca6ea6Compare January 5, 2021 16:51

pull-request-sizeBot added size/XL and removed size/L labels Jan 5, 2021

ktmud force-pushed the manage-github-workflow branch from e8faa48 to 98e3922Compare January 5, 2021 22:35

ktmud closed this Jan 5, 2021

ktmud reopened this Jan 5, 2021

ktmud force-pushed the manage-github-workflow branch 3 times, most recently from 35c8d68 to 29df9f4Compare January 6, 2021 00:06

ktmud added 6 commits January 6, 2021 12:54

build: another approach to cancel Github workflows
556b52e
Add a script to cancel previous GitHub workflows and optimize the workflows in hope to speed up CI. - Skip dependabot for python jobs - Merge Prefer TypeScript and License Check - Cancel all tests in one workflow

Display number of jobs to cancel
0622739

build: ignore github actions from dependabot
be3098b

Merge mysql postgres and sqlite tests
abf1d21

Don't use pull_request_target
6434014

Dont ignore pull request checks
85993fd

ktmud force-pushed the manage-github-workflow branch from 29df9f4 to 85993fdCompare January 6, 2021 20:54

nytai approved these changes Jan 6, 2021
View reviewed changes

nytai reviewed Jan 6, 2021
View reviewed changes

villebro approved these changes Jan 7, 2021
View reviewed changes

ktmud merged commit f482849 into apache:masterJan 7, 2021

ktmud deleted the manage-github-workflow branch January 7, 2021 08:47

ktmud mentioned this pull request Jan 10, 2021
build: add cancel-workflow-runs action locally #12394
Merged
6 tasks

ktmud mentioned this pull request Jan 20, 2021
fix(CI): properly configure cancel duplicates #12625
Merged
6 tasks

ktmud mentioned this pull request Jan 21, 2021
fix(CI): remove duplicate hive and presto tests #12638
Merged
6 tasks

sebastianliebscher mentioned this pull request Sep 9, 2023
build: use GitHub Actions concurrency groups to cancel workflows #25243
Closed
9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.0.0 First shipped in 1.0.0 labels Mar 12, 2024

qfcwell pushed a commit to qfcwell/superset that referenced this pull request May 12, 2026
build: try to speed up Github workflows (apache#12090)
7a60607

Navigation Menu

Uh oh!

build: try to speed up Github workflows - #12090

build: try to speed up Github workflows#12090
ktmud merged 6 commits into
apache:masterfrom
ktmud:manage-github-workflow

ktmud commented Dec 17, 2020•
edited
Loading

Uh oh!

codecov-io commented Dec 17, 2020•
edited by codecov Bot
Loading

Uh oh!

nytaiJan 6, 2021

Uh oh!

villebro left a comment

Uh oh!

villebroJan 7, 2021

Uh oh!

ktmudJan 7, 2021

Uh oh!

villebroJan 7, 2021

Uh oh!

villebroJan 7, 2021

Uh oh!

villebroJan 9, 2021

Uh oh!

ktmudJan 9, 2021

Uh oh!

villebroJan 10, 2021

Uh oh!

ktmudJan 10, 2021

Uh oh!

villebroJan 7, 2021

Uh oh!

ktmudJan 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

ktmud commented Dec 17, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

Uh oh!

codecov-io commented Dec 17, 2020• edited by codecov BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ktmud commented Dec 17, 2020•
edited
Loading

codecov-io commented Dec 17, 2020•
edited by codecov Bot
Loading