Skip to content

tools: automate icu update - #47727

Merged
nodejs-github-bot merged 9 commits into
nodejs:mainfrom
marco-ippolito:feat/automate-icu-update
May 9, 2023
Merged

tools: automate icu update#47727
nodejs-github-bot merged 9 commits into
nodejs:mainfrom
marco-ippolito:feat/automate-icu-update

Conversation

@marco-ippolito

Copy link
Copy Markdown
Member

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/actions

@nodejs-github-botnodejs-github-bot added meta Issues and PRs related to the general management of the project. tools Issues and PRs related to the tools directory. labels Apr 26, 2023
@marco-ippolito
marco-ippolitoforce-pushed the feat/automate-icu-update branch from 7261c4a to 64c627cCompareApril 26, 2023 14:40
@targos

Copy link
Copy Markdown
Member

You can remove the -small suffix from most places. We ship full ICU now.

@marco-ippolitomarco-ippolito changed the title tools: automate icu-small updatetools: automate icu updateApr 26, 2023
@marco-ippolito
marco-ippolitoforce-pushed the feat/automate-icu-update branch from 64c627c to 1eaaa62CompareApril 26, 2023 14:53
Comment threadtools/dep_updaters/update-icu.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should be validating this checksum matches the appropriate one from
https://github.com/unicode-org/icu/releases/download/release-${NEW_VERSION}/icu4c-${NEW_MAJOR_VERSION}_${NEW_MINOR_VERSION}-src.md5

(We've had mismatches before.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and even check the GPG key against KEYS

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we check the generated checksum and the one deposited match? if they dont match we skip?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be my recommendation.

Comment threadtools/dep_updaters/update-icu.sh Outdated
Comment threadtools/dep_updaters/update-icu.sh Outdated

@richardlaurichardlauApr 26, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll likely need to update ./test/fixtures/tz-version.txt to match the timezone version shipped in the new version of ICU.
Refs: #47456 (comment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and there are potential 'golden' value breakage, but not much to do about that.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont we have an action that does this? #47302

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marco-ippolito The timezone data gets baked into the ICU data file, which updating ICU will overwrite. If ./test/fixtures/tz-version.txt doesn't match the timezone data version in the ICU data file then the associated test will fail.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have managed to get the latest version from https://data.iana.org/time-zones/releases/ but its a bit hacky @srl295 do you know any api we could fetch the latest version?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what you should do is extract the tz version from the ICU you're updating to, and put that in tz-version.txt.

Maybe

$ node -p process.versions.tz | tee test/fixtures/tz-version.txt

And then in reviewing the PR make sure it's not a regression.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that will work because the update script added here doesn't rebuild Node.js, so that will output the tz version of whatever node binary is on the runner instead of the one from the data. I presume the tz version must be in the updated ICU files but maybe it'll be too complicated to extract without building Node.js with the new data?

I suppose we could ignore it and fix up manually if the opened PR breaks because of a different tz version in the new ICU data.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@richardlau it ought to build to make sure node works… the PR will do that, though. the tz version is deep in the gzipped binary .dat file.

Yes, it could be ignored and fixed manually . As i said it's likely there will be some manual tweaks as Node has been made more sensitive to these changes.

In that case, just don't modify tz-version.txt at all in this workflow.

@richardlau

Copy link
Copy Markdown
Member

cc FYI @srl295

@srl295srl295 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with issues noted

@srl295
srl295 self-requested a review April 26, 2023 15:23

@srl295srl295 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, just needs some tweaking

  • Also should update the GUIDE to mention this workflow.

Comment threadtools/dep_updaters/update-icu.sh Outdated
Comment threadtools/dep_updaters/update-icu.sh Outdated
Comment threadtools/dep_updaters/update-icu.sh Outdated
Comment threadtools/dep_updaters/update-icu.sh Outdated
srl295

This comment was marked as duplicate.

srl295

This comment was marked as duplicate.

srl295

This comment was marked as duplicate.

@srl295

srl295 commented Apr 26, 2023

Copy link
Copy Markdown
Member

Sorry, bad unicorn day, took me a couple tries to get through

image

@marco-ippolito
marco-ippolitoforce-pushed the feat/automate-icu-update branch 2 times, most recently from debe898 to 44862bfCompareApril 27, 2023 08:06
@srl295

Copy link
Copy Markdown
Member

You can remove the -small suffix from most places. We ship full ICU now.

except that the in-repo one still has an icu-small path IIRC

Comment threadtools/dep_updaters/update-icu.sh Outdated
Comment threadtools/dep_updaters/update-icu.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what you should do is extract the tz version from the ICU you're updating to, and put that in tz-version.txt.

Maybe

$ node -p process.versions.tz | tee test/fixtures/tz-version.txt

And then in reviewing the PR make sure it's not a regression.

@srl295

srl295 commented Apr 28, 2023

Copy link
Copy Markdown
Member

Two requests (to recap):

  • Don't assume ICU has only a major + minor version. Get the whole version number and process it.
  • Don't read the 'latest' time zone version from iana and require that. See deps: time zone updater workflow problems #47702
  • I suppose on that last point, IF there is a time zone update at the same time as a new ICU is available AND that zone update is actually newer than what's in ICU, then it might be worth folding in a TZ update at the same time. But it might not be worth engineering this in a single workflow.

@srl295

Copy link
Copy Markdown
Member

Also note that tz-version isn't the only thing that will break, there may be other test failures.

@marco-ippolito
marco-ippolito requested a review from srl295May 2, 2023 07:03
@marco-ippolito

Copy link
Copy Markdown
MemberAuthor

@srl295 should be good now

@marco-ippolitomarco-ippolito removed the commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. label May 9, 2023
@nodejs-github-botnodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels May 9, 2023
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator
Commit Queue failed
- Loading data for nodejs/node/pull/47727
✔ Done loading data for nodejs/node/pull/47727
----------------------------------- PR info ------------------------------------
Title tools: automate icu update (#47727)
Author Marco Ippolito (@marco-ippolito)
Branch marco-ippolito:feat/automate-icu-update -> nodejs:main
Labels meta, tools
Commits 9
- tools: automate icu-small update
- fix: replace - with .
- fix: md5 from source
- fix: comparing checksum
- fix: mention the script in the maintaining guide
- fix: lint
- fix: update timezone.txt
- fix: remove tz update
- fix: use version as whole
Committers 1
- Marco Ippolito PR-URL: https://github.com/nodejs/node/pull/47727
Refs: https://github.com/nodejs/security-wg/issues/828
Reviewed-By: Steven R Loomis Reviewed-By: Rafael Gonzaga ------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/47727
Refs: https://github.com/nodejs/security-wg/issues/828
Reviewed-By: Steven R Loomis Reviewed-By: Rafael Gonzaga --------------------------------------------------------------------------------
ℹ This PR was created on Wed, 26 Apr 2023 14:31:28 GMT
✔ Approvals: 2
✔ - Steven R Loomis (@srl295): https://github.com/nodejs/node/pull/47727#pullrequestreview-1418824208
✔ - Rafael Gonzaga (@RafaelGSS) (TSC): https://github.com/nodejs/node/pull/47727#pullrequestreview-1417161365
✔ Last GitHub CI successful
✘ No Jenkins CI runs detected
--------------------------------------------------------------------------------
✔ Aborted `git node land` session in /home/runner/work/node/node/.ncu
https://github.com/nodejs/node/actions/runs/4927566555

@marco-ippolitomarco-ippolito added request-ci Add this label to start a Jenkins CI on a PR. and removed commit-queue-failed An error occurred while landing this pull request using GitHub Actions. labels May 9, 2023
@github-actionsgithub-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label May 9, 2023
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@marco-ippolitomarco-ippolito added the commit-queue Add this label to land a pull request using GitHub Actions. label May 9, 2023
@nodejs-github-botnodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels May 9, 2023
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator
Commit Queue failed
- Loading data for nodejs/node/pull/47727
✔ Done loading data for nodejs/node/pull/47727
----------------------------------- PR info ------------------------------------
Title tools: automate icu update (#47727)
Author Marco Ippolito (@marco-ippolito)
Branch marco-ippolito:feat/automate-icu-update -> nodejs:main
Labels meta, tools
Commits 9
- tools: automate icu-small update
- fix: replace - with .
- fix: md5 from source
- fix: comparing checksum
- fix: mention the script in the maintaining guide
- fix: lint
- fix: update timezone.txt
- fix: remove tz update
- fix: use version as whole
Committers 1
- Marco Ippolito PR-URL: https://github.com/nodejs/node/pull/47727
Refs: https://github.com/nodejs/security-wg/issues/828
Reviewed-By: Steven R Loomis Reviewed-By: Rafael Gonzaga ------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/47727
Refs: https://github.com/nodejs/security-wg/issues/828
Reviewed-By: Steven R Loomis Reviewed-By: Rafael Gonzaga --------------------------------------------------------------------------------
ℹ This PR was created on Wed, 26 Apr 2023 14:31:28 GMT
✔ Approvals: 2
✔ - Steven R Loomis (@srl295): https://github.com/nodejs/node/pull/47727#pullrequestreview-1418824208
✔ - Rafael Gonzaga (@RafaelGSS) (TSC): https://github.com/nodejs/node/pull/47727#pullrequestreview-1417161365
✔ Last GitHub CI successful
ℹ Last Full PR CI on 2023-05-09T17:03:40Z: https://ci.nodejs.org/job/node-test-pull-request/51718/
- Querying data for job/node-test-pull-request/51718/
✔ Last Jenkins CI successful
--------------------------------------------------------------------------------
✔ No git cherry-pick in progress
✔ No git am in progress
✔ No git rebase in progress
--------------------------------------------------------------------------------
- Bringing origin/main up to date...
From https://github.com/nodejs/node
* branch main -> FETCH_HEAD
✔ origin/main is now up-to-date
- Downloading patch for 47727
From https://github.com/nodejs/node
* branch refs/pull/47727/merge -> FETCH_HEAD
✔ Fetched commits as 33231b0e8908..15ff4c572aa6
--------------------------------------------------------------------------------
Auto-merging .github/workflows/tools.yml
[main 4f311994aa] tools: automate icu-small update
Author: Marco Ippolito Date: Wed Apr 26 12:06:22 2023 +0200
3 files changed, 80 insertions(+)
create mode 100755 tools/dep_updaters/update-icu.sh
mode change 100644 => 100755 tools/icu/shrink-icu-src.py
[main da76c2a76b] fix: replace - with .
Author: Marco Ippolito Date: Wed Apr 26 18:03:27 2023 +0200
1 file changed, 4 insertions(+), 6 deletions(-)
[main bdff48f4f9] fix: md5 from source
Author: Marco Ippolito Date: Thu Apr 27 10:02:04 2023 +0200
1 file changed, 9 insertions(+), 4 deletions(-)
[main 6c92d489d5] fix: comparing checksum
Author: Marco Ippolito Date: Thu Apr 27 10:49:07 2023 +0200
1 file changed, 9 insertions(+)
[main 3301ee5878] fix: mention the script in the maintaining guide
Author: Marco Ippolito Date: Thu Apr 27 11:04:49 2023 +0200
1 file changed, 3 insertions(+)
[main 361a6a53cc] fix: lint
Author: Marco Ippolito Date: Thu Apr 27 12:05:29 2023 +0200
1 file changed, 1 insertion(+), 1 deletion(-)
[main 8dc41afd62] fix: update timezone.txt
Author: Marco Ippolito Date: Fri Apr 28 15:39:45 2023 +0200
1 file changed, 10 insertions(+)
[main 03215a2cd9] fix: remove tz update
Author: Marco Ippolito Date: Tue May 2 08:50:23 2023 +0200
1 file changed, 10 deletions(-)
[main ca71746872] fix: use version as whole
Author: Marco Ippolito Date: Tue May 2 09:02:24 2023 +0200
1 file changed, 5 insertions(+), 5 deletions(-)
✔ Patches applied
There are 9 commits in the PR. Attempting autorebase.
Rebasing (2/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
tools: automate icu-small update

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD 3e1deb57e5] tools: automate icu-small update
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Wed Apr 26 12:06:22 2023 +0200
3 files changed, 80 insertions(+)
create mode 100755 tools/dep_updaters/update-icu.sh
mode change 100644 => 100755 tools/icu/shrink-icu-src.py
Rebasing (3/18)
Rebasing (4/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: replace - with .

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD 536b8f1b98] fix: replace - with .
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Wed Apr 26 18:03:27 2023 +0200
1 file changed, 4 insertions(+), 6 deletions(-)
Rebasing (5/18)
Rebasing (6/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: md5 from source

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD efb870cbb4] fix: md5 from source
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Thu Apr 27 10:02:04 2023 +0200
1 file changed, 9 insertions(+), 4 deletions(-)
Rebasing (7/18)
Rebasing (8/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: comparing checksum

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD a6accc6f79] fix: comparing checksum
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Thu Apr 27 10:49:07 2023 +0200
1 file changed, 9 insertions(+)
Rebasing (9/18)
Rebasing (10/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: mention the script in the maintaining guide

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD 6e2396c965] fix: mention the script in the maintaining guide
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Thu Apr 27 11:04:49 2023 +0200
1 file changed, 3 insertions(+)
Rebasing (11/18)
Rebasing (12/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: lint

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD 199d159dfc] fix: lint
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Thu Apr 27 12:05:29 2023 +0200
1 file changed, 1 insertion(+), 1 deletion(-)
Rebasing (13/18)
Rebasing (14/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: update timezone.txt

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD e17d10a215] fix: update timezone.txt
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Fri Apr 28 15:39:45 2023 +0200
1 file changed, 10 insertions(+)
Rebasing (15/18)
Rebasing (16/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: remove tz update

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD a9a48e0f65] fix: remove tz update
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Tue May 2 08:50:23 2023 +0200
1 file changed, 10 deletions(-)
Rebasing (17/18)
Rebasing (18/18)

Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
fix: use version as whole

PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis srl295@gmail.com
Reviewed-By: Rafael Gonzaga rafael.nunu@hotmail.com

[detached HEAD 7a644b887e] fix: use version as whole
Author: Marco Ippolito marcoippolito54@gmail.com
Date: Tue May 2 09:02:24 2023 +0200
1 file changed, 5 insertions(+), 5 deletions(-)

Successfully rebased and updated refs/heads/main.

ℹ Add commit-queue-squash label to land the PR as one commit, or commit-queue-rebase to land as separate commits.

https://github.com/nodejs/node/actions/runs/4929182555

@marco-ippolitomarco-ippolito added commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. commit-queue Add this label to land a pull request using GitHub Actions. and removed commit-queue-failed An error occurred while landing this pull request using GitHub Actions. labels May 9, 2023
@nodejs-github-botnodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label May 9, 2023
@nodejs-github-bot
nodejs-github-bot merged commit 1b17793 into nodejs:mainMay 9, 2023
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Landed in 1b17793

targos pushed a commit that referenced this pull request May 12, 2023
PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis <srl295@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
@targostargos mentioned this pull request May 15, 2023
danielleadams pushed a commit that referenced this pull request Jul 6, 2023
PR-URL: #47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis <srl295@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
MoLow pushed a commit to MoLow/node that referenced this pull request Jul 6, 2023
PR-URL: nodejs#47727
Refs: nodejs/security-wg#828
Reviewed-By: Steven R Loomis <srl295@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commit-queue-squashAdd this label to instruct the Commit Queue to squash all the PR commits into the first one.metaIssues and PRs related to the general management of the project.toolsIssues and PRs related to the tools directory.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@marco-ippolito@nodejs-github-bot@targos@richardlau@srl295@RafaelGSS