sqlite: split up large test file by cjihrig · Pull Request #54014 · nodejs/node

cjihrig · 2024-07-24T04:37:24Z

The original test/parallel/test-sqlite.js test appears to time out in the CI occasionally. This commit splits the test into several smaller test files.

Fixes: #54006

nodejs-github-bot · 2024-07-24T08:32:56Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60578/

mcollina

lgtm

nodejs-github-bot · 2024-07-24T11:46:34Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60584/

cjihrig · 2024-07-24T12:40:00Z

Weird. Now Windows is failing because the tmp directory cleanup is failing with EPERM.

richardlau · 2024-07-24T17:22:12Z

Weird. Now Windows is failing because the tmp directory cleanup is failing with EPERM.

I'm a bit worried that #53617 may have introduced a subtle change in behaviour/regression.

nodejs-github-bot · 2024-07-24T19:46:43Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60600/

nodejs-github-bot · 2024-07-24T21:38:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60604/

nodejs-github-bot · 2024-07-24T23:06:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60609/

The current test is large and can time out. It should be split into multiple smaller tests as done in nodejs#54014. However, that approach appears to change GC behavior such that the database files are not cleaned up quickly enough on Windows. Forcing any unfinalized SQL statements to be GC'ed appears to fix the problem. Mark the original test as flaky until the necessary code changes are made.

cjihrig · 2024-07-25T03:36:45Z

The issue with these smaller tests appears to be related to relying on GC to finalize the prepared statements and close the database connections. Fixing it properly will take a little more time, so I have opened #54031 to mark the original test as flaky in the mean time.

The current test is large and can time out. It should be split into multiple smaller tests as done in #54014. However, that approach appears to change GC behavior such that the database files are not cleaned up quickly enough on Windows. Forcing any unfinalized SQL statements to be GC'ed appears to fix the problem. Mark the original test as flaky until the necessary code changes are made. PR-URL: #54031 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Stefan Stojanovic <stefan.stojanovic@janeasystems.com> Reviewed-By: Richard Lau <rlau@redhat.com>

cjihrig · 2024-08-06T13:38:08Z

There have been so many problems lately with EPERM on Windows that I'm not sure the failures on Windows were ever related to the changes in this PR. Starting the CI all over again to see how it goes.

nodejs-github-bot · 2024-08-06T13:46:08Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60908/

codecov · 2024-08-06T14:58:28Z

Codecov Report

Attention: Patch coverage is 73.58491% with 14 lines in your changes missing coverage. Please review.

Project coverage is 87.10%. Comparing base (e0634f5) to head (ee5b9c1).
Report is 459 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_sqlite.cc	73.58%	1 Missing and 13 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #54014 +/- ##
==========================================
- Coverage 87.10% 87.10% -0.01% 
==========================================
Files 647 647 Lines 181739 181775 +36 Branches 34887 34897 +10 ==========================================
+ Hits 158310 158337 +27 + Misses 16738 16736 -2 - Partials 6691 6702 +11

Files with missing lines	Coverage Δ
src/node_sqlite.h	`0.00% <ø> (ø)`
src/node_sqlite.cc	`82.34% <73.58%> (-0.38%)`	⬇️

... and 22 files with indirect coverage changes

nodejs-github-bot · 2024-08-07T14:03:14Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60937/

nodejs-github-bot · 2024-08-07T21:34:14Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60952/

cjihrig · 2024-08-07T21:35:09Z

I've made the changes I outlined in #54014 (comment). This is now more than just a test change. The CI seems happy (still resuming some unrelated failures). I would appreciate re-reviews.

nodejs-github-bot · 2024-08-08T04:48:15Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60962/ 🟡

nodejs-github-bot · 2024-08-12T13:48:31Z

Landed in 0301309...b8a2550

This commit adds statement tracking to the DatabaseSync class. When a database is closed manually or via garbage collection, it will force all associated prepared statements to be finalized. This should mitigate "zombie" connections which can introduce test flakiness in the CI on Windows. PR-URL: #54014Fixes: #54006 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

The original test/parallel/test-sqlite.js test appears to time out in the CI occasionally. This commit splits the test into several smaller test files. Fixes: #54006 PR-URL: #54014 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

PR-URL: #54014Fixes: #54006 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

This commit adds statement tracking to the DatabaseSync class. When a database is closed manually or via garbage collection, it will force all associated prepared statements to be finalized. This should mitigate "zombie" connections which can introduce test flakiness in the CI on Windows. PR-URL: #54014Fixes: #54006 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

The original test/parallel/test-sqlite.js test appears to time out in the CI occasionally. This commit splits the test into several smaller test files. Fixes: #54006 PR-URL: #54014 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

PR-URL: #54014Fixes: #54006 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>

The current test is large and can time out. It should be split into multiple smaller tests as done in #54014. However, that approach appears to change GC behavior such that the database files are not cleaned up quickly enough on Windows. Forcing any unfinalized SQL statements to be GC'ed appears to fix the problem. Mark the original test as flaky until the necessary code changes are made. PR-URL: #54031 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com> Reviewed-By: Stefan Stojanovic <stefan.stojanovic@janeasystems.com> Reviewed-By: Richard Lau <rlau@redhat.com>

nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Jul 24, 2024

MoLow approved these changes Jul 24, 2024
View reviewed changes

MoLow added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2024

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2024

tniessen approved these changes Jul 24, 2024
View reviewed changes

mcollina approved these changes Jul 24, 2024
View reviewed changes

anonrig approved these changes Jul 24, 2024
View reviewed changes

richardlau approved these changes Jul 24, 2024
View reviewed changes

cjihrig mentioned this pull request Jul 25, 2024
test: mark 'test/parallel/test-sqlite.js' as flaky #54031
Merged

cjihrig mentioned this pull request Jul 25, 2024
test_runner: remove redundant bootstrap boolean #54013
Merged

cjihrig force-pushed the split-up-sqlite-test branch from ab6834b to 31e8790Compare August 6, 2024 13:36

cjihrig added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 6, 2024

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 6, 2024

avivkeller added the sqlite Issues and PRs related to the SQLite subsystem. label Aug 6, 2024

test: unmark test-sqlite as flaky
ee5b9c1

cjihrig force-pushed the split-up-sqlite-test branch from 3ca154d to ee5b9c1Compare August 7, 2024 13:52

cjihrig added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 7, 2024

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 7, 2024

targos approved these changes Aug 8, 2024
View reviewed changes

cjihrig added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue Add this label to land a pull request using GitHub Actions. and removed needs-ci PRs that need a full CI run. labels Aug 12, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Aug 12, 2024

nodejs-github-bot closed this Aug 12, 2024

cjihrig deleted the split-up-sqlite-test branch August 12, 2024 13:53

RafaelGSS mentioned this pull request Aug 19, 2024
v22.7.0 proposal #54452
Merged

targos added the dont-land-on-v20.x label Sep 21, 2024

Uh oh!

Uh oh!

Conversation

cjihrig commented Jul 24, 2024

Uh oh!

nodejs-github-bot commented Jul 24, 2024

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 24, 2024

Uh oh!

cjihrig commented Jul 24, 2024

Uh oh!

richardlau commented Jul 24, 2024

Uh oh!

nodejs-github-bot commented Jul 24, 2024

Uh oh!

nodejs-github-bot commented Jul 24, 2024

Uh oh!

nodejs-github-bot commented Jul 24, 2024

Uh oh!

cjihrig commented Jul 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjihrig commented Aug 6, 2024

Uh oh!

nodejs-github-bot commented Aug 6, 2024

Uh oh!

codecovBot commented Aug 6, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nodejs-github-bot commented Aug 7, 2024

Uh oh!

nodejs-github-bot commented Aug 7, 2024

Uh oh!

cjihrig commented Aug 7, 2024

Uh oh!

nodejs-github-bot commented Aug 8, 2024• edited by cjihrig Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Aug 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cjihrig commented Jul 25, 2024•
edited
Loading

codecovBot commented Aug 6, 2024•
edited
Loading

nodejs-github-bot commented Aug 8, 2024•
edited by cjihrig
Loading