Skip to content

test: add tests for REPL custom evals (second attempt) - #57850

Merged
nodejs-github-bot merged 1 commit into
nodejs:mainfrom
dario-piotrowicz:dario/test/repl-custom-eval-2
Jun 15, 2025
Merged

test: add tests for REPL custom evals (second attempt)#57850
nodejs-github-bot merged 1 commit into
nodejs:mainfrom
dario-piotrowicz:dario/test/repl-custom-eval-2

Conversation

@dario-piotrowicz

@dario-piotrowiczdario-piotrowicz commented Apr 12, 2025

Copy link
Copy Markdown
Member

this PR reintroduces the REPL custom eval tests that have been introduced in #57691 but reverted in #57793

the tests were working fine locally but caused failures in CI (like this one) the problem, I believe, being that the getReplOutput would return the output value right away even though the repl processing is asynchronous. Some small delay there is, I imagine, present in the CI runs and that's what I think was causing the issue.

I addressed this problem by returning a promise that resolves to the repl's output using the repl prompt as the signal to understand when the input processing has concluded, instead of returning the output right away.

@nodejs-github-botnodejs-github-bot added needs-ci PRs that need a full CI run. repl Issues and PRs related to the REPL subsystem. labels Apr 12, 2025
@dario-piotrowicz

Copy link
Copy Markdown
MemberAuthor

cc. @tniessen, @lpinca 🙂 (sorry again for my earlier mistake on this change 🙇)

@dario-piotrowicz
dario-piotrowiczforce-pushed the dario/test/repl-custom-eval-2 branch from 8c394e5 to 9413135CompareApril 12, 2025 17:12
@dario-piotrowiczdario-piotrowicz changed the title test: add tests for REPL custom evalstest: add tests for REPL custom evals (second attempt)Apr 12, 2025
@dario-piotrowicz
dario-piotrowiczforce-pushed the dario/test/repl-custom-eval-2 branch from 9413135 to 1e33c67CompareApril 12, 2025 17:14
@codecov

codecovBot commented Apr 12, 2025

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.16%. Comparing base (58e1cba) to head (a5ef49b).
Report is 34 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #57850 +/- ##
==========================================
- Coverage 90.17% 90.16% -0.01% 
==========================================
Files 636 637 +1 Lines 188028 188122 +94 Branches 36895 36908 +13 ==========================================
+ Hits 169547 169622 +75 - Misses 11233 11237 +4 - Partials 7248 7263 +15 
Files with missing linesCoverage Δ
lib/repl.js95.12% <ø> (-0.01%)⬇️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BridgeARBridgeAR left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BridgeARBridgeAR added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 15, 2025
@github-actionsgithub-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 15, 2025
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@lpinca

Copy link
Copy Markdown
Member

There are still failures. Here is one in the latest CI run: https://ci.nodejs.org/job/node-test-commit-linux-containered/50117/nodes=ubuntu2204_sharedlibs_withoutssl_x64/testReport/(root)/parallel/test_repl_custom_eval/

---
duration_ms: 826.053
exitcode: 1
severity: fail
stack: |-
> > > > Test failure: 'does show previews if `preview` is set to `true`'
Location: test/parallel/test-repl-custom-eval.js:146:3
AssertionError [ERR_ASSERTION]: The input did not match the regular expression /'Hello custom' \+ ' eval World!'\n\/\/ 'Hello custom eval World!'/. Input:
"'Hello custom' + ' eval World!'"
at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-repl-custom-eval.js:159:12)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async Test.run (node:internal/test_runner/test:1069:7)
at async Promise.all (index 7)
at async Suite.run (node:internal/test_runner/test:1461:7)
at async startSubtestAfterBootstrap (node:internal/test_runner/harness:308:3) {
generatedMessage: true,
code: 'ERR_ASSERTION',
actual: "'Hello custom' + ' eval World!'",
expected: /'Hello custom' \+ ' eval World!'\n\/\/ 'Hello custom eval World!'/,
operator: 'match'
}
...

@dario-piotrowicz

Copy link
Copy Markdown
MemberAuthor

oof... 😓

sorry about that, thanks @lpinca, I'll have a look 🙏

@dario-piotrowicz
dario-piotrowiczforce-pushed the dario/test/repl-custom-eval-2 branch from 1e33c67 to 584b6d9CompareApril 18, 2025 08:44
@dario-piotrowicz

Copy link
Copy Markdown
MemberAuthor

@lpinca I couldn't really reproduce the issue locally but I refactored the code to make it more robust regarding timing, I hope that this will fix the CI check, could you re-run the tests? 🙏

@lpincalpinca added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 18, 2025
@github-actionsgithub-actionsBot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Apr 18, 2025
@github-actions

Copy link
Copy Markdown
Contributor
Failed to start CI
 ⚠ Commits were pushed since the last approving review:
⚠ - test: add tests for REPL custom evals
✘ Refusing to run CI on potentially unsafe PR
https://github.com/nodejs/node/actions/runs/14532544242

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@lpincalpinca removed the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Apr 18, 2025
@dario-piotrowiczdario-piotrowicz added the request-ci Add this label to start a Jenkins CI on a PR. label May 5, 2025
@github-actionsgithub-actionsBot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels May 5, 2025
@github-actions

Copy link
Copy Markdown
Contributor
Failed to start CI
 ⚠ Commits were pushed since the last approving review:
⚠ - test: add tests for REPL custom evals
✘ Refusing to run CI on potentially unsafe PR
https://github.com/nodejs/node/actions/runs/14833744376

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@dario-piotrowicz
dario-piotrowiczforce-pushed the dario/test/repl-custom-eval-2 branch from 584b6d9 to 8196294CompareMay 11, 2025 20:19
@dario-piotrowicz
dario-piotrowiczforce-pushed the dario/test/repl-custom-eval-2 branch from 9bd4725 to a5ef49bCompareJune 11, 2025 11:51
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@dario-piotrowicz
dario-piotrowicz marked this pull request as ready for review June 11, 2025 16:24
@dario-piotrowicz

Copy link
Copy Markdown
MemberAuthor

@jasnell, @BridgeAR and @gurgunday I've significantly changed the code since your last approval (for the better, the tests now are very much faster and more reliable than my previous iterations), could you please give this one a re-review when you get the chance? 🙏

@dario-piotrowiczdario-piotrowicz removed the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Jun 11, 2025
@dario-piotrowiczdario-piotrowicz added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. request-ci Add this label to start a Jenkins CI on a PR. labels Jun 13, 2025
@github-actionsgithub-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jun 15, 2025
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@dario-piotrowiczdario-piotrowicz added the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 15, 2025
@nodejs-github-botnodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 15, 2025
@nodejs-github-bot
nodejs-github-bot merged commit d6dade5 into nodejs:mainJun 15, 2025
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Landed in d6dade5

@dario-piotrowicz
dario-piotrowicz deleted the dario/test/repl-custom-eval-2 branch June 15, 2025 16:36
targos pushed a commit that referenced this pull request Jun 16, 2025
this commit reintroduces the REPL custom eval tests that have
been introduced in #57691
but reverted in #57793
the tests turned out problematic before because `getReplOutput`,
the function used to return the repl output wasn't taking into
account that input processing and output emitting are asynchronous
operation can resolve with a small delay
the new implementation here replaces `getReplOutput` with
`getReplRunOutput` that resolves repl inputs by running them
and using the repl prompt as an indicator to when the input
processing has completed
PR-URL: #57850
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
aduh95 pushed a commit that referenced this pull request Jul 21, 2025
this commit reintroduces the REPL custom eval tests that have
been introduced in #57691
but reverted in #57793
the tests turned out problematic before because `getReplOutput`,
the function used to return the repl output wasn't taking into
account that input processing and output emitting are asynchronous
operation can resolve with a small delay
the new implementation here replaces `getReplOutput` with
`getReplRunOutput` that resolves repl inputs by running them
and using the repl prompt as an indicator to when the input
processing has completed
PR-URL: #57850
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
aduh95 pushed a commit that referenced this pull request Jul 24, 2025
this commit reintroduces the REPL custom eval tests that have
been introduced in #57691
but reverted in #57793
the tests turned out problematic before because `getReplOutput`,
the function used to return the repl output wasn't taking into
account that input processing and output emitting are asynchronous
operation can resolve with a small delay
the new implementation here replaces `getReplOutput` with
`getReplRunOutput` that resolves repl inputs by running them
and using the repl prompt as an indicator to when the input
processing has completed
PR-URL: #57850
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs that have at least one approval, no pending requests for changes, and a CI started.needs-ciPRs that need a full CI run.replIssues and PRs related to the REPL subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@dario-piotrowicz@nodejs-github-bot@lpinca@jasnell@BridgeAR@gurgunday