Skip to content

fix(test runner): match --test-list entries against the test file path - #42301

Closed
Yury Semikhatsky (yury-s) wants to merge 1 commit into
microsoft:mainfrom
yury-s:fix-42298
Closed

fix(test runner): match --test-list entries against the test file path#42301
Yury Semikhatsky (yury-s) wants to merge 1 commit into
microsoft:mainfrom
yury-s:fix-42298

Conversation

@yury-s

Copy link
Copy Markdown
Member

Summary

  • Tests declared in a file imported by the test file could not be selected by any --test-list entry: matching only considered test.location.file, while file-only entries and load-time file filtering use the test file path.
  • Match entries against either the test file or the declaration file.
  • Document that entries should use the test file path, which may differ from what --list prints for such tests.

Fixes#42298

Tests declared in a file imported by the test file could not be selected
by any --test-list entry: matching only considered test.location.file,
which points to the file where test() was called, while file-only
entries and load-time file filtering use the test file path.
Match entries against either the test file or the declaration file.
Fixes: microsoft#42298
@github-actions

Copy link
Copy Markdown
Contributor

Test results for "tests 1"

22 flaky⚠️ [installation tests] › playwright-should-work-with-relative-home-path.spec.ts:19 › playwright should work with relative home path `@package-installations-ubuntu-latest`
⚠️ [chromium-library] › library/global-fetch.spec.ts:293 › should return security details from response `@frozen-time-library-chromium-linux`
⚠️ [chromium-library] › library/har.spec.ts:639 › should have security details `@frozen-time-library-chromium-linux`
⚠️ [chromium-library] › library/global-fetch.spec.ts:293 › should return security details from response `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/har.spec.ts:639 › should have security details `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/video.spec.ts:736 › screencast › should work with video+trace `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/global-fetch.spec.ts:293 › should return security details from response `@realtime-time-library-chromium-linux`
⚠️ [chromium-library] › library/har.spec.ts:639 › should have security details `@realtime-time-library-chromium-linux`
⚠️ [chromium-library] › library/global-fetch.spec.ts:293 › should return security details from response `@chromium-ubuntu-22.04-node24`
⚠️ [chromium-library] › library/har.spec.ts:639 › should have security details `@chromium-ubuntu-22.04-node24`
⚠️ [chromium-library] › library/chromium/chromium.spec.ts:373 › should produce network events, routing, and annotations for Service Worker `@chromium-ubuntu-22.04-node20`
⚠️ [chromium-library] › library/global-fetch.spec.ts:293 › should return security details from response `@chromium-ubuntu-22.04-node20`
⚠️ [chromium-library] › library/har.spec.ts:639 › should have security details `@chromium-ubuntu-22.04-node20`
⚠️ [chromium-library] › library/chromium/oopif.spec.ts:179 › should take screenshot `@chromium-ubuntu-22.04-node22`
⚠️ [chromium-library] › library/global-fetch.spec.ts:293 › should return security details from response `@chromium-ubuntu-22.04-node22`
⚠️ [chromium-library] › library/har.spec.ts:639 › should have security details `@chromium-ubuntu-22.04-node22`
⚠️ [firefox-library] › library/global-fetch.spec.ts:293 › should return security details from response `@firefox-ubuntu-22.04-node20`
⚠️ [firefox-library] › library/har.spec.ts:639 › should have security details `@firefox-ubuntu-22.04-node20`
⚠️ [firefox-page] › page/page-emulate-media.spec.ts:144 › should keep reduced motion and color emulation after reload `@firefox-ubuntu-22.04-node20`
⚠️ [firefox-page] › page/page-goto.spec.ts:90 › should work with Cross-Origin-Opener-Policy `@firefox-ubuntu-22.04-node20`
⚠️ [webkit-library] › library/global-fetch.spec.ts:293 › should return security details from response `@webkit-ubuntu-22.04-node20`
⚠️ [webkit-library] › library/har.spec.ts:639 › should have security details `@webkit-ubuntu-22.04-node20`

51180 passed, 1233 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "MCP"

4 failed
❌ [chrome] › mcp/config.spec.ts:203 › chromiumSandbox › --sandbox enables the sandbox @mcp-ubuntu-latest-chrome
❌ [firefox] › mcp/cli-core.spec.ts:57 › click link @mcp-windows-latest-firefox
❌ [firefox] › mcp/cli-core.spec.ts:115 › hover @mcp-windows-latest-firefox
❌ [firefox] › mcp/cli-json.spec.ts:220 › request and per-part commands return JSON result @mcp-windows-latest-firefox

8106 passed, 1341 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Hi, I'm the Playwright bot and I took a first look at the CI failures here.

🟢 Failures are all pre-existing flakes — this PR is clear

The 4 failures are in the MCP suite; this PR only touches test-runner --test-list matching (loadUtils.ts), a doc, and a --test-list test. Nothing here exercises the MCP CLI, and each failing test has a red history on other SHAs/PRs.

Details

Overall: The "tests 1" report has no failures (22 flaky only). The "MCP" report has 4 failures, all in code this PR doesn't touch. Confirmed against the aggregated test-results DB.

Pre-existing flake / infra

  • [chrome] › mcp/config.spec.ts:203 › chromiumSandbox › --sandbox enables the sandbox — heavily bimodal: failed 8 of 15 runs, passed 7, rescued on retry 7 times, across 3 different PRs. Sandbox-availability flake, unrelated to --test-list.
  • [firefox] › mcp/cli-core.spec.ts:57 › click linkfailed 15 of 722 runs across 12 PRs, retry-rescued 8 times. Firefox-only; passes clean on chromium/webkit.
  • [firefox] › mcp/cli-core.spec.ts:115 › hoverfailed 3 of 722 runs across 3 PRs, retry-rescued once. Firefox-only.
  • [firefox] › mcp/cli-json.spec.ts:220 › request and per-part commands return JSON resultfailed 2 of 722 runs across 2 PRs, retry-rescued 3 times. Firefox-only.

All four flip verdict on SHAs this PR can't be responsible for, so none is caused by the change.

Triaged by the Playwright bot - agent run

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: --test-list cannot round-trip --list output when test declaration file differs from root loaded file

1 participant

@yury-s