Uh oh!
There was an error while loading. Please reload this page.
test(firefox): fixme flaky codegen pierceFrames disambiguation on Intel macOS - #42131
Merged
Simon Knott (Skn0tt) merged 3 commits intoAug 10, 2026
Merged
Conversation
…el macOS
Hi, I'm the Playwright bot. `cli codegen with pierceFrames › should prepend the frame selector to disambiguate` is flaky, but **only on `firefox-macos-15-large` (Intel macOS)** — 8 failures / 46 runs (17.4%) in the recent window, and zero failures on every other bot (chromium, webkit, firefox arm64, ubuntu). Reproduced locally on Intel macOS with `--repeat-each=15 --workers=6`: 3 flaky, all the same failure.
Root cause is a timing race, not a bad assertion: `buildFullSelectorForFrame` races the cross-frame piercing-selector resolution against a hardcoded 2000ms deadline and, on timeout, falls back to the `internal:control=no-pierce-frames` opt-out. Under load on the Intel macOS runner that race times out, so the recorder emits `page.pierceFrames({ pierce: false }).locator('#frame1')…` instead of the expected piercing selector. `test.slow()` wouldn't help — the 2s budget is internal to the recorder, independent of the test timeout. Same Intel-macOS-firefox-under-load class as #41811.
Scoped `test.fixme` to `browserName === 'firefox' && isMac && process.arch === 'x64'`; the test still runs everywhere else.
Run history for `firefox-macos-15-large` (oldest first, 46 runs):
[🟧](https://github.com/microsoft/playwright/actions/runs/30490403874/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30493031010/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30495285330/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30530696555/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30534756293/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30539645601/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30542293233/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30542345092/attempts/1)[🟧](https://github.com/microsoft/playwright/actions/runs/30559579795/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30565052100/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30568188818/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30568293053/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30568320872/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30577572373/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30581383958/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30581357288/attempts/1)[🟧](https://github.com/microsoft/playwright/actions/runs/30581405526/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30585190825/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30618799783/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30624512498/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30653773411/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30667286182/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30667347031/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30833251658/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30850656923/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30850749658/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30854798001/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30859315578/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30859305376/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30859360344/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885108613/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885140825/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885208556/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885237385/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30885274230/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30917613834/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30927071352/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30927719515/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30930057055/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30933859878/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30934926278/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30940141663/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30945227770/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30954253724/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30954381228/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30955628029/attempts/1)
<details>
<summary>What I verified locally (Intel macOS, x64)</summary>
- Reproduced: `npm run ftest -- tests/library/inspector/cli-codegen-3.spec.ts:1169 --repeat-each=15 --workers=6` -> 3 flaky with the `pierce: false` opt-out.
- After fix: firefox -> `1 skipped`; chromium (`npm run ctest -- …:1169`) -> `1 passed`.
- `npm run flint` -> clean.
- CI run: https://github.com/microsoft/playwright/actions/runs/30984018624
</details>
Suggested-reviewer: dgozman
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>Contributor
Test results for "tests 1"12 flaky51217 passed, 1190 skipped Merge workflow run. |
Contributor
Test results for "MCP"8050 passed, 1284 skipped Merge workflow run. |
Simon Knott (Skn0tt)
commented
Aug 5, 2026
Contributor
Copilot if the 2s race is the problem, maybe we can make those 2s configurable and extend them just under intel macOS tests? It seems like that would be the better fix. |
Co-authored-by: Skn0tt <14912729+Skn0tt@users.noreply.github.com>
Contributor
Implemented in |
| export async function buildFullSelectorForFrame(progress: Progress, frame: Frame, selector: string): Promise<string> { | ||
| const framePath = await generateFrameSelector(progress, frame); | ||
| export async function buildFullSelectorForFrame(progress: Progress, frame: Frame, selector: string, timeout = 2000): Promise<string> { | ||
| const framePath = await generateFrameSelector(progress, frame, timeout); |
Collaborator
There was a problem hiding this comment.
Let's instead check isUnderTest() and always use 10000 in this case?
Contributor
There was a problem hiding this comment.
Implemented in 7fdb420. Frame-selector generation now uses a 10s deadline whenever isUnderTest() is true, while production retains 2s.
Co-authored-by: Skn0tt <14912729+Skn0tt@users.noreply.github.com>
Dmitry Gozman (dgozman)
approved these changes
Aug 10, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, I'm the Playwright bot.
cli codegen with pierceFrames › should prepend the frame selector to disambiguateis flaky, but only onfirefox-macos-15-large(Intel macOS) — 8 failures / 46 runs (17.4%) in the recent window, and zero failures on every other bot (chromium, webkit, firefox arm64, ubuntu). Reproduced locally on Intel macOS with--repeat-each=15 --workers=6: 3 flaky, all the same failure.Root cause is a timing race, not a bad assertion:
buildFullSelectorForFrameraces the cross-frame piercing-selector resolution against a hardcoded 2000ms deadline and, on timeout, falls back to theinternal:control=no-pierce-framesopt-out. Under load on the Intel macOS runner that race times out, so the recorder emitspage.pierceFrames({ pierce: false }).locator('#frame1')…instead of the expected piercing selector.test.slow()wouldn't help — the 2s budget is internal to the recorder, independent of the test timeout. Same Intel-macOS-firefox-under-load class as #41811.Scoped
test.fixmetobrowserName === 'firefox' && isMac && process.arch === 'x64'; the test still runs everywhere else.Run history for
firefox-macos-15-large(oldest first, 46 runs):🟧🟥🟩🟩🟩🟩🟩🟩🟧🟩🟩🟩🟩🟩🟥🟩🟧🟩🟩🟩🟩🟩🟩🟩🟥🟩🟩🟩🟩🟥🟩🟩🟩🟩🟥🟩🟩🟩🟩🟥🟥🟥🟩🟩🟩🟩
What I verified locally (Intel macOS, x64)
npm run ftest -- tests/library/inspector/cli-codegen-3.spec.ts:1169 --repeat-each=15 --workers=6-> 3 flaky with thepierce: falseopt-out.1 skipped; chromium (npm run ctest -- …:1169) ->1 passed.npm run flint-> clean.Suggested-reviewer: dgozman
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com