You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
5 failed
❌ [firefox-library] › library/har.spec.ts:629 › should have connection details @firefox-ubuntu-22.04-node20
❌ [firefox-library] › library/har.spec.ts:639 › should have security details @firefox-ubuntu-22.04-node20
❌ [firefox-library] › library/har.spec.ts:664 › should have connection details for redirects @firefox-ubuntu-22.04-node20
❌ [firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests @firefox-ubuntu-22.04-node20
❌ [firefox-library] › library/har.spec.ts:699 › should return server address directly from response @firefox-ubuntu-22.04-node20
7 flaky⚠️ [chromium-library] › library/chromium/chromium.spec.ts:301 › should report intercepted service worker requests in HAR `@frozen-time-library-chromium-linux` ⚠️ [chromium-library] › library/video.spec.ts:495 › screencast › should capture static page in persistent context Radoslav Kirilov (@smoke) `@realtime-time-library-chromium-linux` ⚠️ [chromium-library] › library/video.spec.ts:736 › screencast › should work with video+trace `@chromium-ubuntu-22.04-node20` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@firefox-ubuntu-22.04-node20` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-ubuntu-22.04-node20` ⚠️ [firefox-library] › library/heap.spec.ts:223 › should not leak workers `@firefox-ubuntu-22.04-node20` ⚠️ [firefox-page] › page/page-goto.spec.ts:90 › should work with Cross-Origin-Opener-Policy `@firefox-ubuntu-22.04-node20`
🔴 The Firefox roll breaks 5 HAR serverIPAddress tests
Hi, I'm the Playwright bot — I took a first look at the failing CI.
This is a real regression from the roll, not flake. Firefox r1541 now reports the HAR serverIPAddress as the bare ::1 instead of the bracketed [::1], so library/har.spec.ts's toMatch(/^127\.0\.0\.1|\[::1\]/) no longer matches. All 5 failures share that exact signature, and they reproduce on every retry — 10/10 attempts red on this PR, 0 green.
Details
Caused by this PR
[firefox-library] › library/har.spec.ts:629 › should have connection details
[firefox-library] › library/har.spec.ts:639 › should have security details
[firefox-library] › library/har.spec.ts:664 › should have connection details for redirects
[firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests
[firefox-library] › library/har.spec.ts:699 › should return server address directly from response
All five fail identically:
Expected pattern: /^127\.0\.0\.1|\[::1\]/
Received string: "::1"
This is a Firefox-only browser roll (roll-into-pw-firefox/1541), and serverIPAddress is populated by the rolled juggler's networking layer — squarely in the diff's blast radius. The new build emits the loopback IPv6 address unbracketed. The fix is either to restore the bracketed [::1] form in the juggler's HAR reporting, or to relax the test to accept a bare ::1. Chromium and WebKit are unaffected, which is exactly why only the firefox-library project went red.
Test-results DB confirms it's the roll, not a flake: across the recent window these tests failed on one branch only — this PR (10 failed / 0 passed). On every other PR and branch that ran them they're green (0 failures over hundreds of runs), so the flake hard-rule is not met — there's no run outside this PR where they misbehave.
Pre-existing flake / infra
The time test runner - frozen job also shows red, but its blob report lists no failing tests — only the 7 flaky tests below, all rescued on retry. That's the frozen-time job's usual flake noise, unrelated to the roll.
Reported flaky (passed on retry), not caused by this PR: chromium.spec.ts:301, video.spec.ts:495, video.spec.ts:736, browsercontext-cookies-third-party.spec.ts:257 & :470, heap.spec.ts:223, page-goto.spec.ts:90.
🔴 The Firefox roll breaks 5 HAR serverIPAddress tests
Hi, I'm the Playwright bot — I took a first look at the failing CI.
One real regression: Firefox r1541 reports the HAR serverIPAddress for the loopback as the bare ::1 instead of the bracketed [::1], so library/har.spec.ts's /^127\.0\.0\.1|\[::1\]/ no longer matches. All 5 firefox-library failures share that signature and fail on every attempt (2/2 red on this PR, 0 red across ~hundreds of other runs). The MCP clipboard and cli-core reds are pre-existing flakes a firefox roll can't be responsible for.
Details
The diff is firefox-only — browser_patches/firefox/juggler/* plus browsers.json. serverIPAddress is populated by juggler's networking layer, squarely in the blast radius, which is why only firefox-library went red while Chromium and WebKit stayed green.
Caused by this PR
[firefox-library] › library/har.spec.ts:629 › should have connection details
[firefox-library] › library/har.spec.ts:639 › should have security details
[firefox-library] › library/har.spec.ts:664 › should have connection details for redirects
[firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests
[firefox-library] › library/har.spec.ts:699 › should return server address directly from response
All five fail identically:
Expected pattern: /^127\.0\.0\.1|\[::1\]/
Received string: "::1"
Test-results DB confirms the roll, not a flake: each failed 2/2 on this PR and 0 times across 356–983 other runs on unrelated SHAs. Fix by restoring the bracketed [::1] form in juggler's HAR reporting, or relaxing the test to accept a bare ::1.
Pre-existing flake / infra
[chromium] › mcp/clipboard.spec.ts:19 › clipboard write without permission dialog (macOS, Windows, Ubuntu) — the clipboard read returns "" instead of "Hello from Playwright!". This is a Chromium-only test (Firefox is skipped), so a Firefox-only roll can't cause it. It flakes across the DB independent of this PR — 24 failures over 5824 other runs, including on main. The three OSes failing together points at a clipboard-access hiccup in that one MCP run.
The 7 tests reported flaky (rescued on retry) — chromium.spec.ts:301, video.spec.ts:495/:736, browsercontext-cookies-third-party.spec.ts:257/:470, heap.spec.ts:223, page-goto.spec.ts:90 — are the usual noise, not caused by the roll.
25 failed
❌ [playwright-test] › playwright.artifacts.spec.ts:127 › should work with screenshot: on @frozen-time-runner-chromium-linux
❌ [playwright-test] › reporter-html.spec.ts:307 › created › should not include image diff with non-images @frozen-time-runner-chromium-linux
❌ [firefox-library] › library/har.spec.ts:629 › should have connection details @tracing-firefox
❌ [firefox-library] › library/har.spec.ts:639 › should have security details @tracing-firefox
❌ [firefox-library] › library/har.spec.ts:664 › should have connection details for redirects @tracing-firefox
❌ [firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests @tracing-firefox
❌ [firefox-library] › library/har.spec.ts:699 › should return server address directly from response @tracing-firefox
❌ [firefox-library] › library/har.spec.ts:629 › should have connection details @firefox-macos-15-large
❌ [firefox-library] › library/har.spec.ts:639 › should have security details @firefox-macos-15-large
❌ [firefox-library] › library/har.spec.ts:664 › should have connection details for redirects @firefox-macos-15-large
❌ [firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests @firefox-macos-15-large
❌ [firefox-library] › library/har.spec.ts:699 › should return server address directly from response @firefox-macos-15-large
❌ [firefox-library] › library/har.spec.ts:629 › should have connection details @firefox-windows-latest
❌ [firefox-library] › library/har.spec.ts:639 › should have security details @firefox-windows-latest
❌ [firefox-library] › library/har.spec.ts:664 › should have connection details for redirects @firefox-windows-latest
❌ [firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests @firefox-windows-latest
❌ [firefox-library] › library/har.spec.ts:699 › should return server address directly from response @firefox-windows-latest
❌ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies @firefox-macos-15-xlarge
❌ [firefox-library] › library/har.spec.ts:629 › should have connection details @firefox-macos-15-xlarge
❌ [firefox-library] › library/har.spec.ts:639 › should have security details @firefox-macos-15-xlarge
❌ [firefox-library] › library/har.spec.ts:664 › should have connection details for redirects @firefox-macos-15-xlarge
❌ [firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests @firefox-macos-15-xlarge
❌ [firefox-library] › library/har.spec.ts:699 › should return server address directly from response @firefox-macos-15-xlarge
❌ [android-native] › android/device.spec.ts:56 › androidDevice.fill
❌ [android-native] › android/webview.spec.ts:44 › should navigate page externally
23 flaky⚠️ [chromium-library] › library/chromium/chromium.spec.ts:436 › should produce network events, routing, and annotations for Service Worker (advanced) `@tracing-chromium` ⚠️ [chromium-library] › library/chromium/launcher.spec.ts:37 › should not create pages automatically `@msedge-dev-windows-latest` ⚠️ [chromium-library] › library/inspector/cli-codegen-2.spec.ts:202 › cli codegen › should upload a file via hidden input triggered by button `@chrome-ubuntu-22.04` ⚠️ [chromium-library] › library/chromium/chromium.spec.ts:213 › should intercept service worker requests (main and within) `@chromium-macos-15-large` ⚠️ [chromium-library] › library/chromium/chromium.spec.ts:373 › should produce network events, routing, and annotations for Service Worker `@chromium-macos-15-large` ⚠️ [chromium-library] › library/chromium/chromium.spec.ts:436 › should produce network events, routing, and annotations for Service Worker (advanced) `@chromium-macos-15-large` ⚠️ [chromium-library] › library/video.spec.ts:390 › screencast › should scale frames down to the requested size `@chromium-macos-15-large` ⚠️ [chromium-library] › library/chromium/oopif.spec.ts:179 › should take screenshot `@chromium-macos-14-xlarge` ⚠️ [chromium-library] › library/video.spec.ts:736 › screencast › should work with video+trace `@chromium-macos-15-xlarge` ⚠️ [chromium-library] › library/beforeunload.spec.ts:130 › should support dismissing the dialog multiple times `@chrome-macos-latest` ⚠️ [chromium-library] › library/selector-generator.spec.ts:343 › selector generator › should trim long text `@chrome-macos-latest` ⚠️ [chromium-library] › library/selector-generator.spec.ts:673 › selector generator › should generate noText: no text engine `@chrome-macos-latest` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@tracing-firefox` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@tracing-firefox` ⚠️ [firefox-page] › page/page-goto.spec.ts:90 › should work with Cross-Origin-Opener-Policy `@tracing-firefox` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@firefox-macos-15-large` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-macos-15-large` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@firefox-windows-latest` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-windows-latest` ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-macos-15-xlarge` ⚠️ [webkit-library] › library/playwright-client.spec.ts:26 › should connect from a page and drive the same browser `@tracing-webkit` ⚠️ [webkit-library] › library/har.spec.ts:472 › should return receive time `@webkit-windows-latest` ⚠️ [webkit-library] › library/tracing.spec.ts:494 › should produce screencast frames fit `@webkit-macos-15-xlarge`
🔴 The Firefox roll breaks the HAR serverIPAddress tests
Hi, I'm the Playwright bot — I took a look at the failing CI.
One real regression from the roll: Firefox r1541 now reports the loopback HAR serverIPAddress as the bare ::1 instead of the bracketed [::1], so library/har.spec.ts's /^127\.0\.0\.1|\[::1\]/ no longer matches. The five affected tests fail on every Firefox platform in this run (ubuntu, macOS, windows, tracing) and on every retry. The test-results DB is unambiguous: 10 failures, all on this PR, 0 across 3034 runs on other SHAs. Everything else that's red is a pre-existing flake a Firefox-only browser roll can't be responsible for — with one Firefox cookie test I'd give a second glance.
Details
The diff is Firefox-only — browser_patches/firefox/juggler/*, preferences/playwright.cfg, and browsers.json. serverIPAddress is populated by juggler's networking layer, squarely in the blast radius, which is why only the firefox-library project went red while Chromium and WebKit stayed green.
Caused by this PR
[firefox-library] › library/har.spec.ts:629 › should have connection details
[firefox-library] › library/har.spec.ts:639 › should have security details
[firefox-library] › library/har.spec.ts:664 › should have connection details for redirects
[firefox-library] › library/har.spec.ts:686 › should have connection details for failed requests
[firefox-library] › library/har.spec.ts:699 › should return server address directly from response
All five fail identically:
Expected pattern: /^127\.0\.0\.1|\[::1\]/
Received string: "::1"
Across the recent window these five failed 10/10 on this PR and 0 times over 3034 runs on unrelated SHAs — the flake hard-rule isn't met anywhere outside this PR. Fix by restoring the bracketed [::1] form in juggler's HAR reporting, or by relaxing the test to accept a bare ::1.
Uncertain — worth a second look
[firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies — hard-failed on @firefox-macos-15-xlarge and was rescued-on-retry on several other Firefox bots in the same run. It has 0 failures over 985 runs outside this PR, so I can't call it a flake, and partitioned-cookie behavior is the kind of thing a Firefox version bump shifts — so it's within the roll's blast radius. It mostly self-rescued and didn't block, but I can't prove it's unrelated. The COOP page-goto.spec.ts:90 flake is a milder instance of the same "only-on-this-PR Firefox flakiness" pattern. Neither is confirmed, but both are worth a glance alongside the HAR fix.
Pre-existing flake / infra
[chromium] › mcp/clipboard.spec.ts:19 › clipboard write without permission dialog (macOS, Windows, Ubuntu) — a Chromium-only test (Firefox is skipped), so a Firefox roll can't touch it. Flakes independently of this PR: 8 failures across 5 other PRs over 731 runs.
[firefox] › mcp/cli-core.spec.ts:394 › --raw on command without output (Windows only) — TimeoutError waiting for aria-ref=e2; Firefox on macOS and Ubuntu passed in the same run. Standing flake: 5 failures across 4 other PRs over 722 runs.
[android-native] › android/device.spec.ts:56 › androidDevice.fill and [android-native] › android/webview.spec.ts:44 › should navigate page externally — Android runs don't exercise juggler's Firefox networking. Both flake across the DB: 12 failures on 2 other PRs over 348 runs.
[playwright-test] › playwright.artifacts.spec.ts:127 and [playwright-test] › reporter-html.spec.ts:307 (both @frozen-time-runner-chromium-linux) — Chromium test-runner tests; a Firefox browser roll can't change Chromium screenshot or HTML-reporter output. These have 0 failures anywhere in the DB and are the frozen-time job's usual harness noise.
The remaining tests reported flaky (rescued on retry) — chromium.spec.ts:301/:213/:373/:436, video.spec.ts:390/:495/:736, launcher.spec.ts:37, cli-codegen-2.spec.ts:202, oopif.spec.ts:179, beforeunload.spec.ts:130, selector-generator.spec.ts:343/:673, heap.spec.ts:223, playwright-client.spec.ts:26, har.spec.ts:472, tracing.spec.ts:494 — are the usual cross-engine noise, not caused by the roll.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Browser build: https://github.com/microsoft/playwright-browsers/commit/f81945fd6dce01624d8286736d47c4d014ab3a50. Build has full platform coverage - check that all tests pass there.