[BUG] playwright tests are getting passed locally but failing in Github actions

[saikrishnaoggu]

Context:

• Playwright Version: [1.25.0]
• Operating System: [Windows]
• Node.js version: [16.7.1]
• Browser: [e.g. All, Chromium, Firefox, WebKit]
• Extra: [any specific details about your environment]

Code Snippet
---> Below is the workflow I have been using.

`name: Playwright Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '16.x'

- name: Install dependencies
   run: npm install -g yarn && yarn install && rm -rf node_modules && yarn install --frozen-lockfile 

- name: Install Playwright Browsers
  run: yarn playwright install --with-deps

- name: Run Playwright tests
  run: npx playwright test --max-failures=10

- uses: actions/upload-artifact@v3
  if: always()
  with:
    name: playwright-report
    path: playwright-report/
    retention-days: 3`

Describe the bug

My playwright tests are getting passed locally but when it is executed through Github Actions it is failing. It is showing Time out error for both web based tests and API tests.

[mxschmitt]

Why are you doing:

npm install -g yarn && yarn install && rm -rf node_modules && yarn install --frozen-lockfile 

and not just:

npm install -g yarn && yarn install --frozen-lockfile 

Also for Yarn we recommend:

yarn playwright install --with-deps

instead of:

npx playwright install --with-deps

Last but not least we recommend the HTML report to debug failing tests. This will give you access to the trace to see how your website was behaving when the click happened - since it essentially tells you it didn't find the element with the selector. See here for the GitHub Actions guide.

[MatMath]

We use ubuntu on the runner and had the same problem this morning.
We updated node 16 to 18 & used yarn playwright install --with-deps instead of npx as suggested and it solved the issue for us.

[saikrishnaoggu]

Hi Mathieu (@MatMath) and Max Schmitt (@mxschmitt)

Thanks for responding, I tried your suggestions. Present my API Tests are getting passed successfully but Web application Tests are getting failed still. It is showing time out error though I was using wait for selector method. Could you please help me to fix this issue?

And also it is not showing "Artifacts" to download playwright reports - Max Schmitt (@mxschmitt)

[saikrishnaoggu]

Hi Mathieu (@MatMath) Max Schmitt (@mxschmitt) - could you please help me out ?

[mxschmitt]

Oh sorry, I thought you were the issue owner - thats why I closed it.

if no artifacts are shown, ensure that you have the html enabled inside your playwright config.

[sergei-maertens]

I'm having similar problems with the Python variant, see for example https://github.com/open-formulieren/open-forms/actions/runs/4115065601/jobs/7103271530

I've added tracing and analyzed the traces with playwright show-trace trace.zip and I can see that the form input labels are there, but somehow they are not being detected.

Locally everything is passing, both headless and headed mode.

Using Node 18 instead of 16 does not resolve it.

edit - dug a little deeper and managed to reproduce and fix locally, it was specific to our environment

[mxschmitt]

Sergei Maertens (@sergei-maertens) please file a separate issue.

[mxschmitt]

Looks like the element which you are trying to click is not available. I recommend trace viewer aka. Html report to debug this further as per #20664 (comment)

Make sure to enable html report in the reporter section and set trace to ‘retain-on-failure’.

[aslushnikov]

I'll close this as per the response above; if you think this is still a bug in Playwright, and you have a good repro, please do not hesitate to file a new issue!

[Cholpond]

Hi Max Schmitt (@mxschmitt) & Mathieu (@MatMath),

I'm encountering issues with Playwright tests for 3D configurators on GitHub Actions; they pass locally but fail in CI with 'Operation canceled' errors. I suspect this might be due to GPU resource limits.

Could anyone share experiences or insights on running similar GPU-intensive tests on GitHub Actions? Also, I wonder what the current support and future plans are for GPU resources in GitHub Actions

Thanks for any insights or possible solutions you can share.

[apotema]

Why are you doing:

npm install -g yarn && yarn install && rm -rf node_modules && yarn install --frozen-lockfile 

and not just:

npm install -g yarn && yarn install --frozen-lockfile 

Also for Yarn we recommend:

yarn playwright install --with-deps

instead of:

npx playwright install --with-deps

Last but not least we recommend the HTML report to debug failing tests. This will give you access to the trace to see how your website was behaving when the click happened - since it essentially tells you it didn't find the element with the selector. See here for the GitHub Actions guide.

That did the trick for me.
Playwright should update their docs.