[Bug]: Devops Pipelines/Releases fail to run npx playwright test

[wupaz]

Playwright version

1.12.3

Operating system

Windows

What browsers are you seeing the problem on?

Chromium

Other information

npm/6.14.13
node/v14.17.3 win32 x64

Azure Release : Agent.Version 2.188.4
Windows 2019

In package.json
"@playwright/test": "^1.12.3",
"playwright": "^1.12.3",

What happened? / Describe the bug

When I try to run npx playwright test in a Pipelines/Releases I got this error

If I run manually on same server same code (in release deploy directory) it work

Code snippet to reproduce your bug

npx playwright test

Relevant log output

021-07-13T17:35:20.4031005Z ##[section]Starting: Run Playwright Test 
2021-07-13T17:35:20.4155036Z ==============================================================================
2021-07-13T17:35:20.4155649Z Task : Command Line
2021-07-13T17:35:20.4156001Z Description : Run a command line with arguments
2021-07-13T17:35:20.4156302Z Version : 1.1.3
2021-07-13T17:35:20.4156597Z Author : Microsoft Corporation
2021-07-13T17:35:20.4157052Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2021-07-13T17:35:20.4157535Z ==============================================================================
2021-07-13T17:35:20.4186740Z ##[debug]Working directory: 'C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e'
2021-07-13T17:35:20.4196097Z ##[debug]Fail on standard error: 'False'
2021-07-13T17:35:20.4196492Z ##[debug]Modify environment: 'False'
2021-07-13T17:35:20.4196878Z ##[debug]C:\windows\system32\cmd.exe /c "npx playwright test"
2021-07-13T17:35:20.4197250Z ##[command]npx playwright test
2021-07-13T17:35:22.5056679Z Error: Error while reading C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:
2021-07-13T17:35:22.5058090Z describe() can only be called in a test file
2021-07-13T17:35:22.5956884Z ##[error]Process completed with exit code 1.

[aslushnikov]

wupaz peculiar! Any chance you can share this e2e\playwright\use-case\add-new-patient.uctest.spec.ts file that has this error? What happens if you remove (fully comment) this file?

[wupaz]

If I comment I got an error on second file, I got this same error for each file I have.

[wupaz]

import { test, expect, Page } from '@playwright/test';
import { AdminPageObject } from "../app-component/administrator-admin.po";
import { EditPatienPageObject } from "../app-component/dialog-edit-patient.po";
import { HeaderPageObject } from "../app-component/header.po";
import { LoginPageObject } from "../app-component/login.po";
import { ProfilesDetailsPageObject } from "../app-component/profiles-details.po";
import { ProfilesListPageObject } from "../app-component/profiles-list.po";
import { HomePageObject } from "../app-component/home.po";
import { Global } from "../Helper/global";
import { navigationHelper } from "../Helper/navigation";
import { PatientHelper } from "../Helper/patient-helper";


test.describe('Test la creation de patient', () => {
 let adminPO: AdminPageObject;
 let headerPO: HeaderPageObject;
 let loginPO: LoginPageObject;
 let profileListPO: ProfilesListPageObject;
 let profileDetailPO: ProfilesDetailsPageObject;
 let homePagePO: HomePageObject;
 let editPatienPagePO: EditPatienPageObject;

 test.beforeAll(async () => {
 });



 test.beforeEach(({ page }) => {
 adminPO = new AdminPageObject(page);
 loginPO = new LoginPageObject(page);
 headerPO = new HeaderPageObject(page);
 profileListPO = new ProfilesListPageObject(page);
 profileDetailPO = new ProfilesDetailsPageObject(page);
 homePagePO = new HomePageObject(page);
 editPatienPagePO = new EditPatienPageObject(page);
 });


 test('Creation dun patient minimum spec', async ({ page }) => {
 await PatientHelper.addRandomPatient(page);
 });

 test('Valider que le nouveau patient creer se retrouve en haut de la liste des patient recent', async ({ page }) => {
 try {
 let newPatient = await PatientHelper.addRandomPatient(page);
 await navigationHelper.navigateBack(page);
 console.log(newPatient);
 let patientNameInCard = await homePagePO.getFirstPatientFullNameFromRecentPatientCard();
 console.log("? ->" + patientNameInCard);
 console.log("? ->" + newPatient.firstName);
 expect(patientNameInCard.search(newPatient.firstName)).toBeTruthy();
 }
 finally { }
 });

 test.afterEach(async () => {
 });

 test.afterAll(async ({ browser }) => {
 });

});

[wupaz]

I try a another Pipelines/release Task (Powershell inline) same error

2021-07-13T21:05:30.4467965Z ##[debug]VstsTaskSdk 0.6.2 commit b4160c33da0010bd83279e9ffa9bbf13bcbd1ce2
2021-07-13T21:05:30.5070126Z ##[debug]Entering C:\AzurePiplinesAgent_Extension\_work\_tasks\InlinePowershell_31f040e5-e040-4336-878a-59a493355534\1.1.7\InlinePowershell.ps1.
2021-07-13T21:05:30.5252957Z ##[debug]Env:INPUT_SCRIPT: 'cd C:\AzurePiplinesAgent_Extension\_work\r4\a/GMaster\DevDrop/test_e2e
2021-07-13T21:05:30.5267119Z ##[debug]npx playwright test'
2021-07-13T21:05:30.5382869Z ##[debug]Env:INPUT_SCRIPTARGUMENTS (empty)
2021-07-13T21:05:30.5753446Z ##[command]& 'C:\windows\TEMP\tmp119A.ps1' 
2021-07-13T21:05:32.1518117Z ##[debug]Error record:
2021-07-13T21:05:32.2318698Z ##[debug]npx : Error: Error while reading C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:
2021-07-13T21:05:32.2321522Z ##[debug]At C:\windows\TEMP\tmp119A.ps1:2 char:1
2021-07-13T21:05:32.2336698Z ##[debug]+ npx playwright test
2021-07-13T21:05:32.2352948Z ##[debug]+ ~~~~~~~~~~~~~~~~~~~
2021-07-13T21:05:32.2381830Z ##[debug] + CategoryInfo : NotSpecified: (Error: Error wh...uctest.spec.ts::String) [], RemoteException
2021-07-13T21:05:32.2428411Z ##[debug] + FullyQualifiedErrorId : NativeCommandError
2021-07-13T21:05:32.2430024Z ##[debug] 
2021-07-13T21:05:32.2450045Z ##[debug]Script stack trace:
2021-07-13T21:05:32.2483976Z ##[debug]at <ScriptBlock>, C:\windows\TEMP\tmp119A.ps1: line 2
2021-07-13T21:05:32.2515067Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-07-13T21:05:32.2531532Z ##[debug]at <ScriptBlock>, C:\AzurePiplinesAgent_Extension\_work\_tasks\InlinePowershell_31f040e5-e040-4336-878a-59a493355534\1.1.7\InlinePowershell.ps1: line 46
2021-07-13T21:05:32.2548591Z ##[debug]at <ScriptBlock>, C:\AzurePiplinesAgent_Extension\_work\_tasks\InlinePowershell_31f040e5-e040-4336-878a-59a493355534\1.1.7\InlinePowershell.ps1: line 42
2021-07-13T21:05:32.2552343Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-07-13T21:05:32.2567609Z ##[debug]at <ScriptBlock>, <No file>: line 22
2021-07-13T21:05:32.2582462Z ##[debug]at <ScriptBlock>, <No file>: line 18
2021-07-13T21:05:32.2600467Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-07-13T21:05:32.2624629Z ##[debug]Exception:
2021-07-13T21:05:32.2677501Z ##[debug]System.Management.Automation.RemoteException: Error: Error while reading C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:
2021-07-13T21:05:32.2948381Z ##[error]Error: Error while reading C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:
2021-07-13T21:05:32.2951699Z ##[debug]Processed: ##vso[task.logissue type=error]Error: Error while reading C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:
2021-07-13T21:05:32.2954160Z ##[debug]Processed: ##vso[task.complete result=Failed]
2021-07-13T21:05:32.2962283Z ##[debug]Error record:
2021-07-13T21:05:32.3020668Z ##[debug]describe() can only be called in a test file
2021-07-13T21:05:32.3085953Z ##[debug]Script stack trace:
2021-07-13T21:05:32.3087620Z ##[debug]at <ScriptBlock>, C:\windows\TEMP\tmp119A.ps1: line 2
2021-07-13T21:05:32.3106179Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-07-13T21:05:32.3125498Z ##[debug]at <ScriptBlock>, C:\AzurePiplinesAgent_Extension\_work\_tasks\InlinePowershell_31f040e5-e040-4336-878a-59a493355534\1.1.7\InlinePowershell.ps1: line 46
2021-07-13T21:05:32.3444443Z ##[debug]at <ScriptBlock>, C:\AzurePiplinesAgent_Extension\_work\_tasks\InlinePowershell_31f040e5-e040-4336-878a-59a493355534\1.1.7\InlinePowershell.ps1: line 42
2021-07-13T21:05:32.4073812Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-07-13T21:05:32.4406380Z ##[debug]at <ScriptBlock>, <No file>: line 22
2021-07-13T21:05:32.4423870Z ##[debug]at <ScriptBlock>, <No file>: line 18
2021-07-13T21:05:32.4454842Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-07-13T21:05:32.4478150Z ##[debug]Exception:
2021-07-13T21:05:32.4526533Z ##[debug]System.Management.Automation.RemoteException: describe() can only be called in a test file
2021-07-13T21:05:32.4533485Z ##[error]describe() can only be called in a test file
2021-07-13T21:05:32.4534612Z ##[debug]Processed: ##vso[task.logissue type=error]describe() can only be called in a test file
2021-07-13T21:05:32.4545005Z ##[debug]Processed: ##vso[task.complete result=Failed]
2021-07-13T21:05:32.4822655Z ##[section]Finishing: Run Playwright Test 

[pavelfeldman]

It looks like your pipeline does not like npx. Try running node_modules\.bin\playwright test from the working folder instead. (Could be .exe,.cmd,.bat, not sure).

This could point to a problem in the tests:

2021-07-13T21:05:32.4526533Z ##[debug]System.Management.Automation.RemoteException: describe() can only be called in a test file
2021-07-13T21:05:32.4533485Z ##[error]describe() can only be called in a test file

[wupaz]

Maybe , but same test work fine on my pc and on deploy server when I run manually exact same code/test

[pavelfeldman]

Max is suggesting that you are running Jest tests in fact. We don't really have enough information to address it. It works locally for you, so Playwright seems to be Ok.

[wupaz]

Heu ... I didn't use Jest test.
Is it possible to active debug/verbose mode to get more information ?

 "dependencies": {
 "@angular/animations": "^12.0.0",
 "@angular/cdk": "^12.0.0",
 "@angular/common": "^12.0.0",
 "@angular/compiler": "^12.0.0",
 "@angular/core": "^12.0.0",
 "@angular/forms": "^12.0.0",
 "@angular/localize": "^12.0.0",
 "@angular/material": "^12.0.0",
 "@angular/material-moment-adapter": "^12.0.0",
 "@angular/platform-browser": "^12.0.0",
 "@angular/platform-browser-dynamic": "^12.0.0",
 "@angular/router": "^12.0.0",
 "@ng-idle/core": "^10.0.0-beta.1",
 "@ng-idle/keepalive": "^10.0.0-beta.1",
 "@types/lodash": "^4.14.171",
 "@types/node": "^14.0.27",
 "@types/smoothscroll-polyfill": "^0.3.1",
 "angular-calendar": "^0.28.16",
 "browserslist": "^4.16.3",
 "core-js": "^3.6.5",
 "date-fns": "^2.15.0",
 "eslint": "^7.28.0",
 "linq-es2015": "^2.5.1",
 "moment": "^2.27.0",
 "ngx-image-cropper": "^3.2.1",
 "ngx-infinite-scroll": "^9.0.0",
 "ngx-mat-select-search": "^3.2.0",
 "ngx-quill": "13.2.0",
 "ngx-toastr": "^13.0.0",
 "ngx-virtual-scroller": "^4.0.3",
 "open-sans-fontface": "^1.4.0",
 "proxy-polyfill": "^0.3.2",
 "quill": "^1.3.7",
 "reflect-metadata": "^0.1.13",
 "rxjs": "^6.6.6",
 "smoothscroll-polyfill": "^0.4.4",
 "ts-helpers": "^1.1.2",
 "tslib": "^2.0.0",
 "typescript-collections": "^1.3.3",
 "web-animations-js": "^2.3.2",
 "webdriver-manager": "^12.1.7",
 "xml-js": "^1.6.11",
 "zone.js": "~0.11.4"
 },
 "devDependencies": {
 "@angular-devkit/build-angular": "^12.0.0",
 "@angular-devkit/schematics": "^12.0.0",
 "@angular/cli": "^12.0.0",
 "@angular/compiler-cli": "^12.0.0",
 "@grapecity/ar-viewer": "^14.2.2655",
 "@playwright/test": "^1.12.3",
 "@types/quill": "1.3.10",
 "codelyzer": "^6.0.0",
 "playwright": "^1.12.3",
 "sass": "^1.32.13",
 "ts-node": "^8.10.2",
 "typescript": "4.2.4"
 }

[pavelfeldman]

Sorry, no ideas. Given that it works both locally and on the server, it suggests something wrong is with your CI environment. Sorry, we can't help much. You can set DEBUG=pw:browser environment variable to see if the browsers fail to run on your CI.

[wupaz]

Maybe this could help

2021-07-21T20:08:48.4562701Z Error: describe() can only be called in a test file
2021-07-21T20:08:48.4569421Z at TestTypeImpl._describe (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\node_modules\@playwright\test\lib\test\testType.js:83:23)
2021-07-21T20:08:48.4571552Z at Function.describe (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\node_modules\@playwright\test\lib\test\transform.js:132:12)
2021-07-21T20:08:48.4577999Z at Object.<anonymous> (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:25:12)
2021-07-21T20:08:48.4579459Z at Module._compile (internal/modules/cjs/loader.js:1085:14)
2021-07-21T20:08:48.4584945Z at Module._compile (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\pirates\lib\index.js:99:24)
2021-07-21T20:08:48.4586122Z at Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
2021-07-21T20:08:48.4588242Z at Object.newLoader (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\pirates\lib\index.js:104:7)
2021-07-21T20:08:48.4600487Z at Module.load (internal/modules/cjs/loader.js:950:32)
2021-07-21T20:08:48.4601569Z at Function.Module._load (internal/modules/cjs/loader.js:790:14)
2021-07-21T20:08:48.4603000Z at Module.require (internal/modules/cjs/loader.js:974:19)
2021-07-21T20:08:48.4603905Z at require (internal/modules/cjs/helpers.js:92:18)
2021-07-21T20:08:48.4616835Z at Loader._requireOrImport (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\@playwright\test\lib\test\loader.js:211:18)
2021-07-21T20:08:48.4618388Z at Loader.loadTestFile (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\@playwright\test\lib\test\loader.js:130:18)
2021-07-21T20:08:48.4619820Z at Runner._run (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\@playwright\test\lib\test\runner.js:210:59)

[wupaz]

Latest version 1.13.0 didn't work (Same error)

[mxschmitt]

Could you try it via the latest development version? npm install -D @playwright/test@next. We might get a better reporting out of that to find out if it's Jest or not.

[wupaz]

I try ... result is same ...

NPM Install log

2021-07-23T00:22:33.8874147Z ##[debug]C:\windows\system32\cmd.exe /c "npm i"
2021-07-23T00:22:33.8875295Z ##[command]npm i
2021-07-23T00:25:25.5528060Z 
2021-07-23T00:25:25.5529050Z > @playwright/test@1.14.0-next-alpha-jul-22-2021 install C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\node_modules\@playwright\test
2021-07-23T00:25:25.5529727Z > node install.js
2021-07-23T00:25:25.5529897Z 
2021-07-23T00:25:25.6594748Z 
2021-07-23T00:25:25.6596462Z > playwright@1.13.0 install C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\node_modules\playwright
2021-07-23T00:25:25.6597390Z > node install.js

Test run Task result log

2021-07-23T00:25:37.0088710Z ##[command]npm run test
2021-07-23T00:25:37.9016537Z 
2021-07-23T00:25:37.9100141Z > gustav@1.0.0 test C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e
2021-07-23T00:25:37.9120322Z > npx playwright test
2021-07-23T00:25:37.9140729Z 
2021-07-23T00:25:39.8566197Z Error: describe() can only be called in a test file
2021-07-23T00:25:39.8567759Z at TestTypeImpl._describe (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\node_modules\@playwright\test\lib\test\testType.js:83:23)
2021-07-23T00:25:39.8569381Z at Function.describe (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\node_modules\@playwright\test\lib\test\transform.js:133:12)
2021-07-23T00:25:39.8571079Z at Object.<anonymous> (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\test_e2e\e2e\playwright\use-case\add-new-patient.uctest.spec.ts:25:12)
2021-07-23T00:25:39.8572253Z at Module._compile (internal/modules/cjs/loader.js:1085:14)
2021-07-23T00:25:39.8573315Z at Module._compile (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\pirates\lib\index.js:99:24)
2021-07-23T00:25:39.8575130Z at Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
2021-07-23T00:25:39.8576280Z at Object.newLoader (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\pirates\lib\index.js:104:7)
2021-07-23T00:25:39.8577451Z at Module.load (internal/modules/cjs/loader.js:950:32)
2021-07-23T00:25:39.8578408Z at Function.Module._load (internal/modules/cjs/loader.js:790:14)
2021-07-23T00:25:39.8579245Z at Module.require (internal/modules/cjs/loader.js:974:19)
2021-07-23T00:25:39.8580061Z at require (internal/modules/cjs/helpers.js:92:18)
2021-07-23T00:25:39.8584330Z at Loader._requireOrImport (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\@playwright\test\lib\test\loader.js:211:18)
2021-07-23T00:25:39.8586159Z at Loader.loadTestFile (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\@playwright\test\lib\test\loader.js:130:18)
2021-07-23T00:25:39.8588036Z at Runner._run (C:\AzurePiplinesAgent_Extension\_work\r4\a\GMaster\DevDrop\Test_e2e\node_modules\@playwright\test\lib\test\runner.js:210:59)
2021-07-23T00:25:39.9869186Z npm ERR! code ELIFECYCLE