[WIP] Add rustdoc GUI tests by GuillaumeGomez · Pull Request #70533 · rust-lang/rust

GuillaumeGomez · 2020-03-29T13:16:45Z

[TODO]

Mark-Simulacrum

Sorry for the delay here, have been quite busy.

GuillaumeGomez · 2020-04-14T16:10:28Z

Updated: I moved what was needed into bootstrap (so the cloning, the setup and the run). Only kept the required dependencies into the docker file.

GuillaumeGomez · 2020-05-26T13:46:56Z

Applied the comments, sorry for the long delay.

Mark-Simulacrum · 2020-06-03T01:52:12Z

Can we just use curl? Looks like the apparent use is for the wget below but that should be fine to just use curl for I imagine?

I guess, I tend to prefer wget because it's simpler but we can use curl.

However, I just copied the code from the chromium repository so I'd prefer not to change it until everything is working so we can use the current code as witness.

GuillaumeGomez · 2020-06-06T11:33:26Z

Updated.

Mark-Simulacrum · 2020-06-25T19:57:49Z

Okay, so this is looking relatively good to me -- I guess maybe we should try to test it? We'll want to add a dedicated Makefile target I think, like check-aux but with GUI tests enabled too (perhaps check-aux-and-gui or something). That can be done roughly here:

rust/src/bootstrap/mk/Makefile.in

Lines 43 to 47 in a8cf399

check-aux: 
$(Q)$(BOOTSTRAP)test\
 src/tools/cargo \
 src/tools/cargotest \
$(BOOTSTRAP_ARGS)

By adding something like:

check-aux-and-gui: check-aux
$(Q)$(BOOTSTRAP) test \
src/test/rustdoc-gui \
$(BOOTSTRAP_ARGS)

We'll also want to enable this builder on try branch, by applying this diff:

diff --git a/src/ci/azure-pipelines/try.yml b/src/ci/azure-pipelines/try.yml
index 818306a0092..eea08d1f26a 100644
--- a/src/ci/azure-pipelines/try.yml+++ b/src/ci/azure-pipelines/try.yml@@ -26,6 +26,7 @@ jobs:
strategy:
matrix:
dist-x86_64-linux: {}
+ x86_64-gnu-aux: {}
# The macOS and Windows builds here are currently disabled due to them not being
# overly necessary on `try` builds. We also don't actually have anything that

Once that's done we can run bors try here on this PR and see what the results are.

rust-log-analyzer · 2020-08-26T13:45:30Z

The job x86_64-gnu-aux of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.


finished in 360.412
Build completed successfully in 0:32:50
Finished dev [unoptimized + debuginfo] target(s) in 0.39s
Cloning into '/checkout/obj/build/browser-UI-test'...
Note: checking out 'e47b1a8697f628429cb684e0b3716737a3a0fa78'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at e47b1a8... Merge pull request #152 from GuillaumeGomez/up-docker
Cloning into '/checkout/obj/build/test-rust-docs-ui'...
Note: checking out 'b8516285a33276dac37c91036ea5966379757ffd'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at b851628... Merge pull request #82 from GuillaumeGomez/update-to-last-rustdoc
Finished release [optimized] target(s) in 0.16s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Could not determine the LLVM submodule commit hash. Assuming that an LLVM rebuild is not necessary.
To force LLVM to rebuild, remove the file `/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/llvm-finished-building`
---
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
HEAD is now at e47b1a8... Merge pull request #152 from GuillaumeGomez/up-docker
HEAD is now at b851628... Merge pull request #82 from GuillaumeGomez/update-to-last-rustdoc
> puppeteer@2.1.1 install /checkout/obj/build/browser-UI-test/node_modules/puppeteer
> node install.js
Chromium downloaded to /checkout/obj/build/browser-UI-test/node_modules/puppeteer/.local-chromium/linux-722234
added 164 packages from 138 contributors and audited 164 packages in 10.829s
found 14 low severity vulnerabilities
run `npm audit fix` to fix them, or `npm audit` for details
Finished release [optimized] target(s) in 0.16s
=> Starting doc-ui tests...
basic-code... ok
basic-code... ok
basic... ok
code-sidebar-toggle-mobile... ok
code-sidebar-toggle... ok
list_code_block... ok
module-mobile... ok
theme-change... FAILED (images "failures/theme-change-test.png" and "theme-change" are different)
toggle-docs... FAILED (images "failures/toggle-docs-test.png" and "toggle-docs" are different)
<= doc-ui tests done: 6 succeeded, 2 failed
command did not execute successfully: "/usr/bin/node" "../browser-UI-test/src/index.js" "--no-sandbox" "--test-folder" "ui-tests" "--failure-folder" "failures" "--variable" "DOC_PATH" "/checkout/obj/build/test-rust-docs-ui/test-docs/doc/test_docs" "--show-text" "--generate-images"
failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --stage 2 src/test/rustdoc-gui
Build completed unsuccessfully in 0:00:29
Build completed unsuccessfully in 0:00:29
make: *** [check-aux-and-gui] Error 1
Makefile:49: recipe for target 'check-aux-and-gui' failed
local time: Wed Aug 26 13:45:20 UTC 2020
network time: Wed, 26 Aug 2020 13:45:20 GMT
== end clock drift check ==
== end clock drift check ==
##[error]Process completed with exit code 2.
Terminate orphan process: pid (6472) (python)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

bors · 2020-08-26T13:45:44Z

💔 Test failed - checks-actions

GuillaumeGomez · 2020-08-26T16:13:38Z

@bors try

bors · 2020-08-26T16:13:49Z

⌛ Trying commit 0756b1e with merge b32cfefd794845bdd30242d49527ffa17dd0b3a1...

rust-log-analyzer · 2020-08-26T16:55:58Z

The job x86_64-gnu-aux of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.


finished in 416.838
Build completed successfully in 0:37:19
Finished dev [unoptimized + debuginfo] target(s) in 0.52s
Cloning into '/checkout/obj/build/browser-UI-test'...
Note: checking out 'e47b1a8697f628429cb684e0b3716737a3a0fa78'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at e47b1a8... Merge pull request #152 from GuillaumeGomez/up-docker
Cloning into '/checkout/obj/build/test-rust-docs-ui'...
Note: checking out 'd74abb60d1bd1e3fbf647d2d2eb521f2aa357f27'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at d74abb6... Merge pull request #84 from GuillaumeGomez/wait
Finished release [optimized] target(s) in 0.21s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Could not determine the LLVM submodule commit hash. Assuming that an LLVM rebuild is not necessary.
To force LLVM to rebuild, remove the file `/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/llvm-finished-building`
---
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
HEAD is now at e47b1a8... Merge pull request #152 from GuillaumeGomez/up-docker
HEAD is now at d74abb6... Merge pull request #84 from GuillaumeGomez/wait
> puppeteer@2.1.1 install /checkout/obj/build/browser-UI-test/node_modules/puppeteer
> node install.js
Chromium downloaded to /checkout/obj/build/browser-UI-test/node_modules/puppeteer/.local-chromium/linux-722234
added 164 packages from 138 contributors and audited 164 packages in 11.278s
found 14 low severity vulnerabilities
run `npm audit fix` to fix them, or `npm audit` for details
Finished release [optimized] target(s) in 0.21s
=> Starting doc-ui tests...
basic-code... ok
basic-code... ok
basic... ok
code-sidebar-toggle-mobile... ok
code-sidebar-toggle... ok
list_code_block... ok
module-mobile... ok
theme-change... FAILED (images "failures/theme-change-test.png" and "theme-change" are different)
toggle-docs... FAILED (images "failures/toggle-docs-test.png" and "toggle-docs" are different)
<= doc-ui tests done: 6 succeeded, 2 failed
command did not execute successfully: "/usr/bin/node" "../browser-UI-test/src/index.js" "--no-sandbox" "--test-folder" "ui-tests" "--failure-folder" "failures" "--variable" "DOC_PATH" "/checkout/obj/build/test-rust-docs-ui/test-docs/doc/test_docs" "--show-text" "--generate-images"
failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --stage 2 src/test/rustdoc-gui
Build completed unsuccessfully in 0:00:38
Build completed unsuccessfully in 0:00:38
make: *** [check-aux-and-gui] Error 1
Makefile:49: recipe for target 'check-aux-and-gui' failed
local time: Wed Aug 26 16:55:48 UTC 2020
network time: Wed, 26 Aug 2020 16:55:48 GMT
== end clock drift check ==
== end clock drift check ==
##[error]Process completed with exit code 2.
Terminate orphan process: pid (3953) (python)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

bors · 2020-08-26T16:57:29Z

💔 Test failed - checks-actions

GuillaumeGomez · 2020-08-26T19:45:02Z

So at this point, only 2 tests are failing, however I need to be able to get the images to understand why. More globally, we need to provide an easy way for contributors to get the images to see what went wrong. cc @rust-lang/infra

Mark-Simulacrum · 2020-08-26T20:12:03Z

I still am uncertain what exactly you mean by images. Is that the docker images? A model like the one I proposed here might be necessary: rust-lang/triagebot#760

jyn514 · 2020-08-26T20:57:55Z

I still am uncertain what exactly you mean by images. Is that the docker images? A model like the one I proposed here might be necessary: rust-lang/triagebot#760

Images like PNGs. https://github.com/GuillaumeGomez/test-rust-docs-ui/tree/master/ui-tests

GuillaumeGomez · 2020-08-26T21:40:56Z

The images generated in case of failures (the PNG screenshots).

Mark-Simulacrum · 2020-08-26T21:47:23Z

Yeah, then I think the triagebot issue I linked to is the best bet for the most up to date mentor-ish issue on integrating retrieval of artifacts into CI. I would comment there to sync up with @lcnr though who expressed interest in working on this.

GuillaumeGomez · 2020-08-26T21:49:30Z

Fine by me. In the meantime, I'd be really interested into being able to know where the images are (no need for automation on this part I hope) so I can at least move forward on this PR. :)

Mark-Simulacrum · 2020-08-26T22:24:39Z

I am confused - you believe you have uploaded them? We don't upload by default from non dist builders at all...

GuillaumeGomez · 2020-08-27T09:04:37Z

Oh snap... Any idea on how we could get access to the images generated then?

Mark-Simulacrum · 2020-08-27T12:40:45Z

...I have already pointed you at an issue typed up with docs on how I would suggest going about doing this? If something in that description is unclear, please comment on that issue with questions. It doesn't require triagebot integration, just work in CI here, for basically all of the steps.

GuillaumeGomez · 2020-08-27T12:44:48Z

I misundestood the issue then... Going to re-read it.

bors · 2020-09-30T10:08:57Z

☔ The latest upstream changes (presumably #77294) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

Mark-Simulacrum · 2020-11-22T21:57:18Z

I'm going to close this for now to keep the list of PRs assigned to me cleaner, but feel free to reopen when you have more time to work on it!

rust-highfive assigned Mark-Simulacrum Mar 29, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 29, 2020

Dylan-DPC-zz changed the title ~~Add rustdoc GUI tests~~[WIP] Add rustdoc GUI testsMar 29, 2020

Mark-Simulacrum reviewed Apr 11, 2020
View reviewed changes

Comment threadsrc/ci/docker/x86_64-gnu-aux/Dockerfile Outdated
Comment threadsrc/ci/docker/x86_64-gnu-aux/Dockerfile Outdated

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 11, 2020

Dylan-DPC-zz mentioned this pull request Apr 14, 2020
Keep code coloring in search results short text #68699
Closed

GuillaumeGomez force-pushed the add-rustdoc-gui-tests branch from 90d7118 to 6e9729eCompare April 14, 2020 16:09

This comment has been minimized.
Sign in to view

GuillaumeGomez force-pushed the add-rustdoc-gui-tests branch from 6e9729e to ef95da7Compare April 14, 2020 18:36

This comment has been minimized.
Sign in to view

Mark-Simulacrum reviewed Apr 15, 2020
View reviewed changes

Comment threadsrc/bootstrap/util.rs Outdated
Comment threadsrc/bootstrap/test.rs Outdated
Comment threadsrc/bootstrap/test.rs Outdated

crlf0710 added the T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. label Apr 24, 2020

crlf0710 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 2, 2020

GuillaumeGomez mentioned this pull request May 9, 2020
Sort "implementations on foreign types" section in the sidebar #71945
Merged

crlf0710 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 15, 2020

GuillaumeGomez force-pushed the add-rustdoc-gui-tests branch 2 times, most recently from b827068 to ea7d406Compare May 26, 2020 15:30

GuillaumeGomez added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 31, 2020

Mark-Simulacrum reviewed Jun 4, 2020
View reviewed changes

GuillaumeGomez force-pushed the add-rustdoc-gui-tests branch from ea7d406 to ade95aaCompare June 6, 2020 11:20

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2020

Update both repositories to new docker image run
0756b1e

GuillaumeGomez force-pushed the add-rustdoc-gui-tests branch from f9a2455 to 0756b1eCompare August 26, 2020 16:13

jyn514 mentioned this pull request Aug 31, 2020
Make the default stage for x.py configurable #76165
Closed

crlf0710 marked this pull request as draft September 18, 2020 12:27

jyn514 added the A-testsuite Area: The testsuite used to check the correctness of rustc label Sep 30, 2020

GuillaumeGomez mentioned this pull request Oct 12, 2020
rustdoc: Tracking issue for heavily increasing the GUI test coverage #42018
Open

Mark-Simulacrum closed this Nov 22, 2020

camelid added the S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. label Dec 9, 2020

GuillaumeGomez mentioned this pull request Dec 12, 2020
Rustdoc gui tests #79979
Merged

Uh oh!

Uh oh!

Conversation

GuillaumeGomez commented Mar 29, 2020• edited by Mark-Simulacrum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GuillaumeGomez commented Apr 14, 2020

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

GuillaumeGomez commented May 26, 2020

Uh oh!

Mark-SimulacrumJun 3, 2020

Choose a reason for hiding this comment

Uh oh!

GuillaumeGomezJun 6, 2020

Choose a reason for hiding this comment

Uh oh!

GuillaumeGomezJun 6, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GuillaumeGomez commented Jun 6, 2020

Uh oh!

Mark-Simulacrum commented Jun 25, 2020

Uh oh!

rust-log-analyzer commented Aug 26, 2020

Uh oh!

bors commented Aug 26, 2020

Uh oh!

GuillaumeGomez commented Aug 26, 2020

Uh oh!

bors commented Aug 26, 2020

Uh oh!

rust-log-analyzer commented Aug 26, 2020

Uh oh!

bors commented Aug 26, 2020

Uh oh!

GuillaumeGomez commented Aug 26, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mark-Simulacrum commented Aug 26, 2020

Uh oh!

jyn514 commented Aug 26, 2020

Uh oh!

GuillaumeGomez commented Aug 26, 2020

Uh oh!

Mark-Simulacrum commented Aug 26, 2020

Uh oh!

GuillaumeGomez commented Aug 26, 2020

Uh oh!

Mark-Simulacrum commented Aug 26, 2020

Uh oh!

GuillaumeGomez commented Aug 27, 2020

Uh oh!

Mark-Simulacrum commented Aug 27, 2020

Uh oh!

GuillaumeGomez commented Aug 27, 2020

Uh oh!

bors commented Sep 30, 2020

Uh oh!

Mark-Simulacrum commented Nov 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

GuillaumeGomez commented Mar 29, 2020•
edited by Mark-Simulacrum
Loading

GuillaumeGomez commented Aug 26, 2020•
edited
Loading