copy doc output files by format by onur-ozkan · Pull Request #104286 · rust-lang/rust

onur-ozkan · 2022-11-11T14:56:06Z

This pr provides copying doc outputs by checking output format without removing output directory on each trigger.

Resolves#103785

rustbot · 2022-11-11T14:56:15Z

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

onur-ozkan · 2022-11-11T14:57:28Z

I am not sure if this conditions will be enough for the purpose. What do you think ? @jyn514

onur-ozkan · 2022-11-13T17:45:20Z

pr is ready for review @Mark-Simulacrum @jyn514

Signed-off-by: ozkanonur <work@onurozkan.dev>

Mark-Simulacrum · 2022-11-13T23:59:07Z

Is there a reason we need to have a shared output directory for our invocations of cargo doc for JSON and not JSON? That seems much less picky in terms of picking file types out by extension...

In particular, it seems like the HTML format has some JSON files so the approach of this PR won't quite work:

~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share$ find doc/rust/html | grep json
doc/rust/html/book/searchindex.json
doc/rust/html/book/second-edition/searchindex.json
doc/rust/html/book/2018-edition/searchindex.json
doc/rust/html/book/first-edition/searchindex.json
doc/rust/html/rust-by-example/searchindex.json
doc/rust/html/error_codes/searchindex.json
doc/rust/html/cargo/searchindex.json
doc/rust/html/clippy/searchindex.json
doc/rust/html/rustc/searchindex.json
doc/rust/html/rustc/json.html
doc/rust/html/embedded-book/searchindex.json
doc/rust/html/rustdoc/searchindex.json
doc/rust/html/nomicon/searchindex.json
doc/rust/html/src/test/formatters/json.rs.html
doc/rust/html/unstable-book/searchindex.json
doc/rust/html/reference/searchindex.json
doc/rust/html/edition-guide/searchindex.json

jyn514 · 2022-11-14T02:25:49Z

Hmm, won't it hurt caching if cargo has to rebuild the dependencies in between? We can only control the target directory cargo uses, not the --out-dir it passes to rustdoc.

I agree this seems to cause issues though ... maybe a stamp file is better after all.

onur-ozkan · 2022-11-14T07:05:45Z

In particular, it seems like the HTML format has some JSON files so the approach of this PR won't quite work:

~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share$ find doc/rust/html | grep json
doc/rust/html/book/searchindex.json
doc/rust/html/book/second-edition/searchindex.json
doc/rust/html/book/2018-edition/searchindex.json
doc/rust/html/book/first-edition/searchindex.json
doc/rust/html/rust-by-example/searchindex.json
doc/rust/html/error_codes/searchindex.json
doc/rust/html/cargo/searchindex.json
doc/rust/html/clippy/searchindex.json
doc/rust/html/rustc/searchindex.json
doc/rust/html/rustc/json.html
doc/rust/html/embedded-book/searchindex.json
doc/rust/html/rustdoc/searchindex.json
doc/rust/html/nomicon/searchindex.json
doc/rust/html/src/test/formatters/json.rs.html
doc/rust/html/unstable-book/searchindex.json
doc/rust/html/reference/searchindex.json
doc/rust/html/edition-guide/searchindex.json

May I know how did you achieve this?

Mark-Simulacrum · 2022-11-14T13:25:36Z

The files in question can be installed with rustup +stable component add rust-docs and then the command and where I ran it from is in the snippet.

Mark-Simulacrum · 2022-11-14T13:28:06Z

Hmm, won't it hurt caching if cargo has to rebuild the dependencies in between?

I think in practice re-building the standard library and its dependencies shouldn't take long enough to be worried about it for now -- and if we follow the suggestion around different destinations via symlink updates or similar (below) then it might not need to rebuild dependencies.

We can only control the target directory cargo uses, not the --out-dir it passes to rustdoc.

I think we do some symlink'ing and such to adjust where the out-dir passed to rustdoc actually ends up, fwiw, but it's not exactly the most ideal thing.

onur-ozkan · 2022-11-15T15:14:13Z

I think in practice re-building the standard library and its dependencies shouldn't take long enough to be worried about it for now -- and if we follow the suggestion around different destinations via symlink updates or similar (below) then it might not need to rebuild dependencies.

Should I work on fixing the issue you mentioned then?

jyn514 · 2022-11-17T02:20:20Z

@ozkanonur yes please, if you use separate --target-dir flags for JSON and non-json output that should sidestep this issue altogether (and we can worry about the compile time later, I think Mark is right it won't be a big deal in practice).

Signed-off-by: ozkanonur <work@onurozkan.dev>

onur-ozkan · 2022-11-20T12:59:04Z

We can only control the target directory cargo uses, not the --out-dir it passes to rustdoc.

I tried to check if I can play with --out-dir by checking doc format, however I couldn't figure how -o is passed to here

rust/src/librustdoc/lib.rs

Line 705 in e07425d

fnmain_args(at_args:&[String]) -> MainResult{

At least now I have more experience about the rustdoc process pipelines.

if you use separate --target-dir flags for JSON and non-json output that should sidestep this issue altogether

With the last commit I aplied it for std doc(I think we don't need this for compiler-docs since it's always including html docs. Please correct me if I am wrong.).

jyn514 · 2022-11-23T02:42:26Z

This looks great! thank you :)

@bors r+

bors · 2022-11-23T02:42:28Z

📌 Commit 7e28df9 has been approved by jyn514

It is now in the queue for this repository.

Rollup of 6 pull requests Successful merges: - rust-lang#104269 (Fix hang in where-clause suggestion with `predicate_can_apply`) - rust-lang#104286 (copy doc output files by format) - rust-lang#104509 (Use obligation ctxt instead of dyn TraitEngine) - rust-lang#104721 (Remove more `ref` patterns from the compiler) - rust-lang#104744 (rustdoc: give struct fields CSS `display: block`) - rust-lang#104751 (Fix an ICE parsing a malformed attribute.) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

rustbot assigned Mark-Simulacrum Nov 11, 2022

rustbot added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 11, 2022

onur-ozkan commented Nov 11, 2022
View reviewed changes

onur-ozkan changed the title ~~copy doc output files by format~~WIP: copy doc output files by formatNov 11, 2022

onur-ozkan force-pushed the fix-doc-bootstrap-recompilation branch from c07f8e9 to 89d4550Compare November 13, 2022 17:44

onur-ozkan changed the title ~~WIP: copy doc output files by format~~copy doc output files by formatNov 13, 2022

This comment has been minimized.
Sign in to view

copy doc output files by format r=ozkanonur
0a275ab
Signed-off-by: ozkanonur <work@onurozkan.dev>

onur-ozkan force-pushed the fix-doc-bootstrap-recompilation branch from c6d6b4b to 0a275abCompare November 13, 2022 18:32

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 13, 2022

refactor doc copying process
7e28df9
Signed-off-by: ozkanonur <work@onurozkan.dev>

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 23, 2022

Dylan-DPC mentioned this pull request Nov 23, 2022
Rollup of 6 pull requests #104776
Merged

bors merged commit d3e9191 into rust-lang:masterNov 23, 2022

rustbot added this to the 1.67.0 milestone Nov 23, 2022

Uh oh!

Uh oh!

Conversation

onur-ozkan commented Nov 11, 2022

Uh oh!

rustbot commented Nov 11, 2022

Uh oh!

onur-ozkanNov 11, 2022

Choose a reason for hiding this comment

Uh oh!

onur-ozkan commented Nov 13, 2022

Uh oh!

This comment has been minimized.

This comment has been minimized.

Mark-Simulacrum commented Nov 13, 2022

Uh oh!

jyn514 commented Nov 14, 2022

Uh oh!

onur-ozkan commented Nov 14, 2022

Uh oh!

Mark-Simulacrum commented Nov 14, 2022

Uh oh!

Mark-Simulacrum commented Nov 14, 2022

Uh oh!

onur-ozkan commented Nov 15, 2022

Uh oh!

jyn514 commented Nov 17, 2022

Uh oh!

onur-ozkan commented Nov 20, 2022

Uh oh!

jyn514 commented Nov 23, 2022

Uh oh!

bors commented Nov 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants