rustdoc: include external files in documentation (RFC 1990)

[QuietMisdreavus]

Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR)

cc #44732

Also fixes#42760, because the prep work for the error reporting made it easy to fix that at the same time.

[rust-highfive]

r? @GuillaumeGomez

(rust_highfive has picked a reviewer for you, use r? to override)

[QuietMisdreavus]

Travis failure is because my test for 42760 failed -_- Back to the drawing board...

[QuietMisdreavus]

update: apparently is_sugared_doc is false for doc comments when you get them in with_desugared_doc. The issue was not fixed, because there's no way for it to see the difference between a doc comment and a raw doc attribute where it's extracting them. >:(

[QuietMisdreavus]

So i wound up modifying syntax::ast::Attribute::with_desugared_doc to set is_sugared_doc on the attribute when it calls the given closure. I'm not sure if this is entirely valid w.r.t. the original intent of that method, but rustdoc was the only in-tree user of that method that i found, so it may work out anyway. It fixed the issue in the test rendering i had, so i'll let travis see if i messed anything else up.

[QuietMisdreavus]

Travis was green, so i'm declaring this Ready To Be Looked At. I updated one test to make sure combining included docs and inline docs worked and squashed the commits. The libsyntax change is still in its own commit, though, since it was only included to make the fix for #42760 work properly, and i'm still not sure whether it's the totally right thing to do.

[GuillaumeGomez]

Oh so you decided to put in the "real" line number instead of the one from the original file?

[QuietMisdreavus]

Yeah. My thought was that if you're trying to trace a line number from the total markdown blob, then you could use that to get the doc fragment, since each attribute is joined with a line break anyway.

[GuillaumeGomez]

It'd be nice to have some outputs (or even tests?) for the case of failing doc codes.

[QuietMisdreavus]

It'd be nice to have some outputs (or even tests?) for the case of failing doc codes.

That's what i was forgetting! Give me a moment, i can make a quick run.

[QuietMisdreavus]

With the following files:

Details

lib.rs:

#![feature(external_doc)]/// hey yo check out these docs, i can splice them in from another file////// ```rust/// panic!("oh no");/// ```///#[doc(include = "some-file.md")]pubstructSomeStruct;/// Item docs.///#[doc="Hello there!"]////// # Example////// ```rust/// // some code here/// panic!("oh no");/// ```pubstructOtherStruct;#[doc(include = "bad-docs.md")]#[doc(include = "no-docs.md")]pubstructAllTheStruct;

some-file.md:

hey check out this included file
this is so sick, i can just paste this anywhere

bad-docs.md:

# some bad docs
```rust
panic!("oh no");
```

(no-docs.md deliberately missing)

Running cargo test gives the following output:

Details

This is with the latest force push. (I neglected to ask rustdoctest to combine doc fragments, so that's fixed now.)

EDIT: The current line numbering refers to the line within the final markdown, not the line in the source file. So i guess to properly translate between this line number and the source line, you need to store both the "markdown line" and the "source line". This could get hairy to properly account for multiple #[doc] attributes written on the same line (rocket has a macro that does this), but i guess if you check the spans while loading, it may work out.

[GuillaumeGomez]

And a column as well for the "real" line number (so a Span?). If you want, I can work on that part (the test one).

[QuietMisdreavus]

@GuillaumeGomez I added a Span to each DocFragment and made sure that they get properly combined in collapse-docs, does this seem all right to you?

[GuillaumeGomez]

Please post a test failure output. :)

[QuietMisdreavus]

I haven't touched the failure reporting code, so it'll look the same as in #44781 (comment). Do you want me to make that part of this PR, or would you like to merge this as-is and take it up yourself afterward?

[GuillaumeGomez]

I don't mind merging as is. Just tell me what you prefer to do.

[QuietMisdreavus]

If i remember right, we discussed fixing up the error reporting in a separate PR, so that's what i was assuming we would do here (and my preference, just so we can get this in >_>). If you want to wrap it all into one PR, just tell me what code is likely to need the span information and i can take a shot at it.

[ollie27]

This doesn't work cross-crate. If you re-export an item that uses #[doc(include)] then it will look for the file relative to the current crate not the crate that actually contains the #[doc(include)].

[QuietMisdreavus]

@ollie27 Oh yikes, that's a good point. On the other hand, i don't really see a good way to reliably do it cross-crate without asking rustc to load the file in. In the RFC discussion we talked about making it an alias for #[doc=include_str!("file.md")] which would do exactly that, but would require work within the compiler to make sure certain macros could be used within attributes and expand correctly. (That's also outside my expertise. >_>)

[SimonSapin]

Having a special case for doc include in rustc might be easier than supporting arbitrary macros in arbitrary attributes. (Or maybe not, I don’t know.)

[bors]

☔ The latest upstream changes (presumably #45046) made this pull request unmergeable. Please resolve the merge conflicts.

[aidanhs]

@QuietMisdreavus I've set this back as with author for the merge conflicts. After that, I'm also not clear after that if you're going to do more work or it's going back to review by @GuillaumeGomez?

[QuietMisdreavus]

@aidanhs I'm actually not sure either. I'm concerned it may need to be scrapped entirely in favor of the #[doc=include_str!()] method alluded to above. I'm not entirely sure how to go about that, though - it would require work on the compiler frontend to extend #34981 to include (at least) some builtin macros as an allowed expression in attributes, then also adding #[doc(include="file.md")] as sugar for that. Should we pull in compiler-team to take a look?

[GuillaumeGomez]

Won't kill to add them in the loop. :)

cc @rust-lang/compiler

[alexcrichton]

triage ping for @QuietMisdreavus, do you know what the next steps here are?

[GuillaumeGomez]

Shouldn't debug! be used instead? (Don't really know the good macro we're supposed to use in such cases...)

[QuietMisdreavus]

Probably? I'm not that solid on what counts as what either, but i'll go ahead and switch it over.

[GuillaumeGomez]

Whitespace after // please.

[GuillaumeGomez]

Whitepace.

[GuillaumeGomez]

All good for me from code reading. Just a few nits and it's good to go imo.

[QuietMisdreavus]

I've force-pushed to address the review comments.

[GuillaumeGomez]

Then it's all good for me. r=me once CI is green.

[QuietMisdreavus]

@bors r=GuillaumeGomez

[bors]

📌 Commit 52ee203 has been approved by GuillaumeGomez

[bors]

⌛ Testing commit 52ee203 with merge 0893ecc8955191f702513ff3a19d420f9650fae5...

[bors]

💔 Test failed - status-travis

[QuietMisdreavus]

Failure... doesn't look legit? I'm not caught up on the known spurious failures.

Installing deploy dependencies
ERROR: Could not find a valid gem 'aws-sdk-resources' (= 2.10.88) in any repository
ERROR: Possible alternatives: aws-sdk-resources
/home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- aws-sdk (LoadError)
from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:59:in `require'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:85:in `requires'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:6:in `<class:S3>'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:5:in `<class:Provider>'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:4:in `<module:DPL>'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:3:in `<top (required)>'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:59:in `const_get'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:59:in `block in new'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/cli.rb:41:in `fold'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:56:in `new'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/cli.rb:31:in `run'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/cli.rb:7:in `run'
from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/bin/dpl:5:in `<top (required)>'
from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `load'
from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `<main>'

[kennytm]

@bors retry #44159

Looks like that previous attempt to fix it doesn't work.

[bors]

⌛ Testing commit 52ee203 with merge 3755fe9...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: GuillaumeGomez
Pushing 3755fe9 to master...