Refactor doc comment parsing

[tesuji]

Split from #74094 and improved.
I have another way to implement is_doc_comment but might be slower: #74183

[tesuji]

May need a crater run for rustdoc.

[tesuji]

Actually, this PR needs compiler members review: r? @petrochenkov

[petrochenkov]

Oh, this looks like something from my todo list.
rustc should never do things like "decoration stripping" and should preserve tokens precisely, making doc comments look presentable to user is purely a rustdoc's job.

[tesuji]

That seems right thing to do. How would I proceed this PR?

[petrochenkov]

(I didn't look at the PR in detail yet, will review later.)

[tesuji]

This condition may seem useless as we never consider /****** [...] */ as doc comment.
I am not sure about /*!******** [..] */, should we keep this behavior.

[petrochenkov]

I think it was supposed to address the /****** case.
If it's not relevant now, then it's better to simplify the code and remove it.

[petrochenkov]

rustc should never do things like "decoration stripping" and should preserve tokens precisely, making doc comments look presentable to user is purely a rustdoc's job.

I'll submit a PR removing the doc comment beautification from the compiler tomorrow.

[petrochenkov]

Otherwise:

• comments, tests - good
• introducing a new tiny module for block comments - not good
• introducing unsafe - not good
• other changes - not sure, looks like a complete rewrite, but its purpose is not clear to me, will look after rebase on my PR

[petrochenkov]

Blocked on #74627.

[petrochenkov]

#74627 has landed.

[petrochenkov]

I still feel skeptical about many changes here (#74209 (comment)).
Could you keep the tests and comments (and maybe simplifications to fn beautify_doc_string), but avoid other changes?

[tesuji]

Some explanation:

introducing a new tiny module for block comments - not good

vertical_trim and horizontal_trim only used by beautify_doc_string.
I find it messier/bloat to move them under src/librustc_ast/util/comments.rs,
move tests to src/librustc_ast/util/comments/tests.rs.
If you find that's good I will definitely move it back.

other changes - not sure, looks like a complete rewrite, but its purpose
is not clear to me, will look after rebase on my PR

I find the logic straight-forward after rewriting line_doc_comment_style and block_doc_comment_style.

I admit the clippy changes are out of nowhere. Will move it to another PR.

introducing unsafe - not good

Yeah, I admit unsafe is not good. My goal was panic safe.
I tried to document safety invariants and wrote some tests for it.
But honestly I am no expert at writing unsafe.
I will try to replace them with safe functions (but it's hard to find ones panic-free)
if you find them annoying/unnecessary.

[bors]

☔ The latest upstream changes (presumably #75642) made this pull request unmergeable. Please resolve the merge conflicts.

[crlf0710]

@lzutao Ping from triage! CI is still red here. Any updates?

[tesuji]

Sorry, I lost interested in this change. Everyone else could pick it if they want.