Make some of lexer's API private

[matklad]

Lexer is a pub type, so it feels wrong that its fields are just pub (I guess it wasn't exported initially), so let's minimize visibility.

Context: I am looking into extracting rust-lexer into a library, which can be shared by rust-analyzer and rustc. I hope that a simple interface like fn next_token(src: &str) -> (TokenKind, usize) would work, but to try this out I need to understand what is the current API of the lexer.

[rust-highfive]

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

[eddyb]

I hope that a simple interface like fn next_token(src: &str) -> (TokenKind, usize) would work, but to try this out I need to understand what is the current API of the lexer.

The API feels like it predates iterators, in some sense, it should probably be streamlined.

[eddyb]

For example, the peeking should ideally be handled with .peekable().

Anyway, this seems like a good start.

@bors r+

[bors]

📌 Commit 1763aea has been approved by eddyb

[matklad]

@eddyb yeah, I feel like this could be improved massively by separating the actual lexing from rustc-specific bits like spans and source maps. See https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/Extracting.20the.20lexer/near/162515922 for the rough proposal

[bors]

⌛ Testing commit 1763aea with merge 52980d0...

[bjorn3]

Why is bors testing this and #59556? It is already merged.

[RalfJung]

@bors r- retry