Make the parser’s ‘expected <foo>, found <bar>’ errors more accurate

[ftxqxd]

As an example of what this changes, the following code:

let x:[int ..4];

Currently spits out ‘expected ], found ..’. However, a comma would also be valid there, as would a number of other tokens. This change adjusts the parser to produce more accurate errors, so that that example now produces ‘expected one of (, +, ,, ::, or ], found ..’.

(Thanks to cramer on IRC for pointing out this problem with diagnostics.)

[brson]

This looks pretty clever and useful.

Is it incorrect to ever compare equality of Tokens without check or check_operator? If so, it may be prudent to not implement PartialEq/Eq to avoid future errors.

[ftxqxd]

No, it’s not incorrect to compare Tokens using ==, and I do keep a number of uses of == in the code here, because we don’t want an error message to say that it expected something which isn’t supported any more (e.g., let x: fn int will say something like ‘expected (, found int’ instead of ‘expected one of ( or <, found int’, because the fn<'a>(…) syntax is deprecated).

That said, I think you’re right in that it’s very easy for someone modifying the parser in the future to write self.token == foo instead of self.check(&foo). Adding a new method which does not add the token to self.expected_tokens would let us remove PartialEq/Eq, but I’m wary about disallowing == for Tokens everywhere, not just in the parser (which is all we really want to disallow). Perhaps Parser.token could use a wrapper around Token that does not implement PartialEq instead of directly using Token?

[brson]

@P1start I'm not sure whether it's necessary. The potential bug is relatively minor: that the error fails to suggest a possible token. I'll leave it to your judgement to decide whether such defensive programming is warranted here.

[ftxqxd]

I decided not to prevent == on self.token. It’s just not worth the bother, and significantly decreases the ergonomics writing code in the parser.

I did end up removing check_operator, and the code to my eyes looks clearer and makes more sense. I also fixed up a few other special-cased ‘expected…’ messages to print out the actual symbol for the token instead of the internal representation of it.

r? @brson