Implement #3956 - initial coding hint support

[clach04]

No description provided.

[clach04]

I've opened this to generate discussion.

1. What is expected to happen with badly declared / unknown encodings? Right now it dies ;-)
2. byte.readlines seems to work OK, so rather than read a few hundred bytes and search for newlines, I'm relying on getlines(). I've deliberately not looking for wide encodings like UTF-16 (any variants) or UTF-32 (any variants)
3. How many lines should be sniffed? Most editors seem to use 3 as the max - what I used for the initial implementation.

[AlightSoulmate]

Would it make sense to use a PEP 263-style regular expression to match encoding declarations?

^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

That would support both # -*- coding: cp1252 -*- and # coding: cp1252, while still only targeting ASCII-compatible coding cookies.

Also, if dictionary comment lines are supported in the future, encoding declarations should be checked before skipping comments. This is related to #3915.

[clach04]

@AlightSoulmate I don't think I've ever seen coding:, without the -*- prefix. Is this something you need or have seen?

I'd recommend against the regex unless a simple string match isn't possible for the usual reasons. Also I don't have a Regex License https://regexlicensing.org/ 😝

[DimitriPapadopoulos]

Doesn't chardet autodetect the encoding? I don't feel it's worth supporting other encodings without chardet.

[clach04]

Doesn't chardet autodetect the encoding? I don't feel it's worth supporting other encodings without chardet.

@DimitriPapadopoulos we can probably agree that charset is supposed to detect encoding, we may have different experiences in how well it does it 😆

However, this topic isn't about detection, it's about declarations, there's nothing to detect. This PR adds code to honor the very common (and old) character encoding declaration, sometimes named "hints".

I personally have zero interest in detection support, it's not reliable enough in my experience to be worth the effort of implementation nor the runtime expense of running the code.

[DimitriPapadopoulos]

I understand, but shouldn't we spend time fixing chardet instead of adding new code to codespell? Shouldn't chardet take into account coding: ?

[clach04]

@DimitriPapadopoulos I want to make sure I understand what you are suggesting; I think you are saying to open an enhancement idea in https://github.com/chardet/chardet (https://pypi.org/project/chardet/)?

I think that's a good idea, I also think it is reasonable for codespell to do the check encoding hit check (which is not the same function that chardet implements, chardet was originally written for incorrect or missing declarations). There is existing utf-8 and latin1 check code in codespell, so this is consistent with some basic sanity checks. I do not know how your chardet idea would be received by the chardet project, but it's worth trying as well. I'm also unclear how well chardet works in codespell (and I'm not motivated to dig into this).