Allow comment in dictionary

[AlightSoulmate]

Fix: #3901

Add validation to build_dict loop in codespell/codespell_lib/_spellchecker.py to prevent crashes mentioned in #3901 :

forlineinf:
line=line.strip()
# check statementifnotlineorline.startswith("#") or"->"notinline:
continue
[key, data] =line.split("->")

Lines starting with # will be recognized as comments and be ignored.
Empty lines and lines missing the -> are also ignored, even though they are not supposed to exist.

Manual verification:
Created example/sample.py to test the revision:

# example/sample.pydeftest():
print("clas")
print("buring")

# example/dict.txt
# 1. This is a comment line
tis->this
opem->open
buring->burying, burning, burin, during,
clas->class, disabled due to name clash in c++
# 2. Below is an empty line
# 3. Below is a line without '->'
servcie-service

Running on project root

python -m codespell_lib -D example/dict.txt example/sample.py

No ValueError.

[DimitriPapadopoulos]

Isn't this a duplicate of #2617?

[AlightSoulmate]

Isn't this a duplicate of #2617?

I overlooked the previous PR, sorry about that. Since the previous one appears to have stalled, I can help complete it by adding the missing tests if you're open to including this feature.

[DimitriPapadopoulos]

I must admit I am always afraid new enhancements might impair performance. But then we don't measure performance, and reading the dictionaries shouldn't be the bottleneck in most use cases. Let's give it a try.

Note that this change would disallow typos and fixes including # - but I think we have decided that's OK 😄

To start with, instead of testing "->" not in line, I would recommend:

- [key, data] = line.split("->")+ try:+ key, data = line.split("->")+ except ValueError:+ continue

[DimitriPapadopoulos]

Then you could also have a look at #2063/#2068, which are about comments in --ignore-words files.

[DimitriPapadopoulos]

Looks good, but shouldn't documentation be updated?

[AlightSoulmate]

The core appeal is to make comments allowed in dictionary, including inlines comments.

Comments allowed
pure comments:

#comment
# comment
# comment
###### comment

inline comments: the first hashtag must be preceded by whitespaces.

abondon->abandon #comment
abondon->abandon # comment
abondon->abandon ###### comment

Illegal comment

abondon->abandon#comment
abondon->abandon# comment

I prefer to ensure there is a space before the hashtag, otherwise hashtags in invalid positions will be interpreted as comments, like:

thenumberone->the#one

Is this setting acceptable ? Referring to #2068, I would also recommend not supporting typos and their correct writing form containing hashes, however we dont know whether users' customized dictionary files includes hashtags or not, so I added some code to deal with that.

Code snippets

forlineinf:
left, pound, _=line.partition("#")
# The first hashtag is treated as comment only if preceded by whitespace# Otherwise it is considered part of `err` or `req`ifpoundandleftandleft[-1] notin (' ', '\t'):
continueline=left.strip()
# Skip empty lines or pure comment linesifnotline:
continuetry:
[key, data] =line.split("->")
exceptValueError:
continue

These lines will be skipped:

aban#don->abandon
abandon->aban#don
abondon->abandon#comment

Blank lines are also skipped.

[AlightSoulmate]

One problem is that without warning, users are less likely to notice that unsatisfactory inline comment lines are skipped, then they wouldn't have corrected them.

[peternewman]

Note that this change would disallow typos and fixes including # - but I think we have decided that's OK 😄

Doing something like requiring ### for the comments would possibly offer the best of both worlds, giving us the option to handle # in future, whilst still allowing comments in dictionaries now.

E.g. we might want to do something with C pre-processor words which begin with a hash. I suspect they're not currently matched as words due to starting with a hash, but that's a different thing...

[DimitriPapadopoulos]

But then I fear ### is too complicated for a simple comment. @larsoner What would you recommend?

[larsoner]

I think going with plain # is good enough. We'll need a big overhaul if we start allowing non-word chars in there...

Also I don't love the try/except with ->, I think the approach in #2617 is safer -- only blank or lines starting with # should be allowed, otherwise they should be treated as corrections and fully parseable (and -> should be in there, etc.)