Expand en_GB->en_US dictionary

[waldyrious]

Based on the resources collected at #103.

I made my best effort to make this exhaustive regarding all the pairs listed in that thread and in the pages linked from it, so I suppose this PR closes#103.

[waldyrious]

Travis is complaining that the following words added to data/dictionary_en-GB_to_en-US.txt should be in aspell:

• cancelation
• customisable
• licenced
• licencing
• manoeuvering
• parallelising
• parallelled
• parallelling
• parametrise
• proveable
• rasterisation
• rasterise
• resizeable
• sanitisation
• unintialised
• vectorisation
• vectorise
• writeable

What should I do?

[peternewman]

Travis is complaining that the following words added to data/dictionary_en-GB_to_en-US.txt should be in aspell:

So those words aren't in aspell's dictionaries, even the US large one:
http://app.aspell.net/lookup?dict=en_US-large

Apart from a few are in the "larger" one (proveable, sanitisation, vectorise) and one in large (writeable).

You can see we currently expect both the GB and US variants to be in aspell:
https://github.com/codespell-project/codespell/blob/master/codespell_lib/_codespell.py#L39-L48

So we can do some, all or none of these:

• Load a bigger dictionary into aspell
• Check with multiple dictionaries (GB and US)
• Remove some or all of the spelling check requirements for that dictionary
• Split into common and rare ones
• Sanity check the PR contents for the edge cases not in any dictionary

[waldyrious]

Thanks for the clarifications. I have a few observations:

So those words aren't in aspell's dictionaries, even the US large one:

You can see we currently expect both the GB and US variants to be in aspell:
/codespell_lib/_codespell.py@master#L39-L48

I'm confused — why would we expect the "err" part of the gb->us dictonary to be in the US aspell dictionary? By definition they would be expected in the GB dictionary instead. Of course, it would make sense to test the "corrections" words against the US aspell dictionary.

Check with multiple dictionaries (GB and US)

Yeah, this is what makes the most sense to me, although I have no strong objections to any of the other options, except "Remove some or all of the spelling check requirements for that dictionary". In any case, I'll defer to more experienced contributors to make the call.

[waldyrious]

PS - checking the word list above against http://app.aspell.net/lookup?dict=en_GB-large led me to realize I had a typo in the dictionary! 😅

Where I wrote unintialised, it should instead be uninitialised. The former shows a frequency of exactly 0 in the Google Books corpus, while the latter shows a frequency of 0.0030. I've just pushed a fix.

[peternewman]

You can see we currently expect both the GB and US variants to be in aspell:
/codespell_lib/_codespell.py@master#L39-L48

I'm confused — why would we expect the "err" part of the gb->us dictonary to be in the US aspell dictionary? By definition they would be expected in the GB dictionary instead. Of course, it would make sense to test the "corrections" words against the US aspell dictionary.

Sorry I wasn't very clear, we currently check against the "en" dictionary, which is a combination of en-GB, en-US (as well as Canadian and Australian):
https://github.com/codespell-project/codespell/blob/master/codespell_lib/tests/test_dictionary.py#L15

Check with multiple dictionaries (GB and US)

I.e. we're already doing that.

Yeah, this is what makes the most sense to me, although I have no strong objections to any of the other options, except "Remove some or all of the spelling check requirements for that dictionary". In any case, I'll defer to more experienced contributors to make the call.

[peternewman]

PS - checking the word list above against http://app.aspell.net/lookup?dict=en_GB-large led me to realize I had a typo in the dictionary!

Where I wrote unintialised, it should instead be uninitialised. The former shows a frequency of exactly 0 in the Google Books corpus, while the latter shows a frequency of 0.0030. I've just pushed a fix.

Ah, well that would suggest that either splitting into a GB-US-rare dictionary, and/or using a bigger source dictionary would be wise, so we could pay more attention to the edge cases in that rare dictionary which aren't being sanity checked by aspell.

[waldyrious]

I'm a bit out of my depth here, so I'll wait for you guys to decide how to proceed :)

[peternewman]

I'm a bit out of my depth here, so I'll wait for you guys to decide how to proceed :)

Given this has slightly stalled for now @waldyrious , do you want to either create a new PR or edit this one to just have the words that pass the existing Travis checks until we work out what to do, or until I get #1485 in, which would probably work around most of these issues?

[waldyrious]

@peternewman sure, done.

[peternewman]

LGTM

[peternewman]

@peternewman sure, done.

Thanks. Please open a new one with all the more complicated edge cases in too.

[waldyrious]

Please open a new one with all the more complicated edge cases in too.

@peternewman, should I wait until #1485 is done to do so?

[peternewman]

I'd probably open it so we don't forget about the words @waldyrious

[waldyrious]

Please open a new one with all the more complicated edge cases in too.

@peternewman, should I wait until #1485 is done to do so?

I'd probably open it so we don't forget about the words @waldyrious

Ok, I've opened #1562.