Check for non-ASCII entries in wpt, list files

[jteresco]

The site update program should report and error and continue if a user's list file or a wpt file as non-ASCII characters. Currently, the site update program crashes without an especially helpful error. This might be best done as a pre-processing step or as the first task in a site update that will be noticed quickly rather than several minutes into the site update process.

[yakra]

https://github.com/TravelMapping/UserData/blob/9ab420d7cc23d79b6d24f224c7186c88ee6ef777/list_files/lowenbrau.list

[yakra]

Thinking this thru, just in terms of WPT files for the moment...
A non-ASCII character on any given line would have to wind up in the primary label, alternate labels, or URL.

• The URL doesn't get stored, and shouldn't matter as long as we can extract the lat_string and lng_string from its parts, and parse them. The proposed MALFORMED_URL datacheck should catch & report all of these, and allow siteupdate to continue without crashing.
• For primary labels, we similarly have the LABEL_INVALID_CHAR datacheck.
• This leaves us with just AltLabels. Are there even any known AltLabels with non-ASCII characters currently? If so, should they be left in for compatibility's sake? If not, should we prohibit them & make sure none creep in in the future?
• The OP in this issue predates the addition of PYTHONIOENCODING='utf-8' by about 10 months. With that now in place, is this even as much of in issue anymore?

[michihdeu]

This leaves us with just AltLabels. Are there even any known AltLabels with non-ASCII characters currently?

I don't think so. They had likely made trouble in the past.

If so, should they be left in for compatibility's sake? If not, should we prohibit them & make sure none creep in in the future?

I think that non-ASCII character are usually not added on purpose but by accident. And by accident, characters can appear everywhere in any file. If you check the file, you need to check the whole file.

[jteresco]

Non-ASCII characters used to cause problems in site updates on FreeBSD before the utf-8 setting allowed that. Notorious for this would be commented-out lines in systems.csv, where international characters are permitted.

[yakra]

@yakra wrote:

This leaves us with just AltLabels. Are there even any known AltLabels with non-ASCII characters currently?

Confirmed, there are none.

@michihdeu wrote:

I think that non-ASCII character are usually not added on purpose but by accident.

This is true.

And by accident, characters can appear everywhere in any file.

*sigh*, also true.
Sometimes I'll be editing some C++, and it won't compile (after pasting?), due to non-ASCII characters I can't see in pluma. I delete the offending line, retype it, and then compilesville, daddy-o.

@jteresco wrote:

Non-ASCII characters used to cause problems in site updates on FreeBSD before the utf-8 setting allowed that.

Now that we use this option, non-ASCII characters are much less of a showstopper.

@michihdeu wrote:

If you check the file, you need to check the whole file.

I'd say, only the necessary parts, just what would affect the functionality of the site.
URLs aren't stored; we just need to extract lat_string & lng_string and convert them to float. To that end, http://madmagazine.com/?[%15]=43.661855&[%FF]=-70.270169 is a valid URL, that can be parsed.

Primary labels are already checked.
This leaves us AltLabels.

I've tested out a version of siteupdate that expands the LABEL_INVALID_CHAR datacheck to AltLabels. On noreaster, it didn't add a noticeable amount of runtime, in one case even performing the data checks faster than the reference, siteupdate.py in #158.
Who knows what other processes noreaster may have been running at the same time though; I'll see if I get more deterministic results on BiggaTomato.

[michihdeu]

URLs aren't stored; we just need to extract lat_string & lng_string and convert them to float. To that end, http://madmagazine.com/?[%15]=43.661855&[%FF]=-70.270169 is a valid URL, that can be parsed.

I didn't know that. I understand now why my previous post was... kind of useless 😃

[yakra]

On noreaster, it didn't add a noticeable amount of runtime, in one case even performing the data checks faster than the reference, siteupdate.py in #158.
Who knows what other processes noreaster may have been running at the same time though; I'll see if I get more deterministic results on BiggaTomato.

On BiggaTomato:
0.2 s slower, best-case
0.4 s slower, average
0.5 s slower, worst-case

Not much harm in expanding LABEL_INVALID_CHAR to AltLabels, just to be safe.

What would be the more useful syntax for reporting them?
me.us001;NH/ME;;;LABEL_INVALID_CHAR;+Foo#Bar, or
me.us001;+Foo#Bar;;;LABEL_INVALID_CHAR;?
Right now, for primary labels only, the Info field is not used, E.G.
chngd.g94;?;;;LABEL_INVALID_CHAR;

[yakra]

Re .list files:

https://github.com/TravelMapping/UserData/blob/9ab420d7cc23d79b6d24f224c7186c88ee6ef777/list_files/lowenbrau.list

The final line, ON ON405 ON/USA PorRd, is followed by 1486 null \0s, before a final CRLF.

We can see that this line is not parsed:
http://travelmapping.net/logs/users/lowenbrau.log
http://travelmapping.net/hb/index.php?units=miles&u=lowenbrau&r=on.on405