Subset of site update process to verify highway data changes

[jteresco]

Idea: add an option to the site update program that only does enough to find common mistakes in highway data. No graphs would need to be generated, user list files would not need to be fully processed, and no sql file would need to be generated.

[jteresco]

This should read the updates.csv and systemupdates.csv and should generate a datacheck log file or other messages that can be used to find datacheck problems.

[michihdeu]

More requirements:

• Log of Data Errors (without FPs) It would be great if the datacheck log mentioned by Jim would output errors without FPs so that one can directly find the relevant entries caused by one's latest update. The entry order should be alphabetically by region.
• Log of Unmatched FPs from datacheckfps.csv
• Log of Unprocessed WPTs in the Repository
• Graph generation for a specific region The user should be able to enter a region code, e.g. "deu-rp". The output file can be manually load to HDX for checking concurrencies et cetera; I think it should likely be a different command to be run

That means, one must be able to download the file.

[yakra]

Log of Data Errors (without FPs) It would be great if the datacheck log mentioned by Jim would output errors without FPs so that one can directly find the relevant entries caused by one's latest update. The entry order should be alphabetically by region.

Having the log file available to inspect immediately would be useful. I found myself wishing for this a couple times.
http://tm.teresco.org/forum/index.php?topic=325

Log of Unmatched FPs from datacheckfps.csv
Log of Unprocessed WPTs in the Repository

unmatchedfps.log and unprocessedwpts.log are produced by the full siteupdate.py, and stored in siteupdate/python-teresco. This functionality could be preserved in a "siteupdate-lite" program, possibly with a commandline switch to disable if it would result in significant time savings.

Graph generation for a specific region The user should be able to enter a region code, e.g. "deu-rp". The output file can be manually load to HDX for checking concurrencies et cetera; I think it should likely be a different command to be run

I'm more lukewarm on this personally. I think we'd be better off just running the full siteupdate.py here. The process can be sped up by editing systems.csv to only include the systems that would be relevant for such a graph. (E.G., for deu-rp, include only deua, deub, deurpl, & eure.)
We'd just have to be sure to include at least one active and at least one preview system (#16).

[michihdeu]

This functionality could be preserved in a "siteupdate-lite" program, possibly with a commandline switch to disable if it would result in significant time savings.

From the last update:

[231.8] Performing data checks.
[247.2] Marking datacheck false positives.
[269.0] Writing log of unmatched datacheck FP entries.
[269.0] Creating route hash table for list processing:

37 seconds. Should be an option, e.g. "-d" to include it

[michihdeu]

The process can be sped up by editing systems.csv to only include the systems that would be relevant for such a graph. (E.G., for deu-rp, include only deua, deub, deurpl, & eure.)

I also thought about this option but I don't like that we have to modify systems.csv because we must remember to undo it again. You can easily forget systems, e.g. of neighboring countries when you've synced wpts at borders. I don't think that it's clever to check data with a modified set of it.

Well, I mean, Jim wrote that he wants to disable graph generation by default. If an option remains to generate (one or all) graphs, it's fine with me.

I've mentioned it because we need an option to download the graph files.

[jteresco]

My plan is to have the same information that's in the DB to populate the datacheck page, but produced in a plain text format useful to find the errors quickly.

[michihdeu]

My plan is to have the same information

This!

I think it's wise to avoid redundant data.

[jteresco]

For a modified systems.csv, there's already an option you can use to limit to your own systems if you want. If you had a deu.csv in the same location as systems.csv, you could add

-s deu.csv

to your command line and it would only process systems listed there.

I expect we'd run into some trouble trying to generate graphs when a small subset of the data is loaded.

[jteresco]

I'm thinking the generation of the nmp_merged files can be left out for the error checking option on the site update program. Opinions?

[jteresco]

I also think it's worth keeping in the user list file processing so points in use and unused alt label logs can be generated. That part of the process looks to take about 25 seconds on my Mac.

[michihdeu]

Good to know the -s option but again, I don't like it for data check because the result might be falsified if you do something wrong and the check should be used to find exactly these errors.

NMP: I have no strong feeling on this. I currently pay no attention to it because of the difficult FP handling. And I forgot the details meanwhile. I wait for improvement: TravelMapping/Web#187. However, I think it's a very important data check! I partially do it manually now.

Once NMP check will be fully integrated, it will be needed here to avoid broken concurrencies on the site.

If user list file processing remains, only the subsequent things could be skipped to save time (plus graphs which were skipped last time):
[326.3] Creating per-traveler stats log entries and augmenting data structure.
[369.8] Writing traveler list logs.
[370.1] Writing stats csv files.
[370.3] SKIPPING generation of graphs.
[370.3] Writing database file TravelMapping.sql.
[382.7] Processed 292 highway systems.

[jteresco]

Since those last ones are fairly quick, I've left them in so far.

What are preferences for datacheck.log? Include matched FPs (labeled as such), put matched FPs in a separate log file, or ignore matched FPs? Right now, I have it generating lines like these:

nv.nv599 JonBlvd VISIBLE_HIDDEN_COLOC nv.us095buslsv@+NV596
FP chnnx.g70 +G22_W VISIBLE_HIDDEN_COLOC chnnx.g22@G70_W
FP chngx.g72 +G65/G76 VISIBLE_HIDDEN_COLOC chngx.g65@G72_E
isl.f235 ToHel VISIBLE_HIDDEN_COLOC isl.f235@+X20
FP imn.ttcir Start VISIBLE_HIDDEN_COLOC imn.a002@+TTSF
FP imn.ttcir Finish VISIBLE_HIDDEN_COLOC imn.a002@+TTSF
az.us066hiskin +Coc/Yav VISIBLE_HIDDEN_COLOC az.az066@Coc/Yav
mo.us066hisstl FR144 VISIBLE_HIDDEN_COLOC mo.mo266@+X01
ca.us099hisind +PalmSprLim VISIBLE_HIDDEN_COLOC ca.ca111pal@PalmSprLim
FP alb.a001 +X935790 +X557517 SH5 SHARP_ANGLE 153.23

[michihdeu]

It's just a log for contributors testing their latest changes. I would exclude FPs to keep it as simple as possible for contributors. Another log file for unmatched FPs.