datacheck fixes for #32 & #85

[yakra]

Most discussion is in Near Match FPs thread on the forum.

There was a mysterious bug in e7f2f9d that was fixed in b642e06. The fact that there was a bug is no surprise, but the symptom left me scratching my head. I want to explore this a little more...
Whatever the case, I think this should be ready to merge in.

[yakra]

There was a mysterious bug in e7f2f9d that was fixed in b642e06. The fact that there was a bug is no surprise, but the symptom left me scratching my head. I want to explore this a little more...

Scratching my head no more. #116 consists of me talking to myself about it.

[jteresco]

Going to give this a shot!

[yakra]

Merged! Still going to tmtest, or straight to the production site?

[michihdeu]

I think a production site update is in progress....

[jteresco]

I decided to go for it right to production. The site update, with no new graph generation, just finished. Do things look good?

[yakra]

-rw-r--r-- 1 terescoj terescoj 169418 Nov 3 14:34 datacheck.log
:D

[yakra]

I'll have a look around datacheck.log, nearmatchfps.log, and unmatchedfps.log, and see what I can see.

[yakra]

No HIDDEN_JUNCTION entries are shown on datacheck.php.
The old siteupdate.py, which skips master graph generation with the -k switch, is still on noreaster:

[yakra@noreaster /home/terescoj/travelmapping/DataProcessing/siteupdate/python-teresco]$ ls -l siteupdate.py -rwxr-xr-x 1 terescoj terescoj 168305 Oct 31 19:30 siteupdate.py

[yakra]

@jteresco, if you run another site update soon, I'll update datacheckfps.csv to remove duplicate entries.

[jteresco]

I can run one any time you'd like.

[yakra]

TravelMapping/HighwayData#2335
then a git pull from /home/terescoj/travelmapping/DataProcessing/
and I'm ready for another one

[jteresco]

pulled in that directory, and launched.

[yakra]

TravelMapping/HighwayData#2335 is still open

[jteresco]

Yes, it would have helped if I merged that first...

In any case, I see this:

[302.9] Writing stats csv files.
[303.1] Setting up for graphs of highway data.
Full graph has 523732 vertices, 593703 edges.
ERROR: segment name mismatch in HighwayGraphCollapsedEdgeInfo: edge1 named OR18,OR99W edge2 named OR99W
Edge compressed graph has 384093 vertices, 454064 edges.
[336.4] Writing graph waypoint simplification log.
[336.6] Writing master TM simple graph file, tm-master-simple.tmg
Traceback (most recent call last):
File "./siteupdate.py", line 2812, in (sv, se) = graph_data.write_master_tmg_simple(args.graphfilepath+'/tm-master-simple.tmg')
File "./siteupdate.py", line 1712, in write_master_tmg_simple
tmgfile = open(filename, 'w')
FileNotFoundError: [Errno 2] No such file or directory: '2018-11-03@15:15:29/graphdata/tm-master-simple.tmg'
Sat Nov 3 15:21:17 EDT 2018
localupdate.sh: running nmpbyregion
localupdate.sh: creating zip archive of all nmp files created by nmpbyregion
localupdate.sh: installing logs, stats, nmp_merged, graphs, archiving old contents in /home/tmp/tm/2018-11-03@15:15:29
localupdate.sh: loading primary DB
Sat Nov 3 15:21:18 EDT 2018
localupdate.sh: cannot open TravelMapping-2018-11-03@15:15:29.sql: No such file or directory

I think it's probably just that it's trying to write to a directory that's not created when the -k flag is specified. You only need the in-memory graph for the datachecks, right? So if there's a -k (args.skipgraphs), then the actual writing of the files should also be omitted.

[yakra]

Aah, I was just running the raw siteupdate.py, ergo no graphfilepath was specified.
Shall I leave the fix for this to you, or take a stab at it myself?

[jteresco]

If you have a chance to experiment, go for it. I will be headed out for several hours pretty soon.

[yakra]

Would it be as simple as moving
https://github.com/TravelMapping/DataProcessing/blob/master/siteupdate/python-teresco/siteupdate.py#L2811-L2818

print(et.et() +"Writing master TM simple graph file, tm-master-simple.tmg", flush=True)
(sv, se) =graph_data.write_master_tmg_simple(args.graphfilepath+'/tm-master-simple.tmg')
graph_list.append(GraphListEntry('tm-master-simple.tmg', 'All Travel Mapping Data', sv, se, 'simple', 'master'))
print(et.et() +"Writing master TM collapsed graph file, tm-master.tmg.", flush=True)
(cv, ce) =graph_data.write_master_tmg_collapsed(args.graphfilepath+'/tm-master.tmg')
graph_list.append(GraphListEntry('tm-master.tmg', 'All Travel Mapping Data', cv, ce, 'collapsed', 'master'))
graph_types.append(['master', 'All Travel Mapping Data',
'These graphs contain all routes currently plotted in the Travel Mapping project.'])

into the else block starting at line 2822?

[jteresco]

It might be - give it a shot. I can start one more site update in a few minutes here.

[yakra]

I won't hold you up. It'll take longer for me to test this.

[yakra]

It was inside the ELSE before. Should be good. I'll open another PR, but if I'm too late, I'm too late; no biggie.

[yakra]

#117

[yakra]

Merged! :D

[jteresco]

siteupdate.py complete, loading the DB now.

[yakra]

Looks like the update completed. I see HIDDEN_JUNCTION errors on datacheck.php now.
I'll give things a look over. Thanks!

[jteresco]

Thank you for all the work on this. Best part is that we now have two people who have some reasonable understanding of the site update code.