diff --git a/README.md b/README.md index 805b2aef..b51da840 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,2 @@ # DataProcessing -Data Processing Scripts and Programs for Travel Mapping Project +Data Processing Scripts and Programs for the Travel Mapping Project. diff --git a/RUNNING.md b/RUNNING.md new file mode 100644 index 00000000..52e066ab --- /dev/null +++ b/RUNNING.md @@ -0,0 +1,67 @@ +This document describes how to use the site update code on the Travel Mapping server to test your changes to highway data before submitting a pull request. The idea is to speed the process of getting your changes and updates into the site by finding things like missing or misnamed files, missing/erroneous/duplicate csv entries, a WPT file with malformed lines, mismatches between .csv and _con.csv files, misspelled CSV entries or file names, etc, etc, etc. By testing before submitting the pull request, problems could be caught by those making the changes rather than having to wait for the next "official" site update to find out. + +### Software Requirements + +All you will need is an ssh client to connect to the FreeBSD server (currently noreaster.teresco.org) that is used by this project. If you are on Windows, the most popular option is likely [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/). If you are on a Mac, your Terminal application already has what you need in the command-line ssh client. Same for other Unix-like environments (Linux, FreeBSD, etc.). + +### Obtaining an account and logging in + +Request an account by email to travmap@teresco.org. We will select a username and you can set a password when you first log in. To connect with PuTTY from Windows, you will create a new connection and enter "noreaster.teresco.org" in the host name, and the port number that you will be given with your account information (we run sshd on a nonstandard port to enhance security). You will be prompted for your username and password. From a Mac Terminal or other Unix-like command-line environment, you will connect with + +``` +ssh -l username -p portnum noreaster.teresco.org +``` + +Where you would replace "username" with your assigned username, and "portnum" with the number you are given with your account information. + +### First time setup + +The first time you connect, you will need to clone the appropriate GitHub repositories into your account. You will need the HighwayData repository, the UserData repository, and the DataProcessing repository. Most often, you will want your own fork of the first, and the master versions of the latter two, as this process is intended to help test changes to highway data. We'll assume that's the case here, and will use "jcool" as the GitHub account into which you have forked the HighwayData repository. + +Once you are logged in, you will see a prompt something like this: + +``` +[jcool@noreaster ~]$ +``` + +That is your Unix command prompt, and by default it is in a "shell" program called Bash. Basically, a shell is a way for you to issue commands directly to the operating system. Our first commands will clone the needed repositories from GitHub. Again, this assumes a GitHub username of "jcool". Type each of these, in turn, at that $ prompt. You may be prompted for a GitHub username and password. Any output from successful or unsuccessful commands will appear in your terminal, followed by a new $ prompt. + +``` +git clone https://github.com/jcool/HighwayData.git +git clone https://github.com/TravelMapping/UserData.git +git clone https://github.com/TravelMapping/DataProcessing.git +``` + +Don't forget to replace "jcool" with your GitHub username into which you have forked the HighwayData repository. + +If all were successful, you should now have copies of each of the repositories in your account on the server. + +### Running the site update code + +At this time, your best bet is to run the full site update program, even though you don't need everything it does. https://github.com/TravelMapping/DataProcessing/issues/73 has been created as a reminder to simplify the process for this purpose. + +To do this, you will enter the following at your $ prompt: + +``` +cd ~/DataProcessing/siteupdate/python-teresco +python3 siteupdate.py -k +``` + +The process that launches will likely run for several minutes. The "-k" omits graph generation, which is one of the slowest parts of the process. You can leave it out if you want to generate graphs. + +If the program runs to completion without reporting errors, you are likely in good shape to make your pull request with your highway data changes. If not, you have things to fix. + +### Updating before subsequent runs + +When you make changes to your data on GitHub, you will need to update your clone on noreaster to match before you run the site update process. It's also a good idea to make sure the user data and data processing clones are up-to-date as well. To do this, you will issue these commands: + +``` +cd ~/DataProcessing +git pull +cd ~/UserData +git pull +cd ~/HighwayData +git pull +``` + +At this point, you can run the site update program as described above. diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 00000000..dabeb726 --- /dev/null +++ b/SETUP.md @@ -0,0 +1,32 @@ +This document is intended to describe how one can set up an environment to run TravelMapping's site update program to generate the same logs, stats, and database file that are produced as part of TM's regular site update process. A separate document is planned in the [Web repository](https://github.com/TravelMapping/Web/) to describe how to take the information generated by the update to populate a database and install and configure the Web-facing code to use it. The expectation is that this is run in a Unix-like environment such as the Mac OS X Terminal, or Linux or FreeBSD. It is likely the same or similar commands will work on Windows in something like Cygwin or Git Bash, but this has not been tested. A [separate document](RUNNING.md) describes how to use a shell account on TravelMapping's server to test your own changes to the highway data before submitting a pull request. + +### Needed Software + +Standard tools expected include bash, bzip2, ssh. + +The system should have a Python3 installation. As of this writing, Python 3.6.4 is being used. Below, we will assume that Python can be launched with the command "python3". + +### Cloning Needed Repositories + +Information from three repositories is needed to run the site update process: + +1. [DataProcessing](https://github.com/TravelMapping/DataProcessing/) +2. [HighwayData](https://github.com/TravelMapping/HighwayData/) +3. [UserData](https://github.com/TravelMapping/UserData/) + +These should be cloned into the same parent directory. Typically, this might be called "travelmapping" and sit off the user's home directory. Below, we will assume that directory is in an environment variable TMBASE, so we refer to it as $(TMBASE). + +Once all repositories have been updated to match the latest versions on GitHub (using "git pull"), the site update program can be run. To run the basic data processing to ensure that all data can be loaded correcly and that no other errors are encountered: + +``` +cd $(TMBASE)/DataProcessing/siteupdate/python-teresco +python3 siteupdate.py +``` + +There is also a collection of bash scripts that run this program, typically launched with + +``` +sh siteupdate.sh +``` + +that also transfers files to the DB and web servers and reloads the DB. diff --git a/siteupdate/python-teresco/.gitignore b/siteupdate/python-teresco/.gitignore new file mode 100644 index 00000000..51ac628b --- /dev/null +++ b/siteupdate/python-teresco/.gitignore @@ -0,0 +1,3 @@ +graphs +logs +nmp_merged diff --git a/siteupdate/python-teresco/localupdate.sh b/siteupdate/python-teresco/localupdate.sh new file mode 100644 index 00000000..7eb22959 --- /dev/null +++ b/siteupdate/python-teresco/localupdate.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# +# script to run a site update on the same system as the web front end +# +# Jim Teresco, Tue May 15 15:57:28 EDT 2018 +# +set -e +install=1 +pull=1 +tmbase=$HOME/travelmapping +tmwebbase=/home/www/tm +datestr=`date '+%Y-%m-%d@%H:%M:%S'` +logdir=logs +statdir=stats +graphdir=graphs +nmpmdir=nmp_merged +graphflag= +date +# process command line args +for arg in "$@"; do + if [ "$arg" == "--nographs" ]; then + # -k to siteupdate.py supresses graph generation + graphflag="-k" + fi + if [ "$arg" == "--noinstall" ]; then + install=0 + fi + if [ "$arg" == "--nopull" ]; then + pull=0 + fi + shift +done +if [ "$pull" == "1" ]; then + echo "$0: updating TM repositories" + (cd $tmbase/HighwayData; git pull) + (cd $tmbase/UserData; git pull) +fi + +echo "$0: creating directories" +mkdir -p $datestr/$logdir $datestr/$statdir $datestr/$nmpmdir $datestr/$graphdir +# put this back later: need to put tm-master.nmp somewhere else +#if [ "$graphflag" != "-k" ]; then +# mkdir -p $datestr/$graphdir +#fi + +echo "$0: launching siteupdate.py" +PYTHONIOENCODING='utf-8' ./siteupdate.py -d TravelMapping-$datestr $graphflag -l $datestr/$logdir -c $datestr/$statdir -g $datestr/$graphdir -n $datestr/$nmpmdir | tee $datestr/$logdir/siteupdate.log 2>&1 || exit 1 +date + +if [ "$install" == "0" ]; then + echo "$0: SKIPPING file copies and DB update" + exit 0 +fi +echo "$0: installing logs, stats, nmp_merged, graphs, archiving old contents in /tmp/$datestr" +mkdir -p /tmp/$datestr +mv $tmwebbase/$logdir /tmp/$datestr +mv $datestr/$logdir $tmwebbase +mv $tmwebbase/$statdir /tmp/$datestr +mv $datestr/$statdir $tmwebbase +mv $tmwebbase/$nmpmdir /tmp/$datestr +mv $datestr/$nmpmdir $tmwebbase +if [ "$graphflag" != "-k" ]; then + mv $tmwebbase/$graphdir /tmp/$datestr + mv $datestr/$graphdir $tmwebbase +fi +rmdir $datestr +echo "$0: switching to DB copy" +ln -sf $tmwebbase/lib/tm.conf.updating $tmwebbase/lib/tm.conf +touch $tmwebbase/dbupdating +echo "$0: loading primary DB" +mysql --defaults-group-suffix=tmapadmin -u travmapadmin TravelMapping < TravelMapping-$datestr.sql +/bin/rm $tmwebbase/dbupdating +echo "$0: switching to primary DB" +ln -sf $tmwebbase/lib/tm.conf.standard $tmwebbase/lib/tm.conf +echo "$0: loading DB copy" +mysql --defaults-group-suffix=tmapadmin -u travmapadmin TravelMappingCopy < TravelMapping-$datestr.sql +echo "$0: complete" +#echo "$0: sending email notification" +#mailx -s "Travel Mapping Site Update Complete" travelmapping-siteupdates@teresco.org < canonical name " + r.list_entry_name() + " in line " + line) + break + + if r.system.devel(): + self.log_entries.append("Ignoring line matching highway in system in development: " + line) + continue + # r is a route match, r.root is our root, and we need to find + # canonical waypoint labels, ignoring case and leading + # "+" or "*" when matching + canonical_waypoints = [] + canonical_waypoint_indices = [] + checking_index = 0; + for w in r.point_list: + lower_label = w.label.lower().strip("+*") + list_label_1 = fields[2].lower().strip("*") + list_label_2 = fields[3].lower().strip("*") + if list_label_1 == lower_label or list_label_2 == lower_label: + canonical_waypoints.append(w) + canonical_waypoint_indices.append(checking_index) + r.labels_in_use.add(lower_label.upper()) + else: + for alt in w.alt_labels: + lower_label = alt.lower().strip("+") if list_label_1 == lower_label or list_label_2 == lower_label: canonical_waypoints.append(w) canonical_waypoint_indices.append(checking_index) r.labels_in_use.add(lower_label.upper()) - else: - for alt in w.alt_labels: - lower_label = alt.lower().strip("+") - if list_label_1 == lower_label or list_label_2 == lower_label: - canonical_waypoints.append(w) - canonical_waypoint_indices.append(checking_index) - r.labels_in_use.add(lower_label.upper()) - # if we have not yet used this alt label, remove it from the unused list - if lower_label.upper() in r.unused_alt_labels: - r.unused_alt_labels.remove(lower_label.upper()) + # if we have not yet used this alt label, remove it from the unused list + if lower_label.upper() in r.unused_alt_labels: + r.unused_alt_labels.remove(lower_label.upper()) - checking_index += 1 - if len(canonical_waypoints) != 2: - self.log_entries.append("Waypoint label(s) not found in line: " + line) - else: - self.list_entries.append(ClinchedSegmentEntry(line, r.root, \ - canonical_waypoints[0].label, \ - canonical_waypoints[1].label)) - # find the segments we just matched and store this traveler with the - # segments and the segments with the traveler (might not need both - # ultimately) - #start = r.point_list.index(canonical_waypoints[0]) - #end = r.point_list.index(canonical_waypoints[1]) - start = canonical_waypoint_indices[0] - end = canonical_waypoint_indices[1] - for wp_pos in range(start,end): - hs = r.segment_list[wp_pos] #r.get_segment(r.point_list[wp_pos], r.point_list[wp_pos+1]) - hs.add_clinched_by(self) - if hs not in self.clinched_segments: - self.clinched_segments.add(hs) - - if lineDone: - break - if not lineDone: - self.log_entries.append("Unknown region/highway combo in line: " + line) + checking_index += 1 + if len(canonical_waypoints) != 2: + self.log_entries.append("Waypoint label(s) not found in line: " + line) + else: + self.list_entries.append(ClinchedSegmentEntry(line, r.root, \ + canonical_waypoints[0].label, \ + canonical_waypoints[1].label)) + # find the segments we just matched and store this traveler with the + # segments and the segments with the traveler (might not need both + # ultimately) + #start = r.point_list.index(canonical_waypoints[0]) + #end = r.point_list.index(canonical_waypoints[1]) + start = canonical_waypoint_indices[0] + end = canonical_waypoint_indices[1] + for wp_pos in range(start,end): + hs = r.segment_list[wp_pos] #r.get_segment(r.point_list[wp_pos], r.point_list[wp_pos+1]) + hs.add_clinched_by(self) + if hs not in self.clinched_segments: + self.clinched_segments.add(hs) + self.log_entries.append("Processed " + str(len(self.list_entries)) + \ " good lines marking " +str(len(self.clinched_segments)) + \ " segments traveled.") @@ -1335,7 +1328,7 @@ def __init__(self,graph,segment=None,vertex_info=None): self.vertex2.incident_collapsed_edges.remove(edge1) removed += 1 if removed != 1: - print("ERROR: edge1 " + str(edge1) + " removed from " + removed + " adjacency lists instead of 1.") + print("ERROR: edge1 " + str(edge1) + " removed from " + str(removed) + " adjacency lists instead of 1.") removed = 0 if edge2 in self.vertex1.incident_collapsed_edges: self.vertex1.incident_collapsed_edges.remove(edge2) @@ -1344,7 +1337,7 @@ def __init__(self,graph,segment=None,vertex_info=None): self.vertex2.incident_collapsed_edges.remove(edge2) removed += 1 if removed != 1: - print("ERROR: edge2 " + str(edge2) + " removed from " + removed + " adjacency lists instead of 1.") + print("ERROR: edge2 " + str(edge2) + " removed from " + str(removed) + " adjacency lists instead of 1.") self.vertex1.incident_collapsed_edges.append(self) self.vertex2.incident_collapsed_edges.append(self) @@ -1879,7 +1872,7 @@ def __init__(self,filename,descr,vertices,edges,format,category): parser.add_argument("-u", "--userlistfilepath", default="../../../UserData/list_files",\ help="path to the user list file data") parser.add_argument("-d", "--databasename", default="TravelMapping", \ - help="Database name for mysql 'USE' statement and .sql file name") + help="Database name for .sql file name") parser.add_argument("-l", "--logfilepath", default=".", help="Path to write log files") parser.add_argument("-c", "--csvstatfilepath", default=".", help="Path to write csv statistics files") parser.add_argument("-g", "--graphfilepath", default=".", help="Path to write graph format data files") @@ -2473,6 +2466,14 @@ def run(self): fpfile.write("No unmatched FP entries.") fpfile.close() +# Create hash table for faster lookup of routes by list file name +print(et.et() + "Creating route hash table for list processing:",flush=True) +route_hash = dict() +for h in highway_systems: + for r in h.route_list: + route_hash[(r.region + ' ' + r.list_entry_name()).lower()] = r + for a in r.alt_route_names: + route_hash[(r.region + ' ' + a).lower()] = r # Create a list of TravelerList objects, one per person traveler_lists = [] @@ -2481,7 +2482,7 @@ def run(self): for t in traveler_ids: if t.endswith('.list'): print(" " + t,end="",flush=True) - traveler_lists.append(TravelerList(t,highway_systems,args.userlistfilepath)) + traveler_lists.append(TravelerList(t,highway_systems,route_hash,args.userlistfilepath)) print(" processed " + str(len(traveler_lists)) + " traveler list files.") # Read updates.csv file, just keep in the fields array for now since we're @@ -3171,7 +3172,7 @@ def run(self): # Once all data is read in and processed, create a .sql file that will # create all of the DB tables to be used by other parts of the project sqlfile = open(args.databasename+'.sql','w',encoding='UTF-8') -sqlfile.write('USE '+args.databasename+';\n') +# Note: removed "USE" line, DB name must be specified on the mysql command line # we have to drop tables in the right order to avoid foreign key errors sqlfile.write('DROP TABLE IF EXISTS datacheckErrors;\n') @@ -3243,7 +3244,7 @@ def run(self): sqlfile.write(";\n") # next, a table of highways, with the same fields as in the first line -sqlfile.write('CREATE TABLE routes (systemName VARCHAR(10), region VARCHAR(8), route VARCHAR(16), banner VARCHAR(3), abbrev VARCHAR(3), city VARCHAR(100), root VARCHAR(32), mileage FLOAT, PRIMARY KEY(root), FOREIGN KEY (systemName) REFERENCES systems(systemName));\n') +sqlfile.write('CREATE TABLE routes (systemName VARCHAR(10), region VARCHAR(8), route VARCHAR(16), banner VARCHAR(6), abbrev VARCHAR(3), city VARCHAR(100), root VARCHAR(32), mileage FLOAT, PRIMARY KEY(root), FOREIGN KEY (systemName) REFERENCES systems(systemName));\n') sqlfile.write('INSERT INTO routes VALUES\n') first = True for h in highway_systems: @@ -3255,7 +3256,7 @@ def run(self): sqlfile.write(";\n") # connected routes table, but only first "root" in each in this table -sqlfile.write('CREATE TABLE connectedRoutes (systemName VARCHAR(10), route VARCHAR(16), banner VARCHAR(3), groupName VARCHAR(100), firstRoot VARCHAR(32), mileage FLOAT, PRIMARY KEY(firstRoot), FOREIGN KEY (firstRoot) REFERENCES routes(root));\n') +sqlfile.write('CREATE TABLE connectedRoutes (systemName VARCHAR(10), route VARCHAR(16), banner VARCHAR(6), groupName VARCHAR(100), firstRoot VARCHAR(32), mileage FLOAT, PRIMARY KEY(firstRoot), FOREIGN KEY (firstRoot) REFERENCES routes(root));\n') sqlfile.write('INSERT INTO connectedRoutes VALUES\n') first = True for h in highway_systems: @@ -3423,7 +3424,7 @@ def run(self): sqlfile.write(";\n") # updates entries -sqlfile.write('CREATE TABLE updates (date VARCHAR(10), region VARCHAR(60), route VARCHAR(80), root VARCHAR(32), description VARCHAR(512));\n') +sqlfile.write('CREATE TABLE updates (date VARCHAR(10), region VARCHAR(60), route VARCHAR(80), root VARCHAR(32), description VARCHAR(1024));\n') sqlfile.write('INSERT INTO updates VALUES\n') first = True for update in updates: @@ -3434,7 +3435,7 @@ def run(self): sqlfile.write(";\n") # systemUpdates entries -sqlfile.write('CREATE TABLE systemUpdates (date VARCHAR(10), region VARCHAR(48), systemName VARCHAR(10), description VARCHAR(50), statusChange VARCHAR(16));\n') +sqlfile.write('CREATE TABLE systemUpdates (date VARCHAR(10), region VARCHAR(48), systemName VARCHAR(10), description VARCHAR(128), statusChange VARCHAR(16));\n') sqlfile.write('INSERT INTO systemUpdates VALUES\n') first = True for systemupdate in systemupdates: diff --git a/siteupdate/python-teresco/siteupdate.sh b/siteupdate/python-teresco/siteupdate.sh index 36d539c6..06db3131 100644 --- a/siteupdate/python-teresco/siteupdate.sh +++ b/siteupdate/python-teresco/siteupdate.sh @@ -2,6 +2,7 @@ # set -e read_data=1 +transfer=1 logdir=logs statdir=stats graphdir=graphs @@ -16,6 +17,9 @@ if [ $# -eq 1 ]; then if [ "$1" == "--nographs" ]; then graphflag="-k" fi + if [ "$1" == "--noxfer" ]; then + transfer=0 + fi fi if [ "$read_data" == "1" ]; then echo "siteupdate.sh: launching siteupdate.py" @@ -28,6 +32,10 @@ else echo "siteupdate.sh: SKIPPING siteupdate.py" fi date +if [ "$transfer" == "0" ]; then + echo "siteupdate.sh: SKIPPING file transfers and DB update" + exit 0 +fi echo "siteupdate.sh: Bzipping TravelMapping.sql file" bzip2 -9f TravelMapping.sql echo "siteupdate.sh: Transferring TravelMapping.sql.bz2 to blizzard"