sync, 2018-05-16

README.md

@@ -1,2 +1,2 @@
 # DataProcessing
-Data Processing Scripts and Programs for Travel Mapping Project
+Data Processing Scripts and Programs for the Travel Mapping Project.

RUNNING.md

@@ -0,0 +1,67 @@
+This document describes how to use the site update code on the Travel Mapping server to test your changes to highway data before submitting a pull request. The idea is to speed the process of getting your changes and updates into the site by finding things like missing or misnamed files, missing/erroneous/duplicate csv entries, a WPT file with malformed lines, mismatches between .csv and _con.csv files, misspelled CSV entries or file names, etc, etc, etc. By testing before submitting the pull request, problems could be caught by those making the changes rather than having to wait for the next "official" site update to find out.
+
+### Software Requirements
+
+All you will need is an ssh client to connect to the FreeBSD server (currently noreaster.teresco.org) that is used by this project. If you are on Windows, the most popular option is likely [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/). If you are on a Mac, your Terminal application already has what you need in the command-line ssh client. Same for other Unix-like environments (Linux, FreeBSD, etc.).
+
+### Obtaining an account and logging in
+
+Request an account by email to travmap@teresco.org. We will select a username and you can set a password when you first log in. To connect with PuTTY from Windows, you will create a new connection and enter "noreaster.teresco.org" in the host name, and the port number that you will be given with your account information (we run sshd on a nonstandard port to enhance security). You will be prompted for your username and password. From a Mac Terminal or other Unix-like command-line environment, you will connect with
+
+```
+ssh -l username -p portnum noreaster.teresco.org
+```
+
+Where you would replace "username" with your assigned username, and "portnum" with the number you are given with your account information.
+
+### First time setup
+
+The first time you connect, you will need to clone the appropriate GitHub repositories into your account. You will need the HighwayData repository, the UserData repository, and the DataProcessing repository. Most often, you will want your own fork of the first, and the master versions of the latter two, as this process is intended to help test changes to highway data. We'll assume that's the case here, and will use "jcool" as the GitHub account into which you have forked the HighwayData repository.
+
+Once you are logged in, you will see a prompt something like this:
+
+```
+[jcool@noreaster ~]$ 
+```
+
+That is your Unix command prompt, and by default it is in a "shell" program called Bash. Basically, a shell is a way for you to issue commands directly to the operating system. Our first commands will clone the needed repositories from GitHub. Again, this assumes a GitHub username of "jcool". Type each of these, in turn, at that $ prompt. You may be prompted for a GitHub username and password. Any output from successful or unsuccessful commands will appear in your terminal, followed by a new $ prompt.
+
+```
+git clone https://github.com/jcool/HighwayData.git
+git clone https://github.com/TravelMapping/UserData.git
+git clone https://github.com/TravelMapping/DataProcessing.git
+```
+
+Don't forget to replace "jcool" with your GitHub username into which you have forked the HighwayData repository.
+
+If all were successful, you should now have copies of each of the repositories in your account on the server.
+
+### Running the site update code
+
+At this time, your best bet is to run the full site update program, even though you don't need everything it does. https://github.com/TravelMapping/DataProcessing/issues/73 has been created as a reminder to simplify the process for this purpose.
+
+To do this, you will enter the following at your $ prompt:
+
+```
+cd ~/DataProcessing/siteupdate/python-teresco
+python3 siteupdate.py -k
+```
+
+The process that launches will likely run for several minutes. The "-k" omits graph generation, which is one of the slowest parts of the process. You can leave it out if you want to generate graphs.
+
+If the program runs to completion without reporting errors, you are likely in good shape to make your pull request with your highway data changes. If not, you have things to fix.
+
+### Updating before subsequent runs
+
+When you make changes to your data on GitHub, you will need to update your clone on noreaster to match before you run the site update process. It's also a good idea to make sure the user data and data processing clones are up-to-date as well. To do this, you will issue these commands:
+
+```
+cd ~/DataProcessing
+git pull
+cd ~/UserData
+git pull
+cd ~/HighwayData
+git pull
+```
+
+At this point, you can run the site update program as described above.

SETUP.md

@@ -0,0 +1,32 @@
+This document is intended to describe how one can set up an environment to run TravelMapping's site update program to generate the same logs, stats, and database file that are produced as part of TM's regular site update process. A separate document is planned in the [Web repository](https://github.com/TravelMapping/Web/) to describe how to take the information generated by the update to populate a database and install and configure the Web-facing code to use it. The expectation is that this is run in a Unix-like environment such as the Mac OS X Terminal, or Linux or FreeBSD. It is likely the same or similar commands will work on Windows in something like Cygwin or Git Bash, but this has not been tested. A [separate document](RUNNING.md) describes how to use a shell account on TravelMapping's server to test your own changes to the highway data before submitting a pull request.
+
+### Needed Software
+
+Standard tools expected include bash, bzip2, ssh.
+
+The system should have a Python3 installation. As of this writing, Python 3.6.4 is being used. Below, we will assume that Python can be launched with the command "python3".
+
+### Cloning Needed Repositories
+
+Information from three repositories is needed to run the site update process:
+
+1. [DataProcessing](https://github.com/TravelMapping/DataProcessing/)
+2. [HighwayData](https://github.com/TravelMapping/HighwayData/)
+3. [UserData](https://github.com/TravelMapping/UserData/)
+
+These should be cloned into the same parent directory. Typically, this might be called "travelmapping" and sit off the user's home directory. Below, we will assume that directory is in an environment variable TMBASE, so we refer to it as $(TMBASE).
+
+Once all repositories have been updated to match the latest versions on GitHub (using "git pull"), the site update program can be run. To run the basic data processing to ensure that all data can be loaded correcly and that no other errors are encountered:
+
+```
+cd $(TMBASE)/DataProcessing/siteupdate/python-teresco
+python3 siteupdate.py
+```
+
+There is also a collection of bash scripts that run this program, typically launched with
+
+```
+sh siteupdate.sh
+```
+
+that also transfers files to the DB and web servers and reloads the DB.

siteupdate/python-teresco/.gitignore

@@ -0,0 +1,3 @@
+graphs
+logs
+nmp_merged

siteupdate/python-teresco/localupdate.sh

@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# script to run a site update on the same system as the web front end
+#
+# Jim Teresco, Tue May 15 15:57:28 EDT 2018
+#
+set -e
+install=1
+pull=1
+tmbase=$HOME/travelmapping
+tmwebbase=/home/www/tm
+datestr=`date '+%Y-%m-%d@%H:%M:%S'`
+logdir=logs
+statdir=stats
+graphdir=graphs
+nmpmdir=nmp_merged
+graphflag=
+date
+# process command line args
+for arg in "$@"; do
+ if [ "$arg" == "--nographs" ]; then
+	# -k to siteupdate.py supresses graph generation
+	graphflag="-k"
+ fi
+ if [ "$arg" == "--noinstall" ]; then
+	install=0
+ fi
+ if [ "$arg" == "--nopull" ]; then
+	pull=0
+ fi
+ shift
+done
+if [ "$pull" == "1" ]; then
+ echo "$0: updating TM repositories"
+ (cd $tmbase/HighwayData; git pull)
+ (cd $tmbase/UserData; git pull)
+fi
+
+echo "$0: creating directories"
+mkdir -p $datestr/$logdir $datestr/$statdir $datestr/$nmpmdir $datestr/$graphdir
+# put this back later: need to put tm-master.nmp somewhere else
+#if [ "$graphflag" != "-k" ]; then
+# mkdir -p $datestr/$graphdir
+#fi
+
+echo "$0: launching siteupdate.py"
+PYTHONIOENCODING='utf-8' ./siteupdate.py -d TravelMapping-$datestr $graphflag -l $datestr/$logdir -c $datestr/$statdir -g $datestr/$graphdir -n $datestr/$nmpmdir | tee $datestr/$logdir/siteupdate.log 2>&1 || exit 1
+date
+
+if [ "$install" == "0" ]; then
+ echo "$0: SKIPPING file copies and DB update"
+ exit 0
+fi
+echo "$0: installing logs, stats, nmp_merged, graphs, archiving old contents in /tmp/$datestr"
+mkdir -p /tmp/$datestr
+mv $tmwebbase/$logdir /tmp/$datestr
+mv $datestr/$logdir $tmwebbase
+mv $tmwebbase/$statdir /tmp/$datestr
+mv $datestr/$statdir $tmwebbase
+mv $tmwebbase/$nmpmdir /tmp/$datestr
+mv $datestr/$nmpmdir $tmwebbase
+if [ "$graphflag" != "-k" ]; then
+ mv $tmwebbase/$graphdir /tmp/$datestr
+ mv $datestr/$graphdir $tmwebbase
+fi
+rmdir $datestr
+echo "$0: switching to DB copy"
+ln -sf $tmwebbase/lib/tm.conf.updating $tmwebbase/lib/tm.conf
+touch $tmwebbase/dbupdating
+echo "$0: loading primary DB"
+mysql --defaults-group-suffix=tmapadmin -u travmapadmin TravelMapping < TravelMapping-$datestr.sql
+/bin/rm $tmwebbase/dbupdating
+echo "$0: switching to primary DB"
+ln -sf $tmwebbase/lib/tm.conf.standard $tmwebbase/lib/tm.conf
+echo "$0: loading DB copy"
+mysql --defaults-group-suffix=tmapadmin -u travmapadmin TravelMappingCopy < TravelMapping-$datestr.sql
+echo "$0: complete"
+#echo "$0: sending email notification"
+#mailx -s "Travel Mapping Site Update Complete" travelmapping-siteupdates@teresco.org <<EOF
+#A Travel Mapping site update has just successfully completed.
+#The complete log is available at http://tm.teresco.org/logs/siteupdate.log .
+#Please report any problems to travmap@teresco.org .
+#EOF
+date